BatchedMesh: Proposal

[donmccurdy]

Currently the APIs available for reducing draw calls include:

• InstancedMesh
• BufferGeometryUtils.mergeBufferGeometries([a, b, c, ...])

I think we might be able to solve some common difficulties with reducing draw calls, even where the objects to be rendered are not instances of the same thing. I've floated the idea in a couple of earlier comments (https://github.com/mrdoob/three.js/issues/19164#issuecomment-631237304, https://github.com/mrdoob/three.js/issues/18918#issuecomment-602395984) but wanted to open a separate issue for it.

Calling this BatchedMesh for now, it's meant to be a way of (1) merging, (2) rendering, and (3) updating a group of objects that would otherwise be drawn as separate Mesh instances. The objects must all be the same primitive type (i.e. triangles), must share a material, and must have compatible vertex attributes (e.g. if one has normals, they all have normals). With those requirements we could offer an API like:

// Create BatchedMesh with max vertex and triangle counts to support. If 
// geometry is not indexed, maxTriangleCount is optional. The 'template'
// geometry could just be the first geometry we'll add later — it isn't stored,
// but is used to determine which vertex attributes to allocate.
const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxTriangleCount );

// Append geometry, up to max vertex and triangle limit.
const id1 = batchedMesh.add( geometry1, matrix1 );
const id2 = batchedMesh.add( geometry2, matrix2 );
const id3 = batchedMesh.add( geometry3, matrix3 );

scene.add( batchedMesh );

renderer.render( scene, camera );

// Update object position dynamically. For small objects this may be cheaper than
// a separate draw call. For large objects, it may be better to draw the object
// separately than to update it with dynamic batching.
batchedMesh.setMatrixAt( id1, otherMatrix );

renderer.render( scene, camera );

This offers a few advantages over mergeBufferGeometries(). First, you get an ID that lets you remove or modify each geometry later. Second, we can potentially do much faster raycasting (we know the bounding box and transform of each member geometry) and return that ID as a result. Third, we can do frustum culling (maybe?) by updating the index buffer.

[takahirox]

Sounds like it fits to WebGL_multi_draw extension.

https://developer.mozilla.org/en-US/docs/Web/API/WEBGL_multi_draw

Advantage:

• Visibility handling (invisible due to frustum culling or .visible=false) may be easier. You can easily specify which objects in BatchedMesh should be rendered by controlling furstList, countsList, and drawCount of multiDrawArraysWEBGL().
• You can use gl_DrawID in shader. So if you use uniform array, you can apply different uniform parameter for each object in BatchedMesh. It would be more flexible, for example you can apply different material color for each object, and more performant, for example you will no longer need to directly update positions in setMatrixAt() in CPU.

Disadvantage:

• Some major browsers, eg: Firefox and Safari, don't support the extension yet
• If we use the uniform array approach, the maximum uniform array size can limit the number of objects which can be added to BatchedMesh.

[donmccurdy]

Sounds like it fits to WebGL_multi_draw extension.

Oh even better! Do you know if there's signal from Firefox or Safari on plans to support yet? If it's not on the way I would probably lean toward doing a first implementation without it, but trying to keep a compatible API so we can upgrade later. This also gives the advantage that we can implement it in examples/jsm/* without core changes just yet.

[donmccurdy]

and more performant, for example you will no longer need to directly update positions in setMatrixAt() in CPU.

This would be really helpful... And it saves us from having to keep an extra (un-transformed) copy of vertex data in memory. If I understand correctly, dynamic batching in popular game engines will not do this, but instead rewrites vertices on the CPU, I'm not sure why that would be if this extension is widely availlable. 🤔

[jbaicoianu]

@donmccurdy I did some experiments with this idea a few years back, it's definitely an improvement if you have lots of mostly-static geometry in the scene.

One trick I found that made it easier to work with is to use THREE.Mesh instead of geometry + matrix, and then the batchedMesh.add() calls return a proxy object, THREE.MergedObject or something similarly-named, which lets you continue to use .position .rotation etc. as if it were still a separate standalone object. This proxy object can be set up so that changes are automatically carried through to the merged object's combined buffers, so the developer can use the same code for both merged and unmerged cases:

const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxTriangleCount );

// Append meshes, up to max vertex and triangle limit.
const obj1 = batchedMesh.add( mesh1 );
const obj2 = batchedMesh.add( mesh2 );
const obj3 = batchedMesh.add( mesh3 );

function update(dt) {
  obj1.position.set(Math.sin(performance.now() / 1000), 0, 0);
  obj1.rotation.y += dt;
  // etc
}

[takahirox]

I have been wanting to introduce WEBGL_multi_draw extension to Three.js but I gave up before because the existing Three.js API doesn't really fit to the extension. So I'm very positive to introduce a new Three.js API (BatchedMesh) which may be fitting to the extension.

@donmccurdy

Do you know if there's signal from Firefox or Safari on plans to support yet?

The WebGL_multi_draw support ticket for Firefox has been made but it isn't first priority (P2), so I don't think at least it lands soon.

https://bugzilla.mozilla.org/show_bug.cgi?id=1536673

I have no idea about Safari.

I would probably lean toward doing a first implementation without it, but trying to keep a compatible API so we can upgrade later.

That sounds good to me. Even if we use the extension, we will need fallback path until all major browsers support the extension.

Even without the extension, we may be able to use the uniform array approach by adding an extra vertex data having id. It may be a bit memory costly tho.

And it saves us from having to keep an extra (un-transformed) copy of vertex data in memory.

Yeah. And we will not need to transfer updated positions to WebGL, it is another performance advantage.

If I understand correctly, dynamic batching in popular game engines will not do this, but instead rewrites vertices on the CPU, I'm not sure why that would be if this extension is widely availlable. 🤔

Interesting... I guess updating big uniform array every frame might be costly or they want scalability (the maximum uniform array size limits the number of objects)?

How can I know the maximum uniform array size? Max Fragment/Vertex Uniform Vectors?

https://webglreport.com/?v=2

And I'm wondering how many objects per one BachedMesh we will want to support.

@jbaicoianu

Personally I prefer your API idea because it would be more user friendly. But coming up some thoughts in my mind, for example what obj1/2/3.clone() should mean? Should we restrict some methods for that usage?

Update: I prefer passing geometry rather than mesh on this API idea because meshes can have different materials.

const obj1 = batchedMesh.add( mesh1 );

[donmccurdy]

Maybe two ways to approach the proxy objects:

• (a) they inherit from Object3D, and can be added to the scene graph and manipulated there
  • More overhead, we're paying the full cost of the scene graph traversal, but it's a very convenient API. Will hit performance cliffs for large object counts.
• (b) they are just simple objects that expose proxied position/rotation/scale/matrix properties
  • We get notified when something changes and overhead is lower, but there's no scene graph.
  • I think this is probably a bad compromise. Users can't reuse code from un-batched objects, it's inconsistent with the InstancedMesh API, and still has per-object allocation overhead.

Perhaps relevant, Babylon.js' instancing API offers a choice of proxy objects and "thin" instancing (without proxies), and the latter improves performance "drastically" for large counts. I don't necessarily want to jump straight to offering two APIs, but do wonder if there are lessons learned here.

Also – the performance difference between "merged" and "naive" on this example is... not much. Perhaps because it's the same geometry? Both around 25-30 FPS on my machine. Without a better understanding of why that is, it might be better to skip all the avoidable scene graph overhead for now, until we can prove that we've got a performance win, and then make the API easier to use from there.

[drcmda]

This proxy object can be set up so that changes are automatically carried through to the merged object's combined buffers, so the developer can use the same code for both merged and unmerged cases:

why not simply using mesh and that's it? i wonder why we need proxies or object ids at all? once the meshes are added the batchedmesh instance has full access to all its children and can read out the transforms whenever it wants, or maybe i just don't understand the limitations that it would cause. but in any case, i would always prefer an api surface that can be expressed imperatively and declaratively and that is something three's own object3d/mesh does exceptionally well.

const batchedMesh = new BatchedMesh(templateGeometry, material, maxVertexCount, maxTriangleCount)

batchedMesh.add(mesh1)
batchedMesh.add(mesh2)
batchedMesh.add(mesh3)
...
function update(dt) {
  mesh1.position.set(Math.sin(performance.now() / 1000), 0, 0)
  mesh1.rotation.y += dt
}
...
batchedMesh.remove(mesh1)
...

edit:

i take it the main problem with meshes is the material. batched mesh already sets one, so repurposing mesh like that probably is unwanted. could an intermediate object maybe help? if this isn't thrown into a batch it could maybe render black or not at all.

class MeshInstance extends THREE.Object3D {
  constructor(geom) {
    super()
    this.geometry = geom
    // this.needsUpdate = false → is this the reason we're talking proxies? aren't flags a precedent?
  }
}

[jbaicoianu]

why not simply using mesh and that's it? i wonder why we need proxies or object ids at all? once the meshes are added the batchedmesh instance has full access to all its children and can read out the transforms whenever it wants, or maybe i just don't understand the limitations that it would cause

Using a proxy object (either a literal ES6 Proxy or just some intermediate with getters/setters) can help with performance. There isn't really an easy built-in way to observe an object for changes and only run some code (eg, reprojecting some vectors) for objects when they change position, orientation, scale, or any other properties that could be expressed as uniforms or attributes - using a proxy object you can present the same interface, but set dirty flags which can then be checked each frame to know whether that object needs updating in the batched mesh.

but in any case, i would always prefer an api surface that can be expressed imperatively and declaratively and that is something three's own object3d/mesh does exceptionally well.

Part of my experiment involved exposing this in a declarative way for JanusWeb, if I recall it just involved wrapping the objects I wanted to merge in a <CombinedMesh> container, and that automatically merged its children into itself as they loaded

<CombinedMesh>
  <Object src="model1" pos="1 2 3" />
  <Object src="model2" pos="0 0 1" />
  ...etc...
</CombinedMesh>

i take it the main problem with meshes is the material. batched mesh already sets one, so repurposing mesh like that probably is unwanted. could an intermediate object maybe help? if this isn't thrown into a batch it could maybe render black or not at all.

As proposed yeah, the BatchedMesh object takes a material as the argument, which would make the mesh.material property superfluous. I can imagine an additional BatchedMeshGroup object which doesn't take a material arg, and when you add a mesh (or a hierarchy of objects) to it, it could extract all the unique materials, and create a BatchedMesh object for each one.

[looeee]

Also – the performance difference between "merged" and "naive" on this example is... not much. Perhaps because it's the same geometry? Both around 25-30 FPS on my machine.

There's a big difference on my machine. When I put the slider to full it's 60fps on both instanced and merged, 16fps on naive.

[boytchev]

I'm little late to the party.

Here is a set of ideas (not necessarily mutually compatible)

• Skipping the templateGeometryand using the first added geometry as template. Then, all other missing attributes could be set to a default value, all other extra attributes will be ignored. This means, that the actual internal construction is done after the first added geometry.

  new BatchedMesh( material, maxVertexCount, maxTriangleCount );

• Skip maxVertexCount and maxTriangleCount. Then the internal construction is delayed just before the first render (after it adding new geometries should be either forbidden, or should cost regeneration of the internal structures)

  new BatchedMesh( material );

• Replace templateGeometry with an array of actual geometries, the first one is used as template.

  new BatchedMesh( arrayOfGeometries, material );

• Add a mesh (i.e. its geometry and its matrix, but not its material). Yes, I know it has been discussed in previous posts.

  const id = batchedMesh.add( mesh );

  This might be a shortcut of:

  const id = batchedMesh.add( mesh.geometry, mesh.matrixWorld );

• Normalized ids. Instead of some id, return consecutive indices - 0, 1, 2, 3, ... This means, that BatchedMesh must handle internally the mapping from indices to original ids. This will make it easy to do something like this:

  for( var i=0; i<n; i++) batchedMesh.setMatrixAt( i, otherMatrix );

• This is not an idea, just an observation. BatchedMesh is somehow the opposite of BufferGeometry.groups. The first is combining geometries and drawing them in one call; the second is splitting a geometry and drawing each part in a separate call.

• I did not understand how to access the other properties of individual geometries (like bounding box and sphere), or methods (like recalculating normals)

• Should there be a method for unbatching a batched mesh into separate meshes; or converting it to an ordinary mesh (by merging geometries)?

[donmccurdy]

Skipping the templateGeometry and using the first added geometry as template. Then, all other missing attributes could be set to a default value, all other extra attributes will be ignored.

Many vertex attributes do not have meaningful default values (e.g. UVs, normals, tangents), and because of how WebGL semantics work we can't really ignore attributes for specific parts of the batch. I'm hoping that being strict about these requirements will create fewer surprises for users than hiding them behind leaky workarounds. I'm not 100% set of having the "template geometry" argument, but we do get complaints when a subclass does not accept the same initial arguments as its parent class (geometry, material in this case).

Skip maxVertexCount and maxTriangleCount. Then the internal construction is delayed just before the first render (after it adding new geometries should be either forbidden, or should cost regeneration of the internal structures)

Note that these are maximum vertex/index counts, they don't have to be exact. I think being able to preallocate a certain amount of space for adding/removing objects in the batch — without incurring cost of re-rebuilding — is an important benefit. InstancedMesh takes a similar approach.

Normalized ids. Instead of some id, return consecutive indices - 0, 1, 2, 3, ... This means, that BatchedMesh must handle internally the mapping from indices to original ids.

Hm this is a good point. We probably do want users to be able to iterate over the objects. But we also want stable IDs, and if an object is removed from the batch, would that invalidate higher indexes? Unlike with InstancedMesh, we actually could support a sparse index list here, e.g. having objects at indexes 1-4 and 6-10, and "nothing" at index=5, could be supported if we wanted.

I did not understand how to access the other properties of individual geometries ...

I don't think we can support every feature and use case of BufferGeometry here, e.g. if you need to recompute normals it would be best to keep the original BufferGeometry around.

---

Two other thoughts. First, .add(...) is already inherited from Object3D so maybe .addGeometry(...) is better here. Second, to support dynamic updates we have to keep the original geometry in memory. That's not ideal if the user doesn't need dynamic updates. Could get around this with a static option:

batchedMesh.addGeometry( geometry, matrix, static = false ); // stores geometry
batchedMesh.addGeometry( geometry, matrix, static = true );  // does not store geometry, setMatrixAt will fail

In either case we can retain the geometry's bounding box to support raycasting.

[donmccurdy]

Alternative, index-based API:

const templateGeometry = geometries[ 0 ];

const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxIndexCount );

batchedMesh.setGeometryAt( 0, geometries[ 0 ], matrix1 );
batchedMesh.setGeometryAt( 1, geometries[ 1 ], matrix2, static = true );

console.log( batchedMesh.count ); // → 2

batchedMesh.setGeometryAt( 1, null );

console.log( batchedMesh.count ); // → 1 (trailing nulls are dropped?)

batchedMesh.setMatrixAt( 0, matrix3 );

^this could be compatible with a proxy- or handle-based API, too, but I guess I like the idea of not requiring proxies to use BatchedMesh, and/or making it possible to build that proxy interface in userland.

[boytchev]

I see.

Two questions:

1. What is the intended (expected) predominant use of BatchedMesh - for static geometries or for dynamic geometries?
2. By saying "We probably do want users to be able" ... is there any study how would users use such functionality?

The answers may give clues what features need specific attention for optimization. Personally, I see myself using BatchedMesh for a project with hundreds of rooms in dungeons (https://boytchev.github.io/meiro/) - the rooms the same material, but are geometrically different, they are static, and having them batched will benefit from furstum culling.

[donmccurdy]

What is the intended (expected) predominant use of BatchedMesh - for static geometries or for dynamic geometries?

Both are important; I wouldn't want to pick just one. But BufferGeometryUtils.mergeBufferGeometries(...) works reasonably well for static geometry today, whereas it doesn't work at all for dynamic geometry, so perhaps that's the bigger limitation in current support.

By saying "We probably do want users to be able" ... is there any study how would users use such functionality?

Being able to loop over the batch later seems fundamental for a container API, e.g. for raycasting and frustum culling. This thread is intended to gather feedback on what users need from the API.

[gkjohnson]

I really like this addition. I'd hacked together a similar helper awhile ago for a prototype that used mesh skinning to get this type of batching with dynamic transforms:

https://twitter.com/garrettkjohnson/status/1390406757134917638

On my use case a bit -- the robotics models I was loading were deeply nested with a lot of rigid meshes that shared a few materials and the number of draw calls was taxing the framerate in VR. I had already written the code for all the raycast interactions so I wanted something that would "just work" and be a drop-in replacement without other changes. The approach used skinning, bone weights, and a similar "proxy" approach for raycasting and transform updating

I think everything I described above or has been described regarding "proxies" can be implemented on top of the originally proposed API and more. I like the approach of manually adding geometry for three.js core because it's simple, flexible, and doesn't impose any transform update overhead (matrix world updates and batched transform assignment). The Proxy meshes approach could make a nice example class, though, for those that want that convenience.

Hm this is a good point. We probably do want users to be able to iterate over the objects. But we also want stable IDs, and if an object is removed from the batch, would that invalidate higher indexes?

I think a sparse index makes sense. I'm imagining a mapping from id -> position in index array (and length of geometry) so it can be removed or updated easily.

Second, to support dynamic updates we have to keep the original geometry in memory. That's not ideal if the user doesn't need dynamic updates. Could get around this with a static option:

What's your reasoning for making the BatchedMesh keep a handle to the original geometry? I would expect something like an updateGeometry( id, geometry ) function that sets the previously stored index position and range of vertices to the new setting. It feels more natural for the user to keep a hold of the geometry if they expect to update it.

Definitely looking forward to this!

[donmccurdy]

What's your reasoning for making the BatchedMesh keep a handle to the original geometry?

I was thinking of the setMatrixAt( i, matrix ) function (similar to InstancedMesh), and needing to overwrite vertex positions... but preferably not overwrite every attribute of the mesh. With WEBGL_multi_draw (or the skinning-based approach you mention?) overwriting vertex data wouldn't be necessary, though.

[gkjohnson]

Oh I see I misunderstood the first few posts in the thread -- I didn't realize you were originally intending to transform every vertex on the CPU. I think doing the vertex transformations in a shader would be a good approach. If I were making a more purpose-built, from scratch implementation of my prototype above I wouldn't use skinning because it involves multiple weights per vertex and therefore more transformations. In theory without the mult_draw extension you'd only need a single new vertex attribute storing the geometry index (just another 1 byte per vertex for up to 256 geometries?) which would be used to access a matrix4 uniform to rigidly transform the geometry.

If I understand correctly, dynamic batching in popular game engines will not do this, but instead rewrites vertices on the CPU, I'm not sure why that would be if this extension is widely available. 🤔

I've heard of this, too, and it's not clear why something like the skinning or shader-transform approach wouldn't be preferred even if the extension is available. It's hard to find a lot about dynamic batching online that doesn't just discuss Unity's approach at a high level but I did find this GPU Gems 2 article on it that touches on a few different batching approaches and mentions it but is fairly dated (2005). And from Godot's documention on batching and vertex baking it seems they "bake" vertex transforms and colors to avoid per-instance parameters. Perhaps it's worth asking Juan?

[gkjohnson]

I went ahead and messaged Juan from Godot and Aras from Unity to get their take on the modern benefits of baking vertices for dynamic batching and bottom line is it doesn't sound like something we should be using.

For Unity it sounds like the dynamic batching with vertex baking was designed for use on platforms that don't support shaders. And for Godot apparently that style of vertex baking is only done for 2D rendering because it affords some other tricks in that scenario. Juan also mentioned mobile limitations (max UBO size, texture fetch performance) and a special case in tile-based deferred rendering (which we're not using) that might crop up with the uniform matrix4x4 array or texture approach but considering we're already using data textures for skinning I'd think that's what we should go with unless we find that the uniform limitation wouldn't be a problem.

Lastly given that here are a few more thoughts on a few of the API functions:

class BatchedMesh extends Mesh {

    // no template geometry needed? Derive the required attributes from the first geometry
    // added which cannot be changed after
    constructor( material, maxVertexCount, maxTriangleCount );

    // include offset and count so only a subset of the geometry can be added (a la geometry groups
    // for cases where only a piece of the geometry is rendered with the appropriate material)
    addGeometry( geometry, offset = 0, count = Infinity ) : id;
    updateGeometry( id, geometry, offset = 0, count = Infinity );
    removeGeometry( id );

    setMatrixAt( id, matrix );

    // I'm not exactly sure how visibility would work without the multi draw arrays extension. Perhaps
    // all vertices for that geometry are collapsed to a single point outside of the screen clip?
    setVisibilityAt( id, visible );

}

[donmccurdy]

Thanks @gkjohnson! I'm happy to take their word for it and skip vertex baking, or at most consider it a short-term solution. I think you're suggesting an N * 16 data texture or UBO, with object IDs in a vertex attribute to perform lookups? Or more directly reusing the skinning system here?

I'm having trouble finding numbers on real UBO sizes in WebGL. Assuming the minimum is similar to OpenGL's 16kb, we have an upper bound of something like 250 objects per batch with float32 storage, which may be a plus for data textures here.

include offset and count so only a subset of the geometry can be added...

Could also support this by respecting the draw range on the given geometry. 👍

I'm not exactly sure how visibility would work without the multi draw arrays extension...

Might be cheap enough to rewrite or rearrange the index buffer for visibility changes, no need to change the vertex data.

[gkjohnson]

I think you're suggesting an N * 16 data texture or UBO, with object IDs in a vertex attribute to perform lookups? Or more directly reusing the skinning system here?

I think a good short term / simplest implementation of the class as we've described would just be done on top of SkinnedMesh with BatchedMesh extending that class. There actually wouldn't be any core changes to do this. The BatchedMesh class would just internally maintain skinned geometry with weights, skeleton, and bones. SkinnedMesh uses DataTextures by default and falls back to mat4 uniforms if the system doesn't allow floating point textures.

The reason to use a purpose-built implementation instead of just using a skinned mesh would be for shader complexity and memory improvements. For skinning 4 bone weights and 4 bone indices are specified per vertex and the shader has to do 16 texture lookups to get all the matrices (4 pixels per matrix). A more optimal implementation for a BatchedMesh class would only require 1 index per vertex in an attribute, no weight attribute, and only 4 texture samples in the shader.

I'm having trouble finding numbers on real UBO sizes in WebGL. Assuming the minimum is similar to OpenGL's 16kb, we have an upper bound of something like 250 objects per batch with float32 storage, which may be a plus for data textures here.

In my opinion if the project has chosen DataTextures as the solution for skinning it makes sense to do the same here.

Could also support this by respecting the draw range on the given geometry. 👍

That's true and maybe that should be the default but I'm imagining the use case I had above where multiple meshes had geometry groups and some groups between meshes shared a material. If you needed to adjust draw range to only stuff in a certain part of the geometry that would impose a pattern like this:

const group = geometry.groups[ index ]; 
const initialDrawRange = { ...geometry.drawRange };
geometry.setDrawRange( group.start, group.count );

batchedMesh.addGeometry( geometry );

geometry.setDrawRange( initialDrawRange.start, initialDrawRange.count );

Might be cheap enough to rewrite or rearrange the index buffer for visibility changes, no need to change the vertex data.

Yeah I'd think that would be pretty fast enough! I'm not exactly clear on how we'd easily fall back to that if multi draw arrays extension is available in the long term but it's probably not worth worrying about now

[takahirox]

I would like to vote for DataTexture. In addition to what you folks mentioned above, Three.js doesn't support UBO yet and UBO is only for WebGL2. We may be able to revisit the choice later if we encounter texture fetch or upload performance issue.

I started to make a prototype of mainly the renderer and shaders with this proposal + WEBGL_multi_draw because I'm interested in WEBGL_multi_draw extension performance and I want to know how likely the change in the renderer and shaders will be (and I want to know if @mrdoob accepts it). Let me know if someone already started the implementation because I don't want to be conflicted.

[takahirox]

Made a prototype.

Branch: https://github.com/mrdoob/three.js/compare/dev...takahirox:BatchedMesh?expand=1

Online demo: https://raw.githack.com/takahirox/three.js/BatchedMeshTest/examples/index.html#webgl_multidraw

https://user-images.githubusercontent.com/7637832/132141130-f84d2a3d-97d1-4c3f-b88b-46f0dba8ae65.mp4

40 fps (BatchedMesh) vs 30 fps (regular Mesh) on my Windows 10 Chrome.

[drcmda]

would it be much trouble to add multi material support like @takahirox indicated? i think multiple geometries sharing the same material is quite unusual. in pretty much 100% of all cases that i've worked on (assemblies for instance), geometries have distinct materials, and that would then allow us to deal with instanced GLTF.

[gkjohnson]

would it be much trouble to add multi material support like @takahirox indicated? i think multiple geometries sharing the same material is quite unusual

BatchedMesh will only have performance benefits if all "sub meshes" share a single shader / material just like InstancedMesh. If you want to batch multiple meshes that share a shader but have different material properties you'll have to do some preprocessing to use things like vertex colors in place of diffuse material color and / or use a texture atlas which would also require adjusting geometry UVs. With more shader changes you could index into an array uniform for individual sub mesh surface properties (roughness, color, textures, etc).

The multi draw extension provides a gl_DrawID in the shader to identify which geometry is being rendered and can therefore be used to determine which surface properties to use which is nice but if we can't use multi draw then vertex properties should be able to enable that.

[drcmda]

gl_DrawID sounded like the golden goose for a moment. thanks for clearing that up! with how powerful materials in threejs have gotten we can still probably get a lot out of it with vertex props.

[manthrax]

This looks awesome!!

[hschwane]

Hi all, I am relatively new to Webgl/Three.js but was working with desktop OpenGL before. I was a little surprised that Three.js does not support batching already. This looks great and my current project would benefit from this for sure. I am looking forward to this feature a lot. After reading through the thread I also have some ideas I wanted to share:

BatchedMesh will only have performance benefits if all "sub meshes" share a single shader / material

As far as I understand it would be possible to render different materials in one batch, as long as they use the same shader. Using vertex attributes would be one way to do it, but it would also be possible to write the material settings into a UBO or a data texture and and access it with a material id. The material id would be stored in a vertex attribute, or together with the transformations matrices for the model.

If I understand correctly, In the current proposal the user would be required to manually set up the BatchedMesh and then add the corresponding pseudo-objects to the scene graph to track Transformation. Would it be possible to add change tracking to the existing THREE.Scene? When a new mesh is added to the scene it could then automatically check if it can be added to an existing batch, or create a new batch. When transformations or materials change the scene would also detect it and set a "changed"-flag or store the changed object in some other way. On rendering we update the buffers and data textures that were changed, and then render the batches. I understand that would be a lot more work and seems like "version 2" of the batching. But it would also allow the batching to work automatically in existing code that uses three.js with minimal changes.

[gkjohnson]

Would it be possible to add change tracking to the existing THREE.Scene? When a new mesh is added to the scene it could then automatically check if it can be added to an existing batch, or create a new batch.

I think this will be best managed explicitly through user code. Expecting the Scene / Renderer to track changes or find batchable meshes in a scene will add complexity and unnecessary performance overhead. It also won't work magically in all but the simplest cases. ShaderMaterials or "onBeforeCompile"-modified shaders cannot necessarily be guaranteed to work correctly.

My feeling is that if you'd like a "smart" batching system that it would be very possibly to build one as an extension or wrapper on top of the three.js API.

[hschwane]

Yes I understand that would be a lot of extra work and might not work correctly with other already existing three.js features. Was just an idea I wanted to add to the list.

The explicit batching will already be a great help in improving performance. And everything beyond that could be implemented as a layer on top by those who need it, as you mentioned.

[natarius]

Super stoked about this!

I was just investigating building something like this myself…so super stoked we are gonna get official support!

[jonas-db]

I came across this proposal because I was searching for something similar to InstancedMeshGroup: It seems not possible to use the current InstancedMesh when you have 1 object/group that you want to duplicate a lot but it has multiple geometries and/or materials. Might be a use case for this one?

[vegarringdal]

Hi how is this going ? Looks like a very nice feature. Would we be able to have different colors on the elements? Atm I have some cases where it would have been useful to merge with different colors without using vertex coloring, since Im already loading to big models 😁😂 Also being able to add/remove parts and not get more drawcalls sounds nice.

[vegarringdal]

This probably could be used a LOD ish option too. On large scenes where you have a lot of object, instead of merging all, you could filter out/remove some when far away. Like all small objects, and just keep the large ones so user can navigate better/know where they are.

adding/removing small parts would probably work a lot faster then rebuilding large merged meshes

[arpu]

would it not better to use Occlusion Culling? https://tsherif.github.io/webgl2examples/occlusion.html

[vegarringdal]

Occlusion culling is a nice feature too, Ive played around a lot with it. But mostly just hiding/setting not visible. Think batched mesh is still useful in this combination, since you will be unloading/loading small parts instead of large geometry buffers (normals/position/index) instead of large merged mesh.

From what I can see I need to split my scene into a 3d tile bounding box system, and load/unload depending on where camera is and what its able to see (occlusion culling)

There is a LOT I dont know 3d/webgl/threejs so I might be wrong, but i still believe batch mesh would be very useful for threejs. Ofc webgpu will help in the future, but its not stable, and might have its own issues first years. Who knows, maybe not everyone can use it.

Im sure more experienced 3d artist like @gkjohnson @takahirox @donmccurdy could give more insight on why or why not batched is a good fit or not for threejs.

[takahirox]

I have been thinking if the BatchedMesh API will greatly fit to or can improve for WebGPU API. And I just didn't have time to look into it deeply yet.

[RenaudRohlinger]

Hello. I tried to implement BatchedMesh so here are a few pieces of information from my own experience that could be useful:

• Multiple materials per geometry through Uniforms Array [not recommended]:

The limit size of the Uniforms Array Size is known by Max Fragment Uniform Vectors and Max Vertex Uniform Vectors. https://webglreport.com/?v=2

Overall it should be 1024, also it's important to know that a bug was introduced in the latest version of Safari that reduces this limitation to 256, and that will be fixed in the next patch: https://bugs.webkit.org/show_bug.cgi?id=238914

From my benchmark, Uniforms Array is by far the worst in performance. FPS drops by 50% or so vs Naive implementation (especially on windows). Also, it will restrict the number of instances by the max size of Uniforms Array Size (can be under 1024).

• Datatexture for other material properties: I would recommend DataTexture for uniform per-geometry. Then access the information in the shaders via gl_DrawID. I think it could be worth investigating a custom API that make uniform accessible via Datatexture, something like setUniformAt() or getUniformAt().

From my tests, datatexture is pretty solid and does not affect so much the performances.

Also, it's worth mentioning that the multi_draw api is now supported on Safari. They just introduced a bug in the latest version of Safari but a patch has already been made and should be launch in the next version of Safari

[takahirox]

@mrdoob @Mugen87 Are you interested in BatchedMesh API? Or should we wait for WebGPURenderer (and node based material system)?

[vegarringdal]

☝️ @mrdoob @Mugen87 Have you guys given @takahirox question any thoughts ?

WebGL 2.0 is still getting improved browser support and be around for a long time. So might be useful addition to apps currently made/in development ?

[agviegas]

Hi all! The devs of IFC.js here.

We would benefit greatly from this feature and are willing to invest resources into this. I don't know the state and how much it would take to finish this, but if one/many people are willing to get paid for getting this done, you can count on us to finance it. 🙂

[VegarRingdalAibel]

☝️ @takahirox have you considered what @agviegas said ? From what I could see in some earlier posts you clearly know what you are doing when I look at the prototype :-) Would be a nice addition to threejs when it comes to working with large static models like huge buildings/HVDC rigs/oil rigs 😄

[takahirox]

Yes, I'm interested in but also have some concerns.

• It takes effort to implement the feature. So I want to avoid a scenario that it's refused after implementing. I hope we can start after confirming that at least @mrdoob and @Mugen87 are interested in this feature. (Probably we may need to clarify the changes and benefits more for them for easier disicion.)
• We may be able to start with implementing it under examples not to pollite the core as Don's first suggestion if adding changes for the feature in the core is refused.
• I think material parameters per geometry support is needed. (Personally I think the feature is a bit useless without supporting it.) But supporting it on the current API is not easy without the big change in the core. (Correct me if I'm wrong.) It might be implemented more easily with node based material system. So I might want to wait for WebGPU node based material system officially lands and is matured.

[gkjohnson]

I think material parameters per geometry support is needed. (Personally I think the feature is a bit useless without supporting it.) But supporting it on the current API is not easy without the big change in the core. (Correct me if I'm wrong.)

I strongly disagree with this. If people need different per-object material properties there are a variety of workaround with texture atlases and material property maps but even without that this is an incredibly valuable feature that will help cut down on draw calls significantly which is a huge bottleneck in a lot of applications. Even InstancedMesh only supports a color property. It would be more than enough to just support that, I think

That aside I don't think the first implementation of this feature needs to be fully robust. There's a lot of value without material properties and It will be valuable to see how it's used in practice. Then it can be expanded on in the future.

As I've mentioned above I've made a hacky version of this using skinned meshes (see here) and used the same technique to significantly improve the draw call bottleneck in the VR demo for closed-chain-ik by reducing the complex robot model draw calls to 1.

It takes effort to implement the feature. So I want to avoid a scenario that it's refused after implementing. I hope we can start after confirming that at least @mrdoob and @Mugen87 are interested in this feature.

I sympathize with this. It would be great to make sure the project is on board with the feature - as you can tell from the comments and reactions on the topic it's a very desired feature.

[VegarRingdalAibel]

@gkjohnson @takahirox

I think of this as a improvement to merging everything per color like I do now.

Since you guys know a lot, maybe you can help with some questions if you have time. Sorry of some of the questions are weird. 😂

1: Will we be able to improve frustrum culling in any way with this ? Maybe by moving/sorting? This is something I notice I loose when I merge too much on large buildings on a project Im working on.

2: If I have 50 batched meshes with 1000 items in each, where every batched mesh have different color. Will it be easy/possible to move one of the items between the batched meshes to change color? This is also something that can be hard when everything is merged.

[donmccurdy]

I think material parameters per geometry support is needed.

I strongly disagree with this. If people need different per-object material properties there are a variety of workaround with texture atlases and material property maps ...

I'd prefer that we think of these as two different APIs:

1. An API for non-destructively building, batching, and updating geometry composed of many static or dynamic parts
2. An API for taking many distinct materials and distilling them to a single shader

The second problem (2) is really hard, and I don't see appealing shortcuts. I don't know if it should be done at runtime, or in offline tooling where texture atlasing is more practical. We're probably a very long way from putting anything like that into three.js core. It could be built as another layer on top of (1).

So I'd prefer to see the geometry API, (1), as its own project. Anything else that can consolidate materials would be a complementary project.

If we are just dealing with the geometry API here, changes to three.js core may not be necessary? But we can certainly discuss them if so.

[natarius]

Hi all! The devs of IFC.js here.

We would benefit greatly from this feature and are willing to invest resources into this. I don't know the state and how much it would take to finish this, but if one/many people are willing to get paid for getting this done, you can count on us to finance it. 🙂

Dev from flux.ai here...we would also be happy to chip in here

[takahirox]

I strongly disagree with this. If people need different per-object material properties there are a variety of workaround with texture atlases and material property maps but even without that this is an incredibly valuable feature that will help cut down on draw calls significantly which is a huge bottleneck in a lot of applications.

OK, that makes sense to me. Let me withdraw what I mentioned. BatchedMesh looks valuable even without per-oebject material properties support. Per-object material properties may make the feature more useful, but perhaps the support doesn't need to be first priority.

[Mugen87]

It takes effort to implement the feature. So I want to avoid a scenario that it's refused after implementing. I hope we can start after confirming that at least @mrdoob and @Mugen87 are interested in this feature. (Probably we may need to clarify the changes and benefits more for them for easier disicion.)

To make it official, yes, having BatchedMesh would be great :blush:.

We may be able to start with implementing it under examples not to pollite the core as Don's first suggestion if adding changes for the feature in the core is refused.

Definitely. AFAICT, it would be good to have an initial version of BatchedMesh that does not require changes to the core. So it just manages the geometry data maybe via onBeforeRender() or an explicit update method. When we have something working for a couple or releases, we can think of an integration in the core.

I think material parameters per geometry support is needed. (Personally I think the feature is a bit useless without supporting it.) But supporting it on the current API is not easy without the big change in the core. (Correct me if I'm wrong.) It might be implemented more easily with node based material system. So I might want to wait for WebGPU node based material system officially lands and is matured.

As mentioned in https://github.com/mrdoob/three.js/issues/22376#issuecomment-1205480651, I would not worry about materials for now. BatchedMesh should start as a pure geometry API and the requirement is to have a single material. The primary purpose should be the better handling of dynamic geometry data.

[takahirox]

So I'd prefer to see the geometry API, (1), as its own project.

Ah, cool. If per-object material properties support is out of the scope, the idea can be geometry API, not Mesh. It would be simpler.

If we are just dealing with the geometry API here, changes to three.js core may not be necessary? But we can certainly discuss them if so.

Probably, it may be possible without Three.js core change. But there may be some optimization limitations. For example, matrices may need to be applied to vertices in CPU, not GPU. (It might be possible by using some hooks to edit shader codes and to update matrix data texture?) and WebGL_multi_draw can not be used.

If we add changes to the Three.js core, it can be more optimized. But we may need to add a new geometry class or properties then the core can be a bit more complex. (But not so much according to my prototype?)

If we have extra energy, I hope we can compare the performance between with the Three.js core change and without the Three.js core change.

[Mugen87]

When we have agreed on a geometry batching API, we can still integrate it later in the core with additional benefits. I just think it's easier to get something started if we don't have to touch the renderer for now.