BatchedMesh: Proposal

[donmccurdy]

Currently the APIs available for reducing draw calls include:

• InstancedMesh
• BufferGeometryUtils.mergeBufferGeometries([a, b, c, ...])

I think we might be able to solve some common difficulties with reducing draw calls, even where the objects to be rendered are not instances of the same thing. I've floated the idea in a couple of earlier comments (https://github.com/mrdoob/three.js/issues/19164#issuecomment-631237304, https://github.com/mrdoob/three.js/issues/18918#issuecomment-602395984) but wanted to open a separate issue for it.

Calling this BatchedMesh for now, it's meant to be a way of (1) merging, (2) rendering, and (3) updating a group of objects that would otherwise be drawn as separate Mesh instances. The objects must all be the same primitive type (i.e. triangles), must share a material, and must have compatible vertex attributes (e.g. if one has normals, they all have normals). With those requirements we could offer an API like:

// Create BatchedMesh with max vertex and triangle counts to support. If 
// geometry is not indexed, maxTriangleCount is optional. The 'template'
// geometry could just be the first geometry we'll add later — it isn't stored,
// but is used to determine which vertex attributes to allocate.
const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxTriangleCount );

// Append geometry, up to max vertex and triangle limit.
const id1 = batchedMesh.add( geometry1, matrix1 );
const id2 = batchedMesh.add( geometry2, matrix2 );
const id3 = batchedMesh.add( geometry3, matrix3 );

scene.add( batchedMesh );

renderer.render( scene, camera );

// Update object position dynamically. For small objects this may be cheaper than
// a separate draw call. For large objects, it may be better to draw the object
// separately than to update it with dynamic batching.
batchedMesh.setMatrixAt( id1, otherMatrix );

renderer.render( scene, camera );

This offers a few advantages over mergeBufferGeometries(). First, you get an ID that lets you remove or modify each geometry later. Second, we can potentially do much faster raycasting (we know the bounding box and transform of each member geometry) and return that ID as a result. Third, we can do frustum culling (maybe?) by updating the index buffer.

[takahirox]

OK, starting as geometry API without the changes in the core sounds like a right direction. Anyone wants to implement it? If noone wants, I can take. But if someone wants, please do. And I can help if needed.

(By the way, even if I implement and someone pays for it, I would like to share it among people who joined the discussion.)

[agviegas]

Hey @takahirox sure, all the development we have financed is open source and for everyone 👍 We were thinking of a $1000 bounty, but we are also open to discussing this. FYI we are handling our funds openly through OpenCollective, where we have been funding features like this for quite some time. In case anyone's curious, here you can find more info about us spending money on open source.

[natarius]

We were thinking of a $1000 bounty

Happy to match that!

We also manage our contributions via OpenCollective

[jeybee]

This is not intended to be ready for adding to the repo, but I've modified takahirox's demo so it doesn't rely on WebGL_multi_draw and have been having good results. Here's a gist that may help anyone looking to use something similar before it is properly implemented: https://gist.github.com/jeybee/52b660b3ed4e91c26314cb52857a9204

[vegarringdal]

@takahirox Hi, hope you have had a great summer. I know I might be asking a bit early, but have you had a chance to think about how you want to do it ?

Figured I could try and share what I hope WebGL_Multi_draw can help with. Maybe this can be some help on why Im active on this thread 😄

Ref https://registry.khronos.org/:

CAD vendors rendering large models comprised of many individual parts face scalability issues issuing large numbers of draw calls from WebGL. This extension reduces draw call overhead by allowing better batching.

At work I try and render large modules for offshore, some of these contains really many parts.

On the right hardware loading all goes very well, but on standard hardware (default laptops/small laptops/pads) it all slows down if i dont load smaller parts of modules. This is OK for some parts, but not the user experience always since many times people want also piping going between/over several modules, any might want to see all to get a good overview of scope of work/issues.

Ive been experimenting by hiding all but structure/architecture when moving and it was very successful when it comes to fps. But for user its kinda annoying when all details get hidden close to camera when they move around, even though it pops in/out very quickly.

So what I wanted to try is to try and split the entire model into smaller cubes/meshes per color (merged), where each cube will have 2 parts, very small items, and medium. (and some logic for items between in own cubes/larges parts etc.. would need some trying and failing ).

So with this I could try hide/show depending on how close camera is to the bounding box of the cube, so far away I hide both, then medium when I get closer and in the end the small items. Also hoping this would help with frustum culling witch is really bad when I merge to much.

So without this extension I would get A LOT of draw calls. 😂

and no instance will not help enough, to many individual parts, some parts is geometry only from vendors with little details to use instancing on.

What do you think ? Would something like this work ? Will there be any limit to how many meshes per batch? Would it be possible to move geometry between the batches (if I wanted to edit color, asume we get 1 material per batch) Also thinking since we work with smaller meshes, unloading and reloading might also be a option since we can deal with smaller part

If what I want to do is possible by using WebGL_Multi_draw then this will be a huge improvement for threejs and CAD model rendering.

[vegarringdal]

Finally managed to get a model I could show video of project Im using threejs in.

https://www.youtube.com/watch?v=FzTHbogfr5k

A lot of coloring is done using groups/material index atm. Really hope that feature does not get dropped in webgpu renderer, was a lot more powerful then I thought. But still think WebGL_Multi_draw will give me even more boost. Would it maybe be possible to batch different group drawcalls into WebGL_Multi_draw ?

[takahirox]

I made a draft PR #25050 for the geometry addon approach (BatchedGeometry) that doesn't need any change in the core.

Demo is here https://raw.githack.com/takahirox/three.js/BatchedGeometry/examples/index.html#webgl_geometry_batch

But unfortunately BatchedGeometry is slow. I haven't taken the profile yet, but probably positions and normals recalculation for matrix update and/or attributes upload are slow.

Perhaps applying matrices should be done in the shader (GPU) for the performance. Then, the mesh addon approach (BatchedMesh) would be better that would customize the vertex shader with onBeforeCompile() hook and would upload matrices data with onBeforeRender() hook than BatchedGeometry that recalculates positions/normals in CPU. I will investigate more.

Regarding WEBGL_multi_draw, I think it improves the performance (a lot?) but we may want to start with addons that doesn't change the core https://github.com/mrdoob/three.js/issues/22376#issuecomment-1206253835 To use WEBGL_multi_draw we need the change in the core.

[VegarRingdalAibel]

Yes notice it s was a lot slower. Im thinking go for core changes, make threejs even better. Since someone offered bounty. 💰 Why make an addon no one will use, because it kills performance?

Why not just start with WEBGL_multi_draw in core/or adjust core to be more flexible to support something like this as an addon.

Yes, I know it is not simple/or even possible, and there is A LOT I do not know and never will learn. Just want it if it can help performance

@takahirox, thanks for taking the time to make the demo. 👍

Got a question to you if you have time, WEBGL_multi_draw will this be able to support groups/material index for coloring on material?

[takahirox]

I made a PR #25059 for BatchedMesh addon that uses data texture and custom shader to handle local matrices in the shader (GPU).

On my Windows 10 + Chrome, the performance is good so far.

[agviegas]

Hey, @takahirox this is fantastic. Feel free to create a $1K expense in our OpenCollective for the work you have done so far. We are very eager to finance any other improvements built on top of this, even with more significant bounties if the tasks ahead require more time/knowledge. We'll test this out right away!

[agviegas]

I couldn't resist and tried using this with a whole building. Memory and fps seem nice so far, and the API is quite easy to use. In this case, I used 2 batchedmeshes: one for transparent and other for opaque objects 🙂

Just for the record:

• They raycaster doesn't seem to work properly with this.
• This might have some problems when used with postproduction, probably due to the shaders. For instance, in the image below there's a postproduction effect applied that draws some lines on top of the geometry. As you can see, the postproduction is not reading the batched mesh logic, so it just draws the "raw" geometries (only one instance of each geometry, and without transformation).

[gkjohnson]

Really happy to see some progress on this with #25059. I have a few other use cases that I'd like to see covered that I'll list here and if I have a chance I'll make a PR for some of them.

Batched Tiles Rendering

The 3DTilesRenderer for the 3d tiles spec dynamically loads and unloads tile geometry in the scene based on hierarchical frustum culling and error metrics. Depending on the structure of the tile set this can lead to hundreds or thousands of meshes and geometries in the scene. Ideally the loaded geometry could be added and removed from a BatchedMesh instance with textures being rendered via texture array so the whole tile set could be rendered in a single draw call.

To achieve this we'd want to be able to replace existing geometry at an index in the BatchedMesh as well as pre-allocate a certain buffer size per batched id so that larger geometry can fit later when it's replaced.

Shared Geometry Buffer for Material Swapping

Related to https://github.com/gkjohnson/three-mesh-bvh/issues/513#issuecomment-1767965033 - with a BatchedMesh all underlying geometry must be rendered with a common material. However you may have cases where a user can highlight or select a material to become transparent which will require different material and render state settings. Rather than copying and uploading new geometry data it would be optimal to reuse the same underlying geometry (and possibly matrix buffer) for two BatchedMeshes so a transparency-toggled geometry can be hidden in one and shown in another with minimal overhead.

Possible Changes Required

• Add setGeometryAt( geometry, index ) function for changing underlying geometry.
• Add addGeometry( geometry, length = - 1 ) function for reserving a underlying buffer amount.
• Add ability to share underlying geometry - this may already be possible, though unofficially.

For both of these use cases (and others), of course, the WEBGL_multi_draw extension would help improve performance significantly at least in cases where it can help avoid unnecessary vertex transformations of non-visible geometry.

[gkjohnson]

For those interested - I made a PR (#27078) that addresses the use cases above for "Batched Tile Rendering" with setGeometryAt and an addGeometry function. You can see the demo here (and repo here) that demonstrates rendering dynamically loaded and unloaded tiles and texture using just a single draw call.

I still need to think through the second use case of sharing geometry & possibly transforms between two BatchedMeshes to more easily enable toggling between materials. I'm thinking it may be best to add a BatchGeometry class that can be shared between multiple BatchedMesh instances.

Otherwise if everything else looks good and the project is on-board I'll take a look at adding support for WEBGL_multi_draw into the renderer in addition to support for other basics like raycasting.

[gkjohnson]

After almost 30 PRs we have a fully featured BatchedMesh class in three.js core! The class includes:

• raycasting support
• per-object frustum culling
• per-object sorting for overdraw mitigation and transparent object rendering
• built-in material support
• proper per-object visibility toggle that skips rendering of the object entirely
• support for the WEBGL_multi_draw extension (to enable sorting, visibility toggle, frustum culling)
• a (less efficient) fallback for non-WEBGL_multi_draw-supporting browsers
• other expected built-in three.js features

So I think this can be closed. Looking forward to seeing what kinds of performance improvements this brings to projects!

Thanks to @takahirox and @donmccurdy for getting it started!

[Amatewasu]

Wow, it looks wonderful! 😍

Do you have a link to the documentation? I struggle to find it.

[gkjohnson]

This is not released yet - BatchedMesh and the docs will be available in r159.

[bhouston]

Interesting to see this happen! Would love to see some documentation. 10 years ago almost to the day I did suggest a batching system for ThreeJS here: https://github.com/mrdoob/three.js/issues/4221#issuecomment-31231588. But I am not completely sure how it compares to what was delivered in these PRs.

[gkjohnson]

If you want a sneak peak at the docs they can see them via githack. There are some other updates in #27231, though:

https://raw.githack.com/mrdoob/three.js/dev/docs/index.html?q=Batched#api/en/objects/BatchedMesh

But I am not completely sure how it compares to what was delivered in these PRs.

Heh it's hard for me to tell, I think - three.js has changed a lot in those 10 years it looks like 😅

[enruilo1]

Hi, thanks for such feature, its so incredible... i tried in r158 version and get some issues like there is no raycasting, no reaction to shadows and ambient occlusion samples in doc page. I think is normal no? because i have watch improvements in batched meshes for r159. I also notice that bounding box of scene it is not calculate like with instanceMesh or normal meshes... Bbox of scene with instanceMesh. Bbox of scene with BachedMesh.

Perhaps @gkjohnson can iluminate me a little. PD: I read in another issue the possibility of reuse geometry data inside of BatchedMeshes...it is possible? Because it delete the need of use InstanceMeshes in my app and only use BatchedMeshes because (for me) there is not any other advantage of Instance vs Batched

[gkjohnson]

Please try the implementation in the next release and ask at the forum if you have further questions about InstancedMesh vs BatchedMesh.

[vegarringdal]

Maybe nice to know about for later: multi draw indirect

[gfodor]

thank you for this, this is fantastic work!