Add a DynamicMesh resource for frequent partial Mesh updates

[clayjohn]

Describe the project you are working on

I work on Godot itself, but am making this proposal on behalf of others

Describe the problem or limitation you are having in your project

Users often want to do some CPU-side vertex processing of a mesh. This can include:

1. Procedurally created geometry (stuff like this https://x.com/SamuelLundsten/status/1770019564920713291 or this https://www.youtube.com/watch?v=X574IIBgOko)
2. Baking new meshes (for example, for a character creator)
3. For advanced VFX
4. For sampling mesh information (dynamic painting etc.)

There are two big problems with the current workflow:

1. Godot does not cache the mesh arrays and when a user retrieves them, they have to be copied from the GPU, causing significant stalls
2. Doing partial updates using mesh_surface_update_***_region() is really tricky and requires intimate knowledge of our internal GPU representations of meshes

Right now for anything requiring reading mesh data on the CPU, the best option is to read the data from the GPU once and then cache the arrays yourself. This is a very awkward practice and many users stumble over it.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I suggest that we add a new Mesh type called DynamicMesh. DynamicMesh would cache the mesh arrays on the CPU and allow for seamless partial updates to the GPU.

Further, we would allow meshes to be imported as DynamicMeshes instead of ArrayMeshes so that the array data never has to be read back from the GPU if you know you are going to access it.

DynamicMesh would provide easy access to vertex data on the CPU and an easy way to update mesh data dynamically. The tradeoff will be:

1. Duplication of vertex data in RAM and VRAM (ArrayMesh is only in VRAM), so it will cost a bit of RAM
2. DynamicMesh can't use the compressed format. So rendering performance may be reduced

This proposal will resolve points 1-3 above (procedural geomety, mesh baking, VFX), but not point 4 (sampling mesh information). While it could be useful for sampling mesh information, it is overkill, and would result in a needless waste of RAM for users who just want to sample UVs or normals on a surface. For 4, we will need another proposal to expand on the existing TriangleMesh (expose it to scripting and add optional support for UVs, bones, etc.)

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

DynamicMesh will look similar to ArrayMesh and ImporterMesh except it will have an array of SurfaceData inside it that stores the surface information. DynamicMesh will complement the existing mesh types: ArrayMesh: Highest performance, used for imported resources by default. Only stores data on GPU and prefers compressed format ImporterMesh: CPU-only representation of mesh. Suitable for offline mesh manipulation like we do in the importer DynamicMesh: CPU and GPU representation. GPU representation automatically updates when CPU is changed.

We can track chunks of memory with a bitmask and only update the needed parts of the GPU Representation. Further, we can allow the user to write directly to the arrays to make it very convenient to use e.g. (syntax to be determined):

for i in range(vertex_array_len):
    dynamic_mesh.surface[0].arrays[ARRAY_VERTEX][i] = some_vec3

In other words, there is no need to copy the entire surface array, write to it, and the recreate the entire surface. This should drastically reduce the cost of mesh updates.

To the importer options, we will add a checkbox called use DynamicMesh which will then import the mesh as a DynamicMesh instead of an ArrayMesh. This will allow users to immediately modify or read data from imported meshes without having to do a round trip to the GPU

Open questions:

1. Syntax for writing to the vertex arrays (maybe a function style is better e.g. dynamic_mesh.surface_write_to_array(0, ARRAY_VERTEX, i, some_vec3)
2. How will this interact with eventually writing to GPU buffers with compute shaders? If the GPU representation is updated, should we / can we invalidate the CPU representation? Or is it preferable to ban GPU updates somehow?
3. Should we add conversion functions between Mesh types to support this?
4. How will this interact with the GPU? We don't want to accidentally attempt to write to surface data in frame N+1 while frame N is still rendering.

If this enhancement will not be used often, can it be worked around with a few lines of script?

It can't be worked around in script as Godot discards the CPU representation of a mesh as soon as it is loaded

Is there a reason why this should be core and not an add-on in the asset library?

It will be deeply ingrained in the import process

[Calinou]

How does DynamicMesh compare with the existing ImmediateMesh?

Would this mesh resource be supported in 2D as well? (I don't remember if ImmediateMesh works there.)

[clayjohn]

How does DynamicMesh compare with the existing ImmediateMesh?

They are quite different. ImmediateMesh provides an OpenGL 1.x style vertex pushing API and updates the entire mesh upon changes. Accordingly, it doesn't provide access to the full vertex arrays for reading or allow users to do partial updates.

Would this mesh resource be supported in 2D as well? (I don't remember if ImmediateMesh works there.)

Yep, this will work for 2D as well as 3D. From the renderers perspective it will be no different from an ArrayMesh