gltf: Extreme memory bloat importing files with shared vertex buffers.

[lyuma]

Godot version

4.2 dev2

System information

Godot v4.2.dev2 - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3090 (NVIDIA; 31.0.15.3640) - Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz (16 Threads)

Issue description

This glTF file contains a large number of submeshes which share blend shapes and vertex buffers between gltf primitives. Here you can see this in action: each primitive contains a different index buffer but share the same attributes and targets.

The effect of this is Godot makes extremely large memory allocations for each mesh surface. In this case, the file was not atlased and contains 44 surfaces on one mesh, some with as few as 12 indices.

In this case, the document is so large, that the scene file without textures bloats to ~650MB from the original ~45MB glb file.

It also contains several large textures, so using "Embed as Uncompressed" pushes Godot over the size limit when saving out a .glb scene. However, it is conceivable that a larger scene would go higher than 650MB to Godot's Vector size limit of 1GB.

Hence, the attached test project crashes Godot with #62585 - but the real bug here is the mesh data is way too large. Even if the crash was fixed, this scene should not be anywhere near 1GB in the first place.

---

Since Godot uses separate vertex buffers for each surface, it might be good to compress unused vertices.

I think it should always compress unused vertices. The algorithm is extremely simple: append vertices to a new list as they are referenced from the index buffer (and assign a mapping from old vertex id to new vertex id).

The other option is to assert that len(vertices) <= len(indices) and only do this optimization if the vertex buffer is obviously too large.

Steps to reproduce

1. Open the attached project
2. Allow it to import the scene.
3. Godot crashes while serializing the ArrayMesh due to the PackedByteArray size limitation.
4. Modify the .glb.import and change gltf/embedded_image_handling=3 to gltf/embedded_image_handling=2
5. Open the project again and allow the file to import with Basis Universal.
6. Check .godot/imported and observe the .scn file is upwards of 600MB large.

Minimal reproduction project

glb_duplicate_vertex_buffers.zip

[jsjtxietian]

append vertices to a new list as they are referenced from the index buffer (and assign a mapping from old vertex id to new vertex id

IMHO it seems like the right solution.

[lyuma]

Tested the attached project on two versions of Godot:

Godot 4.2.1:

644M Mar 12 05:01 astolfo.gltf-d06eff3c4b81f23005346d278513030d.scn
astolfo.glb: 💥  (segfault due to >1GB buffer resize bug)

Godot 4.3 dev + the new PR #89418

23M Mar 12 04:58 astolfo.gltf-d06eff3c4b81f23005346d278513030d.scn
41M Mar 12 04:58 astolfo.glb-897b6833427b3b43ebe410179baf99b2.scn