search

donmccurdy / glTF-Transform

glTF 2.0 SDK for JavaScript and TypeScript, on Web and Node.js.
https://gltf-transform.dev
MIT License
1.42k stars 150 forks source link

Merge returning 'invalid string length' error #1532

Closed Kyozon closed 3 weeks ago

Kyozon commented 1 month ago

Describe the bug The gltf-transform tool returns with an error message 'invalid string length' when trying to merge gltf files with a very high instance count.

To Reproduce

The files are too big to attach to this issue, please feel free to suggest another way to share the files.

  1. Open CLI in root folder containing the .gltf files as seen in the below image: Screenshot 2024-10-16 175932

  2. Set NODE_OPTIONS=--max-old-space-size=4096 to get around memory limitations

  3. Use the gltf-transform merge command with all files to be merged, as gltf-transform merge RiverSeedling_1_Cinematic.gltf RiverSeedling_2_Cinematic.gltf RiverSeedling_3_Cinematic.gltf RiverSeedling_4_Cinematic.gltf merged2.glb

  4. See error as per the image attached below: Screenshot 2024-10-16 174755

Expected behavior The large files are merged successfully into a singular gltf/glb.

Versions:

Additional context To give this a bit more context, these files pertain to a geometry nodes scattering system in Blender with approx. 3M instances. As of today, Blender's 4.2 exporter seems to hang indefinitely with an instance count higher than 500K per file. The idea is to export them separately then merge them back to together via gltf-transform into a single .glb/gltf that represents the complete scattering system.

I imagine this is a very niche use case that is not widely used with glTF, but I'm wondering if there any suggestions as to how to solve this tricky problem.

donmccurdy commented 1 month ago

I believe you're probably seeing Node.js limits on string size (500 MB, to my understanding^1). Normally a scene is heavy on binary geometry, textures, or animation — not string data — but with 500K+ nodes, the scene graph definition (represented in JSON) greatly outweighs everything else. If you can export with the EXT_mesh_gpu_instancing glTF extension instead, it will hopefully reduce the size dramatically by using a more efficient binary format to represent the data. In Blender 4.2+ the option is enabled as "GPU instances" here:

gpu_instancing

Once you have an export, you can check any file with...

gltf-transform inspect scene.glb

... and see if it contains the EXT_mesh_gpu_instancing extension. Extensions are listed toward the top of the report, but this my take a while to process for files with 100K+ nodes.

We may need to focus on reducing the file size that way, since the Node.js limits are difficult to avoid here.