.gltf/.glb exporting doesn't clear cache after each export, overloads memory and crashes Blender.

[torrinworx]

Describe the bug When exporting large amounts of large .gltf/.glb files in Blender, system memory constantly increases, Blender then crashes.

So I have a script running that exports thousands of models from Blender to a .gltf file(s). These models are big, about 200MB each, and the goal is to export about 6,000 of them. What my script does is select the objects in the Blender file scene, then exports them. After the code runs for about 17 loops, exporting 17 models, Blender crashes and an error message appears saying that Blender has run out of VRAM: RuntimeError: Error: System is out of GPU and shared host memory

I've ran multiple tests with this setup and crashes every time. As the software is running the system memory slowly creeps up to 100% in Task Manager, then Blender crashes. I've used GPU + CPU compute, only GPU compute using only my 3080 ti, but nothing seems to change.

My team and I believe the textures are the source of the issue, but we aren't sure. The reason we suspect the textures is because the external texture folder is about 11.5GB.

To Reproduce Steps to reproduce the behavior:

1. Create a .blend file, at least 200-300MB in size.
2. Create a texture folder with at least 10-15GB of textures stored as external data

3. Use the following script to export a large amount of .gltf/.glb models from Blender:

   count = 0
   for i in range(100):
   # Select a material from Blender file, apply it to an object:
   for obj in bpy.data.collections[some_collection].all_objects:
     selected_object = bpy.data.objects.get(obj.name)
     selected_object.active_material = bpy.data.materials[some_material]
   
   # Select each object in the scene:
   for obj in bpy.data.collections[some_collection].all_objects:
     obj.select_set(True)
   
   #Export the object to 'Save_Path_To_Model_File'
   bpy.ops.export_scene.gltf(
    filepath=f"{Save_Path_To_Model_File}_{count}.gltf",
    check_existing=True,
    export_format='GLTF_SEPARATE',
    use_selection=True)
   
   count += 1

4. Note: System memory as seen in Task Manager increases as this script is running until it overflows and Blender crashes.

Expected behavior Blender maintains constant memory use and doesn't crash when continually exporting lots of .gltf/.glb files.

Screenshots Blender crash log:

RuntimeError: Error: Python: Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\__init__.py", line 628, in execute
    return gltf2_blender_export.save(context, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 35, in save
    json, buffer = __export(export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 52, in __export
    __gather_gltf(exporter, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 66, in __gather_gltf
    active_scene_idx, scenes, animations = gltf2_blender_gather.gather_gltf2(export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather.py", line 30, in gather_gltf2
    scenes.append(__gather_scene(blender_scene, export_settings))
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather.py", line 67, in __gather_scene
    node = gltf2_blender_gather_nodes.gather_node(
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 31, in gather_node
    children=__gather_children(vnode, blender_object, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 109, in __gather_children
    child_node = gather_node(vtree.nodes[child], export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 31, in gather_node
    children=__gather_children(vnode, blender_object, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 73, in __gather_children
    node = gather_node(c, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 31, in gather_node
    children=__gather_children(vnode, blender_object, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 73, in __gather_children
    node = gather_node(c, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 35, in gather_node
    mesh=__gather_mesh(vnode, blender_object, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 246, in __gather_mesh
    result = gltf2_blender_gather_mesh.gather_mesh(blender_mesh,
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_mesh.py", line 61, in gather_mesh
    primitives=__gather_primitives(blender_mesh, uuid_for_skined_data, vertex_groups, modifiers, materials, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_mesh.py", line 147, in __gather_primitives
    return gltf2_blender_gather_primitives.gather_primitives(blender_mesh,
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_primitives.py", line 81, in gather_primitives
    material = gltf2_blender_gather_materials.gather_material(
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_materials.py", line 53, in gather_material
    pbr_metallic_roughness, uvmap_actives_pbr_metallic_roughness = __gather_pbr_metallic_roughness(blender_material, orm_texture, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_materials.py", line 334, in __gather_pbr_metallic_roughness
    return gltf2_blender_gather_materials_pbr_metallic_roughness.gather_material_pbr_metallic_roughness(
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_materials_pbr_metallic_roughness.py", line 19, in gather_material_pbr_metallic_roughness
    metallic_roughness_texture, use_active_uvmap_metallic_roughness_texture = __gather_metallic_roughness_texture(blender_material, orm_texture, export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_materials_pbr_metallic_roughness.py", line 140, in __gather_metallic_roughness_texture
    return gltf2_blender_gather_texture_info.gather_texture_info(
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_texture_info.py", line 23, in gather_texture_info
    return __gather_texture_info_helper(primary_socket, blender_shader_sockets, 'DEFAULT', export_settings)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 38, in wrapper_cached
    result = func(*args, **kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_texture_info.py", line 46, in __gather_texture_info_helper
    'index': __gather_index(blender_shader_sockets, export_settings),
  File "C:\Program Files\Blender Foundation\Blender 3.2\3.2\scripts\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_texture_info.py", line 129, in __gather_index
    return glt

.blend file/ .gltf I cannot include files here do to my clients confidentiality. But any Blender file 200-300MB in size with at least 10GB of external textures will do.

Version

• OS: Windows 10, 21H2
• Blender Version 3.2.2

Additional context System Specs:

GPU: RTX 3080 ti + GeForce GTX 1660 RAM: 4x16GB = 64Gb CPU: Intel Core i7-7700 3.60GHz

[optimus007]

did you try to clear the stored data after each export ?

for bpy_data_iter in (
    bpy.data.objects,
    bpy.data.meshes,
    bpy.data.lights,
    bpy.data.cameras,
    bpy.data.images,
    bpy.data.textures,
    bpy.data.materials,
):
    for id_data in bpy_data_iter:
        bpy_data_iter.remove(id_data)
for collection in bpy.data.collections:
    bpy.data.collections.remove(collection)
for action in bpy.data.actions:
    bpy.data.actions.remove(action)
for rig in bpy.data.armatures:
    bpy.data.armatures.remove(rig)

[torrinworx]

@optimus007 Wouldn't this just remove every asset in the Blender file itself?

[optimus007]

@optimus007 Wouldn't this just remove every asset in the Blender file itself?

Yup, after each gltf export if you clear these files before loading the next file, would it help with the memory usage issue ?

This is just a suggestion, not sure if it'll solve your issue though

[torrinworx]

Ok I just tested this with Blender 3.3.1 Release Candidate and it seems the issue is solved, no memory leak. I want to wait until the version has been released proper before closing this ticket though.

[julienduroure]

Ok, thanks, great news :) Let stay this ticket open until 3.3.1 is officially released. I will close the duplicate dev.blender.org ticket Thanks!

[torrinworx]

Awesome, thank you @julienduroure and @optimus007

[julienduroure]

3.3.1 is now officially released. Can you please confirm that everything is good now?

[julienduroure]

@torrinworx Any news? Did you get time to check with 3.3.1 that the problem is solved?

[torrinworx]

@julienduroure All good on my end! Thank you for the reminder, been a bit busy with work and forgot about this ticket. Have a nice day!