Gltf loader should allow custom material substitutions

[viridia]

What problem does this solve or what need does it fill?

My engine uses GLTF models, but it also supports special effects that are not part of the GLTF specification, such as animated materials. The way I handle this is by annotating the models (in GltfExtras), and then using those tags to replace the material after the model is loaded.

This is hard to do in Bevy because the loader is a black box, and because all of the materials are encapsulated in handles.

What solution would you like?

I'd like a way to have something like a MaterialResolver or transformer which would allow access to the material definition as the GLTF model is being loaded, and which can replace the GLTF material with a custom material of my choice. This resolver/transformer would be capable of caching materials as well, so if I load 100 different models with the same special effect, I'm not creating 100 copies of the material.

What alternative(s) have you considered?

An alternative approach is to modify the GLTF models after they have been loaded; however this makes using GLTF models more difficult generally, since now every place which references a GLTF asset handle has to delay adding the model to the scene until after the post-processing has been done.

[alice-i-cecile]

In the past, I've used bevy_scene_hook for this. I'd prefer a more targeted solution however: this is a very common need.

When loading models in this way, I'd like easy access to the original material: applying a transformation is very common and works much better with asset-driven workflows.

[viridia]

I'm not sure bevy_scene_hook would work for my use case. My world is semi-procedural, for example a "tree" model is instantiated hundreds of times, with algorithmically chosen location and rotation for each instance. There are no fixed "scenes" in the sense of loading an entire game level from an asset, it's more like Minecraft where the world is made of modular units. I wouldn't want to run the "hook" every time a model is instantiated.

[viridia]

A few more thoughts here:

There's been some discussion of how this might be done via AssetProcessors, once they evolve a bit more. Unfortunately, I don't know whether this will work, and it really depends on what the eventual AssetProcessor API looks like.

Ideally, material substitutions should be done early rather than late: that is, the substitution should happen during (or immediately after) deserialization of the original material, rather than at the end after all the scenes have finished deserializing. The reason for this is that every mesh object is going to have a material handle. If we wait until the end, then not only do we have to replace all the materials, but we also have to update all the handles which point to those materials. This makes the code a lot more complex, since now you have to recursively walk the entire node tree, and map old handles to new.

Unfortunately, I suspect that asset processors are more like a pipeline: what the processor gets as input is the output of the previous stage, meaning that all of the serialization is done and the processor only gets to see the final product.

An additional problem here is that the replaced material may not be the same Rust type: most GLTF-loaded materials are StandardMaterial, but the material that we substitute may be an ExtendedMaterial or some other kind of custom material. We'll need to make sure that the meshes which hold on to these handles are able to dereference the material handles even if the data type changes.

[kaosat-dev]

This is something I have been bumping into a few times and would also love a clean solution to: One possible solution I had in mind (not tested out fully yet, had issues with the various types of Materials), is having a resource containing a mapping of material name to a material , that the gltf loader could use : ie

• if the resource contains a key that matches the loaded material information/name, use that material
• otherwise, fall back to the Standard material

[pranavmalikk]

Any progress on this? I'm attempting to use ExtendedMaterial to replace the standard materials that come pre-loaded with my own shader, but the shader is not reflected on the object. Does anyone have input on how we can maybe remove the standard material that comes pre-set with the glb/gltf or put the ExtendedMaterial directly on the object?

[bananaturtlesandwich]

would love some more work on this