MaterialX should allow specifying a material without a new materialX document

[gareth-morgan-autodesk]

Currently the MaterialX API requires a unique MaterialX Document in order to specify a material. This means at the very least an XML document must be parsed for every material in the scene, and most likely the generation of the MaterialX graph in memory, as well as, in many cases, the code generation. For real life non-trivial scenes with 1000s of material this sets a upper bound on the performance if the materials in a scene are to be represented as MaterialX.

Real life scenes (such as the kind typically encountered in content creation tools and game engines) will frequently have 1000s of unique materials, but they will not have 1000s of unique material networks, typically they will have a small number (<10) actual unique material networks (e.g. different combinations of textures, such as diffuse texture, normal map, roughness map, etc, or different procedural networks), with the remaining 1000s of materials only varying by a small set of simple properties (e.g. base color RGB color vector, roughness float, normal map filename, etc) that are associated with the individual material while sharing the underlying material network.

What is more, in ,most real life scenarios this information is part of the scene description, there is large set of materials, each associated with one of a small number of material "definitions" (meaning the shader or graph defining the material), there is no need to infer this information at runtime. The MaterialX API should have a away of specifying a collection of materials this way, with a single definition (represented by a MaterialX document) and multiple materials each represented by a flat list of material properties. Each property should be a simple name-value pair, with the string name being a MaterialX path, and the value being a "plain old data" MaterialX value (float, int, vector, matrix, or file path)

[jstone-lucasfilm]

@gareth-morgan-autodesk Just to clarify, are you looking for a different approach than the one found in the MaterialX Chess Set, where a set of related materials are defined within a single MaterialX document?

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/Examples/StandardSurface/standard_surface_chess_set.mtlx

This approach should also extend to materials that bind uniform values directly to the inputs of a shading model, and there's no requirement that materials reference node graphs in MaterialX:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/Examples/StandardSurface/standard_surface_glass_tinted.mtlx

[gareth-morgan-autodesk]

So what I'm proposing is different to that, but that is a good example of what I mean. In that file (https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/Examples/StandardSurface/standard_surface_chess_set.mtlx) you have 15 seperate materials: "NG_BishopBlack" "NG_BishopWhite" "NG_CastleBlack" "NG_CastleWhite" "NG_ChessBoard" "NG_KingBlack" "NG_KingWhite" "NG_KnightBlack" "NG_KnightWhite" "NG_PawnBodyBlack" "NG_PawnBodyWhite" "NG_PawnTopBlack" "NG_PawnTopWhite" "NG_QueenBlack" "NG_QueenWhite"

But only 4 actually distinct material networks

• 2 texture inputs: "roughness", "normal"
• 3 texture inputs: "base_color", "roughness", "normal"
• 4 texture inputs: "base_color", "metalness","roughness", "normal"
• 5 texture inputs: "base_color", "metalness","roughness", "subsurface", "normal"

So to populate a scene with that file you need to parse 15 seperate networks, generate the associated MaterialX networks for each of them, and then work out that you only have 4 distinct networks. While you can optimize bits of that, it will never be a fast operation, and is complelely unnessacary, in most real world cases not only are there more materials in a scene (easily 1500 instead of 15) but you would know what material networks you have, as part of your scene description. So what you really want is the ability create those materials with a data structure like this and parse only 4 material networks not 1500:

{
  network: "chessboard_2_textures.mtlx",
  properties: { "diffuse/filename", "metallic/filename", "roughness/filename", "normal/filename" },
  values: { 
  { 
    {"chess_set/bishop_black_base_color.jpg","chess_set/pawn_shared_metallic.jpg","chess_set/bishop_black_roughness.jpg","chess_set/bishop_black_normal.jpg"}, 
    {"chess_set/chessboard_color.jpg","chess_set/chessboard_metallic.jpg","chess_set/chessboard_roughness.jpg","chess_set/chessboard_normal.jpg"}, 
    {"chess_set/pawn_white_base_color.jpg","chess_set/pawn_shared_metallic.jpg","chess_set/pawn_shared_roughness.jpg","chess_set/pawn_white_normal.jpg"}, 
    {"chess_set/pawn_black_base_color.jpg","chess_set/pawn_shared_metallic.jpg","chess_set/pawn_shared_roughness.jpg","chess_set/pawn_black_normal.jpg"}, 

  }
}

[gareth-morgan-autodesk]

So how would that solve the situation described above, and avoid parsing 15 (or 1500) separate documents, when you only in fact have 4 different networks? Is there a way of specifying that in the C++ API(e.g. loadFunctionalNodegraph(document, properties)? )

[jstone-lucasfilm]

Ah, I see what you mean, and I believe this is also supported in modern versions of MaterialX.

MaterialX has the concept of a "functional nodegraph", which allows a single nodegraph to be instantiated an arbitrary number of times with different values bound to its inputs, where those inputs may be any data type that MaterialX supports (including filenames).

Here's one example of this pattern that was developed at Autodesk, where an artistic interface named testmetal is defined as a graph and then instantiated as a MaterialX node with new inputs: https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/TestSuite/libraries/metal/brass_wire_mesh.mtlx

[kwokcb]

I don't see the MaterialX graph representation as needing to handle this so this would reside with the "binding" APIs for MaterialX, or via OpenUSD.

For the material binding API for MaterialX, there is the concept of "variants" and "variant sets" which I believe could be used.

Then it is up to the shader creator to create sets of topologically unique shader graphs (which don't have to be "functional graphs" but just "regular compound graphs"), and apply variants as necessary.

---

This would avoid having to create "functional graphs" which are implementations of new node definitions.

[jstone-lucasfilm]

@gareth-morgan-autodesk When you say "15 separate documents", do you mean parsing 15 separate material elements within a single MaterialX document, and then running shader generation on each of them? In the nomenclature of MaterialX, a document is just a file on disk, and you shouldn't need more than one document to express any of the ideas we're discussing here.

[dbsmythe]

I think a better approach would be to:

• Create a node with all your default material parameters, which could either contain (e.g. within a "material"-type nodegraph) or connect to various shader and other nodes to build the base shader (You'd need 4 of these in your example, one per network)
• Create any number of instantiations of that network material node, setting the specific material parameters on the inputs of the material. See https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/documents/Specification/MaterialX.Specification.md#example-pre-shader-compositing-material for an example of this workflow. In particular, that example defines the nodedef and nodegraph for a "material"-type node, then makes an "Mpaintglassrust" threelayermtl material from it.

[kwokcb]

Hi @dbsmythe , @jstone-lucasfilm , What are your thoughts on the suggestion to variants as it seems less "intrusive". Basically any existing material can be reused as is without creating new definitions. I mostly mention this as folks appear to be having issues with ensuring that custom definitions ae transportable and discoverable between integrations.

[jstone-lucasfilm]

@dbsmythe @kwokcb Indeed there are many ways to express these concepts in MaterialX, including functional nodegraphs, compound nodegraphs, and variants.

Before we get too deep into recommendations for @gareth-morgan-autodesk, though, I'd like to get a better sense of the problem that he's interested in solving.

@gareth-morgan-autodesk In your original problem statement, it seems as if you're aiming to minimize the number of documents (i.e. files on disk), but maybe it's a different quantity (e.g. node graphs, material nodes, shader generator invocations) that you actually want to minimize?

[gareth-morgan-autodesk]

So I think this gist demonstrates the issue pretty well. Its a quick-and-dirty app I've written to very crudely simulated how materials are used in a real life content creation tool or game engine.

The application has single material "type" (a simple MaterialX document with constant values passed to Standard Surface shader), but has a large number of variants of material for that "type", in this case the only difference between the variants is that the RGB base_color value is different for each one. This is obviously a massive over simplification, but it highlights to one-to-many relationship that exists in most real world scenes, where you have a handful of actually distinct material networks, and a large number of materials variants that only vary by a few vector or scalar data items.

Ideally I would be able to specify this directly in the MaterialX C++ API, as this is typically known explicitly ahead of time as part of the scene description and does not need to be inferred. However as there is no way to specify this as part of the MaterialX API, I am forced to create a new MaterialX document for each material, using string manipulation to copy an identical bit of XML with the base color replaced by the value required for this material. This is both a messy, inefficient way of representing this data, it also loses that critically important relationship, once you have generated the materialX documents this way there is no way to easily infer that all the materials are actually just trivial variants of a single material "type" (and trying to infer it from the network after it has been parsed is unnecessary extra work, we have that information in the scene description, we shouldn't have to analyze a network to get it back, it should be obvious from the data model).

As well this fundamental data loss issue, there are huge performance implications to this. In this app we create a mx::GlslMaterial object with the document, node and element associated with each material. The performance implications are shown by the timing results from running MaterialXSample.exe 500 which produces 500 material variants:

Creating 500 materials
Document creation took 7 ms
Importing libraries took 6149 ms
Element lookup took 111 ms
Material creation took 4625 ms
Material compilation took 634338 ms
First frame render took 1041 ms

Just creating the document and importing the standard library takes 12ms per material, the material creation including the shader generation takes 9ms per material, then the shader compilation itself takes about 1.2 seconds per material. The compile time is the uncached result that users will see when they run the application for the first time, each run like this adds 100mb of shader data to the GLCache folder. To emphasize this is for a scene where we know ahead of time there is exactly one shader network used by all the materials, and the materials themselves only vary by a single 3D vector data item and so do not require any different shader code, and yet we have spent 10 minutes creating 100mb of shader data for 500 shaders, two orders of magnitude more than is needed.

As mentioned by @jstone-lucasfilm above we can combine all the documents into a single document (in the sample app you can do this with the -combineMtlxDocs argument) , which avoids the problem of having to attach the standard library to each material, but that requires giving each sub-network, representing the material, to have a unique name, as they are now in the same document, which means the resulting shaders is completely different rather than just varying by a single line (the Material_StandardSurfaceShader_base_color uniform). This makes optimizing the compile time much harder. Plus combining into one document like this makes managing material data much harder (if material is edited or removed in the middle of a document with 1000 materials, how is the application meant to handle that? Pass the entire document to materialX again after editing it?)

[gareth-morgan-autodesk]

that requires giving each sub-network representing the material a unique name, as they are now in the same document, which means the resulting shaders is completely different rather than just varying by a single line (the Material_StandardSurfaceShader_base_color uniform). This makes optimizing the compile time much harder.

To clarify what I mean by this, I can use GlslResourceBindingContext with enableSeparateBindingLocations set to true to ensure public uniforms are added to uniform blocks so the base_color no longer appears in the shader. This reduces the compile time massively, when each material is its own document (as the GLSL code is identical between materials, so the driver will only compile to bytecode once) but doesn't when the materials are combined into a single document (as the names of the GLSL variables and structs are different).

[jstone-lucasfilm]

Hello @gareth-morgan-autodesk!

However as there is no way to specify this as part of the MaterialX API, I am forced to create a new MaterialX document for each material, using string manipulation to copy an identical bit of XML with the base color replaced by the value required for this material.

I think what @dbsmythe, @kwokcb, and I are noting above is that this is not the case at all. There are many ways of expressing the common many-to-one relationship between node graphs and material instances in MaterialX, with some examples being functional nodegraphs, compound nodegraphs, and variants.

I'll focus on the functional nodegraph option here, since it's what we use in production at Lucasfilm/ILM, and Autodesk has provided a great example of this technique in the test suite for the MaterialX project:

https://github.com/AcademySoftwareFoundation/MaterialX/tree/main/resources/Materials/TestSuite/libraries/metal

In this example, the shared interface and nodegraph for a set of testmetal materials are expressed once, with a nodedef for its interface and a nodegraph for its shared nodegraph:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/TestSuite/libraries/metal/libraries/metal_definition.mtlx

This graph can then be instantiated for multiple material instances, with each instance being free to bind the inputs of the associated interface with new values, images, or patterns:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/resources/Materials/TestSuite/libraries/metal/brass_wire_mesh.mtlx#L44

Although the testmetal example doesn't happen to use filenames in its shared interface, this data type is fully supported in functional nodegraphs, and you can see an example of this approach in the shared interface and nodegraph for the triplanarprojection node in the MaterialX standard library:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/libraries/stdlib/stdlib_defs.mtlx#L300 https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/libraries/stdlib/stdlib_ng.mtlx#L204

As in the testmetal example, the interface and graph for triplanarprojection only need to be stated once in MaterialX, and this node can be instantiated in a wide variety of use cases across a material library.

Your notes about improving the performance of material compilation for common shader generation use cases are good ones, and you can find a GitHub Issue on this subject that @ashwinbhat has started, which has received lots of good comments and ideas for improvements in this area. I believe there has been discussion about the idea of "anonymizing" element names before running shader generation, which could allow better reuse of generated shaders when the underlying topology has not been modified, and this would be a great area for contributions in the future.

[dbsmythe]

@kwokcb Re: using material variants vs. instantiating a functional nodegraph material definition: the latter is technically the preferred way as of 1.38 when we made materials into "just nodes" as opposed to separate element types. The intended purpose of material variants is that you could define the same variant name for many materials, and then in the choose that variant for all materials (using the element). E.g. if you had a "red" look, you could define a "red" variant of all the materials used in that look, and the look would materialassign the material choosing the "red" variant.

One certainly could use material variants as a way to specialize a material with specific parameter values, but I feel the instantiation of functional nodegraphs method is considerably cleaner.

[gareth-morgan-autodesk]

I think what @dbsmythe, @kwokcb, and I are noting above is that this is not the case at all. There are many ways of expressing the common many-to-one relationship between node graphs and material instances in MaterialX, with some examples being functional nodegraphs, compound nodegraphs, and variants.

So how is it possible to represent that in the MaterialX C++ API? That allows me a slightly more concise way to specify the material network, but it still means in the example above (see Gist here where I update my example to use that pattern)_I have to parse 500 networks, code generate 500 shaders, and compile 500 shaders, when I need a way of specifying that I only have one network and one shader.

[jstone-lucasfilm]

@gareth-morgan-autodesk If you have multiple materials that all share a single network topology, you can generate and compile that shader a single time, and then bind different uniforms and textures to that shader at runtime.

Are you interested in a system that automatically detects this case and caches shaders based on their network topology? There isn't currently such a system built into MaterialXRender, though I believe USD/Hydra has an early version of this functionality, and I've seen discussion about refinements to MaterialX shader generation that would make it more straightforward for USD/Hydra to detect generated shaders with identical topologies.

[gareth-morgan-autodesk]

@gareth-morgan-autodesk If you have multiple materials that all share a single network topology, you can generate and compile that shader a single time, and then bind different uniforms and textures to that shader at runtime.

But then I'm no longer specifying my material with MaterialX, if I'm going add a custom data path to get materials into my renderer, I might as do away with materialX altogether and represent my materials as JSON of standard surface properties. I want to specify my material with materialX (so its still a well defined material, with a standardized appearance), but not force a one-to-one relationship between material networks and materials.

Are you interested in a system that automatically detects this case and caches shaders based on their network topology? There isn't currently such a system built into MaterialXRender, though I believe USD/Hydra has an early version of this functionality, and I've seen discussion about refinements to MaterialX shader generation that would make it more straightforward for USD/Hydra to detect generated shaders with identical topologies.

No I'm saying we shouldn't need such a system in most cases. IMO that is analogous to forcing all the geometry to be represented as an unconnected triangle soup and then trying to infer the connections between triangles in the renderer. Sure you could do that, but why would you when your scene description stores your triangles as connected arrays?

I've put together this draft PR showing what I am proposing: Its a fairly small change to the core interface (the addition a single new entity an Override), and it doesn't require support by the renderer implementations (Override objects can be trivially converted into regular documents), But if the renderer does use then sensibly the result is a massive performance gain. I've included a PoC implementation of this in MaterialXRenderGlsl and using that in the example app linked above, results in a two order of magnitude speed-up in the 500 material case (basically the load goes from O(n) to o(1) )

[jstone-lucasfilm]

These are meaningful points that you bring up, @gareth-morgan-autodesk, and although MaterialX does include a syntax to express shared material interfaces and per-instance overrides, I can see benefits in clarifying the interpretation of these interfaces and overrides in shader generation, allowing for more efficient shader compilation and rendering.

My sense is that we should discuss these ideas on the MaterialX channel of the ASWF Slack before starting on concrete engineering work, so that our proposed optimizations apply to shared material interfaces and per-instance overrides found in existing MaterialX content across the industry, rather than requiring a brand new syntax to achieve these benefits.