NodeMaterial: Support MaterialX Standard Nodes

[donmccurdy]

I'd like to suggest that we work toward supporting equivalents of the MaterialX Standard Nodes within NodeMaterial. That is not necessarily to say that everything needs to be named the same, but ideally it should be possible to compose any given MaterialX node from, say, <5 three.js nodes. A lot of the standard nodes are actually covered already:

https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf

Note that this does not require supporting the MaterialX format; just having equivalent nodes seems like a good starting point. MaterialX provides open source OSL code for each standard node, as well. MaterialX has support in tools by Adobe, Autodesk, and Pixar — if there's a real chance at being able to exchange shader networks with 3D modeling tools in the long run, I'd be willing to bet that MaterialX is it.

[sunag]

Follow MaterialX a considerable design change in names would be PositionNode and NormalNode to GeometricNode( scope, space )? For example:

new PositionNode( PositionNode.WORLD ) new NormalNode( PositionNode.LOCAL ) ... to new GeometryNode( GeometryNode.POSITION, SpaceNode.WORLD ) new GeometryNode( GeometryNode.NORMAL, SpaceNode.LOCAL ) ...

What do you think?

[sunag]

It is also interesting to think... I too think that don't have to keep the same MaterialX names if there is no need although it wouldn't be bad...

[sunag]

Would it be good to have a position to start "here" perhaps with new names?

[Mugen87]

Would it be good to have a position to start "here" perhaps with new names?

Sounds good! This is probably the easiest way to introduce new policies.

[donmccurdy]

If you'd rather not use the MaterialX names, that's OK with me — especially if you think our own names might be easier for users. The more important thing, I think, would be to try to fill in the gaps for nodes that are currently missing: remap, various noise nodes, etc. Unfortunately a few of them might be difficult, like the convolution nodes "blur" and "heighttonormal".

[bhouston]

I was just about to suggest this to Three.js! Thanks @donmccurdy for beating me to the punch. I suggested to the glTF committee back in 2019 that we add MaterialX support directly into the glTF format as well.

[donmccurdy]

I'd forgotten to include this at the time, but I also wrote a loader for the custom shader graph format used in Shade for iOS. Probably something similar could work for bringing MaterialX into THREE.NodeMaterial.

• Source code: https://github.com/donmccurdy/three-shadenodeloader
• Demo: https://three-shadenodeloader.donmccurdy.com/

[donmccurdy]

Developers at Autodesk shared a (work-in-progress) WASM build of the MaterialX library, here:

https://github.com/autodesk-forks/MaterialX/tree/adsk_contrib/dev/source/JsMaterialX

The source library includes functionality to read/edit/write .mtlx files and node graphs, and to generate GLSL from those node graphs. I'm not sure how much of this functionality is in the WASM bindings today, but the unit tests give some idea of the API. They authors expressed interest in continuing the work, and would likely welcome help on that.

The MaterialX project also provides a collection of sample models, including some Standard Surface materials that might be a good fit for THREE.MeshStandardMaterial and THREE.MeshPhysicalMaterial with node systems.

EDIT: Example of customizing MaterialX GLSL output.

[bhouston]

MaterialX's native format is XML. I think that we should look to only support JSON representation in Three.js.

[bhouston]

One additional thing, MaterialX is in XML. I would suggest having an alternative MaterialX format that is in JSON. Why?

• It can be parsed by JavaScript engines everywhere. Otherwise you have to include large libraries like xml2js: https://www.npmjs.com/package/xml2js -- which is 200KB!
• It is smaller than XML.
• Can use arrays.
• It is faster to parse.
• The glTF standard uses JSON.

Some other sources for this: https://www.w3schools.com/js/js_json_xml.asp

[donmccurdy]

I think alternate serialization (i.e. not XML) is worth considering down the road, but would rather get something working with existing .mtlx files first. Creating a custom variant of MaterialX is not likely to help with interop, which is really the point of choosing an existing graph-based shader system. There are options to parse XML more cheaply than xml2js (https://developer.mozilla.org/en-US/docs/Web/Guide/Parsing_and_serializing_XML), and the MaterialX library (which might turn out to be fundamental here anyway?) embeds pugixml.

Simple example with no dependencies:

// content
const xml = `<?xml version="1.0"?>
<materialx version="1.38" colorspace="lin_rec709" fileprefix="../../../Images/">
  <nodegraph name="NG_Greysphere_Calibration">
    <texcoord name="texcoord1" type="vector2" />
    <place2d name="place2d" type="vector2">
      <input name="texcoord" type="vector2" nodename="texcoord1" />
      <input name="offset" type="vector2" value="-1.66, -0.49" />
      <input name="scale" type="vector2" value="0.21, 0.21" />
      <input name="pivot" type="vector2" value="0.5, 0.5" />
    </place2d>
    <image name="image1" type="color3">
      <input name="texcoord" type="vector2" nodename="place2d" />
      <input name="file" type="filename" value="greysphere_calibration.png" colorspace="srgb_texture" />
      <input name="uaddressmode" type="string" value="clamp" />
      <input name="vaddressmode" type="string" value="clamp" />
    </image>
    <output name="out1" type="color3" nodename="image1" />
  </nodegraph>
  <standard_surface name="SR_Greysphere_Calibration" type="surfaceshader">
    <input name="base" type="float" value="1.0" />
    <input name="base_color" type="color3" nodegraph="NG_Greysphere_Calibration" output="out1" />
    <input name="diffuse_roughness" type="float" value="0" />
    <input name="specular_roughness" type="float" value="0.7" />
    <input name="specular_IOR" type="float" value="1.5" />
  </standard_surface>
  <surfacematerial name="Greysphere_Calibration" type="material">
    <input name="surfaceshader" type="surfaceshader" nodename="SR_Greysphere_Calibration" />
  </surfacematerial>
</materialx>`;

// parse
const parser = new DOMParser();
const dom = parser.parseFromString(xml, 'application/xml');

// evaluate
dom.documentElement.getAttribute('colorspace'); // → "lin_rec709"
dom.querySelector('[name=image1]'); // → <image name="image1" type="color3">

At this point I am just envisioning a loader (i.e. THREE.MaterialXLoader) that creates a THREE.NodeMaterial from *.mtlx documents, did you have something else in mind @bhouston?

[bhouston]

I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?

Besides parsing XML, if we convert THREE.NodeMaterials convert to the MaterialX node definitions, what would be the usefulness of using MAterialX in WASM? I just want to understand.

[donmccurdy]

...what would be the usefulness of using MaterialX in WASM? I just want to understand.

I also want to understand this better, but the MaterialX library does more than just parse the XML. It can:

• Generate GLSL from a node graph
• "Bake" a node graph into a texture
• Write a node graph back to a *.mtlx file

I'm not sure if the WASM bindings will support all that, or if we would even want generated GLSL — as opposed to maintaining our own GLSL code for each node. The size of the MaterialX WASM library might be a factor in this; I haven't tried compiling it myself yet.

[donmccurdy]

Attempting a task list:

• [ ] Evaluate MaterialX WASM bindings.
  • Size? Performance? Importance of GLSL generation?
  • Use WASM at runtime (THREE.MaterialXLoader), only for exporting (THREE.MaterialXExporter), or not at all?
• [ ] Choose MaterialX nodes to support.
  • Standard Nodes?
• [ ] Choose MaterialX BRDFs to support.
  • Standard Surface?
• [ ] Decide how to map MaterialX nodes to three.js nodes.
  • Map to current NodeMaterial node names? Match 1:1 MaterialX node names?
• [ ] Decide serialization path.
  • XML? JSON? Binary?
  • I'd suggest starting with XML, and replacing with JSON or binary if/when it makes sense.
• [ ] Write THREE.MaterialXLoader!

[bhouston]

How large is the WASM runtime? I couldn't find that.

[sunag]

I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?

It sounds good. The use of WASM it's still unclear to me, seems to compete tasks that should be of NodeMaterial... I think that we can use a XML parser and extended the functionalities of NodeMaterial to meet the requirements of MaterialX. For example, Bake and GLSL generator are real things in NodeMaterial.

[donmccurdy]

So far, I'm not having much luck compiling MaterialX to WASM from the link above, which is still a work in progress admittedly.

❯ docker exec -it emscripten sh -c "cd build && cmake .. -DMATERIALX_BUILD_JS=ON -DMATERIALX_BUILD_RENDER=OFF -DMATERIALX_BUILD_TESTS=OFF -DMATERIALX_EMSDK_PATH=/emsdk_portable/ && cmake --build . --target install"
JS: Building Release
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build
[ 16%] Built target MaterialXCore
[ 21%] Built target MaterialXFormat
[ 51%] Built target MaterialXGenShader
[ 75%] Built target MaterialXGenGlsl
[ 79%] Built target MaterialXGenOsl
[ 90%] Built target MaterialXGenMdl
[ 94%] Built target MaterialXGenOgsXml
[ 97%] Built target MaterialXGenOgsFx
[ 99%] Built target MaterialXGenArnold
make[2]: *** No rule to make target '../source/JsMaterialX/JsMaterialXCore/JsMaterial.cpp', needed by 'source/JsMaterialX/JsMaterialX.js'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:640: source/JsMaterialX/CMakeFiles/JsMaterialX.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

[donmccurdy]

Trying to see how far we can get without the WASM bindings, just mapping MaterialX nodes to three.js nodes. Here's roughly how that would look:

https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader

Just covering a small subset of the possible MaterialX nodes and parameters, and using THREE.MeshStandardNodeMaterial, so it's not going to allow all of the Standard Surface parameters. Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that.

Manual mappings seem workable, although there will be a fair bit of glue code involved. If the XML serialization changes from version to version (it looks like this happened with MaterialX v1.36 → v1.37 and v1.37 → v1.38) we may find ourselves wanting a pinned JSON serialization.

[bhouston]

"Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that."

Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.

[bernardkwok]

FYI: I've added a new github issue which gives an outline of proposed work. Autodesk will be taking on some of the work and we'll update status as we progress. (e.g. the build issues are being addressed with a current PR) .Volunteers are always welcome :). I have posted to the gltf-materialx Slack discussion as well. Thanks. https://github.com/autodesk-forks/MaterialX/issues/1184

[donmccurdy]

Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.

Mostly I just want to be clear that THREE.MeshStandardNodeMaterial can only support a very small subset of the Standard Surface properties today, and a majority of the Standard Surface samples require more than this. We'll be missing specular, transmission, subsurface, sheen, coat, volume, and thin film — these aren't in MeshStandardMaterial. Perhaps those will be added to MeshPhysicalMaterial eventually (its clearcoat already matches the glTF PBR BRDF pretty well), and perhaps we'll have a MeshPhysicalNodeMaterial to match, but this could be a long wait.

And even for the inputs we do support (base color, metalness, roughness, ...), mapping MaterialX nodes to three.js nodes will require a fair bit of code that may need to be updated with every MaterialX release. A lightweight WASM library that generates GLSL for each input appropriately would be ideal, I think, although I understand that optimizing an existing C++ library for WASM bundle size can be difficult.

[sunag]

If we think about Surface let's consider this implementation -> https://github.com/mrdoob/three.js/pull/21322 I think that we can fork the core code of transmission, subsurface, sheen and others and implement using FunctionNode for not use WASM in runtime, right? This would allow us to better adapt the code to threejs.

[donmccurdy]

On BRDFs: Perhaps let's focus on supporting the realtime glTF PBR BRDF — rather than custom surface shaders from MaterialX — in three.js. This probably means adding features to MeshPhysicalMaterial, and eventually creating a MeshPhysicalNodeMaterial to match it. Work is happening for this purpose https://github.com/mrdoob/three.js/issues/21000 and https://github.com/mrdoob/three.js/issues/16977.

On THREE.MaterialXLoader: I'd vote to start with getting procedural inputs working with our existing BRDF (i.e. MeshStandardNodeMaterial). That will be an awesome starting point for bringing in procedural materials from authoring software like Substance Designer.

[donmccurdy]

@sunag looking at the node list in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader#diff-1c2f8bf6e61d17450f8562c4102b8fd10fa99377ab8fcf1b6f4567b55fc65861R405, does adding nodes like RemapNode (https://github.com/mrdoob/three.js/pull/21793) sound okay to you? Might skip the "COMPOSITING" section for now but other than that would like to try to gradually cover these. The nodes and their inputs are defined in https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf.

[donmccurdy]

Some additions for NodeMaterial coming up:

• RemapNode: https://github.com/mrdoob/three.js/pull/21793
• Noise2DNode, Noise3DNode, Fractal3DNode: https://github.com/mrdoob/three.js/pull/21800
• CellNoise2DNode, CellNoise3DNode, WorleyNoise2DNode, WorleyNoise3DNode: future work

Will probably create a PR for MaterialXLoader fairly soon. Parsing any particular MaterialX node has been painless, but could benefit from more samples to test, especially procedural node graphs as inputs to Autodesk Standard Surface materials.

[bhouston]

Myself and https://github.com/Threekit are pleased to put a bounty of $1000 USD on an accepted PR of an implementation that supports the import of core standard MaterialX nodes and the creation of a shader graph on top of the standard/physical materials.

[donmccurdy]

I have not been working on this recently, but would be glad to see someone pick up the work in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader, or feel free to take a different direction too. I can gladly try to answer questions where it's helpful.

Two suggestions that might improve probability of PR acceptance:

1. Must be compatible with NodeMaterial (not just generating THREE.RawShaderMaterial GLSL with the MTLX SDK)
2. If WASM dependencies are required for the MTLX SDK (although I suspect we can just parse the XML directly?) the size should be minimized as much as possible.

[sunag]

@donmccurdy I'll be working on it, thanks in advance for your work so far.

[sunag]

@donmccurdy Could we use the functions mx_* of GLSL from MaterialX on Three.js?

I say this because I have doubts about the license, maybe you know something about it?

[mrdoob]

@donmccurdy

although I suspect we can just parse the XML directly?

This would be ideal!

[sunag]

although I suspect we can just parse the XML directly?

I think the same thing... I only see some mx_ functions that we can bring in, although most are somewhat universal, and we can take from other sources if needed too.

[bhouston]

@jstone-lucasfilm, would it be legal for us to include snippets of MaterialX's shaders into ThreeJS? What are the legal ramifications? @sunag above is asking. We are trying to add support for parsing MaterialX's XML directly into Three.JS shader nodes.

I believe @mrdoob specifically wants to keep ThreeJS fully MIT licensed and thus doesn't have to have separate license agreements. Which makes sense, it keeps things simple.

[bhouston]

@jstone-lucasfilm, maybe you could release parts of MaterialX, like its core, under a more permissive license like MIT. I think it would help spread it around.

[jstone-lucasfilm]

@bhouston I believe the Apache 2.0 license in MaterialX is compatible with the MIT license in ThreeJS, since we place no requirements other than maintaining copyright and license details in code that is leveraged in other projects. So I believe you could copy a set of functions from MaterialX to ThreeJS, stating the copyright and license under which these functions are declared, and this should have no impact on the broader license of the ThreeJS project. That's my understanding, in any case, based on our work thus far with MaterialX and the ASWF.

[donmccurdy]

We are talking about .glsl files included in the AcademySoftwareFoundation/MaterialX repository, under Apache 2.0 license, correct? Agreed with @jstone-lucasfilm that to my understanding, Apache 2.0 is broadly considered compatible with the MIT license. I don't know that MaterialX dual-licensing under MIT would change anything — it is still an additional 'notice' license, and should be declared. Disclaimer: IANAL, etc.

We could also put these files under examples/jsm/libs to draw a clearer distinction here, if needed. Code in that folder is already under various additional licenses.

[jo-chemla]

In case it is useful to the conversation here to see where other rendering engines are regarding MaterialX support, here is the list of MaterialX nodes Blender supports, as well as a discussion on the MaterialX addon.

[DennisSmolek]

What is the current status of this? There was a lot of work in 2022 but I'm not sure where things sit. I'm interested in working on this, especially as we just dropped WebGLRenderer support.

Something to note: GLTF is adding support for procedural materials based on the MaterialX Spec. The idea being some sort of interoperability between GLTF and USD versions of files/materials. It's still in draft: https://github.com/KhronosGroup/glTF/pull/2381

@donmccurdy what's the priority of the procedural materials? Is it a main item or just a novelty extension?

[donmccurdy]

For three.js, procedural materials (implemented with TSL) continue to be a very high priority with lots of ongoing effort over the past several years.

For compatibility between TSL, NodeMaterial, and MaterialX in three.js ... I am not currently aware of a widely available, artist-friendly workflow to create and export MaterialX graphs, so it's hard to say. I'm keeping an eye on https://projects.blender.org/blender/blender/issues/112864 in the meantime. Research and improvements from other contributors would be welcome, but I am not actively working on this area.

For glTF, I can't comment on the priority of the draft extension.

[krishanjangid]

is it still open to contribute ?

[donmccurdy]

Hi @krishanjangid! New issues or PRs are certainly welcome. A lot of work has happened since this issue was opened and I'm (personally) not sure what tasks remain anymore. Perhaps @sunag or @hybridherbst would know? If there's nothing to be done right now then we could also close the issue.

For me the main open question is not three.js-specific, but is rather about how to author these MaterialX assets, whether in Blender or Quiltix or something else. I'd love to know if anyone has art workflows in place now, but we don't need to keep this issue open for that reason.

[DennisSmolek]

For me the main open question is not three.js-specific, but is rather about how to author these MaterialX assets, whether in Blender or Quiltix or something else. I'd love to know if anyone has art workflows in place now, but we don't need to keep this issue open for that reason.

IMO There is a lot of work and support for MaterialX workflows, just not with the web as an outlet. It is baked into USD, Maya and anything Autodesk, Houdini is very big on this, NVIDIA w/Omniverse etc. More info here

Before Substance was purchased by Adobe (or maybe it was just at the beginning) they released a MaterialX plugin, however newer versions of substance kinda hide that plugins are even possible and no one has updated the extension to work with the newest version of substance. (Mostly a breaking python version update that needs to happen)

The Arnold team have released nodes to work with Blender if you use it there, but this hasn't seemed to catch on and is kinda a pain to get working right. They started it before the big boom of nodes in Blender and Blender hasn't resolved how they want to handle it with Cycles and Eevee. IE should they be new nodes or just a marker on nodes to say they support it etc. Last I looked work there stalled.

MaterialX nodes are on my shortlist of things to work on as I really want easy ways to transfer materials. But my primary workflow ATM uses r3f which has shaky support of the new Renderer. So with dropping the WebGLRenderer/MaterialX support I haven't had time to focus on it.

Like you said at the start I think MaterialX will be key to exchanging materials. I'm not sure if Three will just piggyback off the work that's happening in getting them into GLTF core and that's how you'll get them in/out, or if maintaining the Loader and creating an Exporter is the way to go..

[hybridherbst]

Here's the plan from my perspective at Needle:

• Get three.js MaterialX loader to support a reasonable and practically useful subset of the MaterialX spec (it will likely never be 100%, MaterialX is a very complex design)

  • this is "almost there" with the work from @sunag, me, @beersandrew and others
  • notably missing: vertex displacement, there are some spec incompatibilites here between what MaterialX does and what TSL does and what Apple's MaterialX flavor does

• What's important is that Needle continues to supports NodeMaterial on the existing WebGLRenderer – even if the three.js team decided this won't work and doesn't want to maintain that bridge, we keep it working in our three.js fork because WebGPURenderer is far from being ready for production

• Once the glTF spec for MaterialX materials is complete, someone needs to refactor the MaterialX loader to support multiple representations (the MaterialX file representation, the glTF representation, and potentially the UsdShade representation)

  • this will also unlock easy-ish representation for compressed textures which is currently non-trivial with the current MaterialXLoader and .mtlx files (we currently map compressed textures in .glb to their expected mtlx name by name, which is brittle)

• We have an exporter for Unity's Shader Graph to MaterialX at the moment, working with three.js MaterialXLoader, which will be released Q1 2025. Blender can also already export a (very limited) subset of MaterialX.

[kwokcb]

For DCC support I agree there is a leaning towards proprietary interchange formats or USD. Direct MaterialX is available for Maya (I'm working on glTF material plug-in for this) and QuiltiX (I worked with the folks there to add a plug-in interface, and have been discussing with them web centric workflows). Consumption is moving towards the consensus on referencing documents in MaterialX format but export or distillation is non-standardized or inaccessible (Pixar's implementation is inside Storm for e.g.).

---

Felix has the interesting point (discussed in a another thread): that MaterialX is not self-contained in terms of resource binding since it requires synchronizing a material asset with a resource asset(s). If ever USD is easily consumable then it could also be a suitable avenue -- but the caveat currently is that there is no "official" USDShade -> MaterialX from Pixar.

It would be nice if MaterialXLoader was generalized a bit to have an interface for consumption while hiding the graph construction logic. I don't see a direct dependence on waiting for glTF MaterialX being finished with any work here helping to hasten the ratification of the spec :). As I mentioned to Felix, I hope to follow this work and contribute if I can.

---

BTW, For vertex displacement, I (and I assume others would be) am interested to hear more about this in the appropriate forum. AFAIK there is no specification for how this behaves in MaterialX and I think it would nice to discuss and get consensus. Apple is heavily involved with MaterialX development so I see no issue with trying to spec this out.