search

mrdoob / three.js

JavaScript 3D Library.
https://threejs.org/
MIT License
101.59k stars 35.29k forks source link

NodeMaterial: Support MaterialX Standard Nodes #20541

Open donmccurdy opened 3 years ago

donmccurdy commented 3 years ago

I'd like to suggest that we work toward supporting equivalents of the MaterialX Standard Nodes within NodeMaterial. That is not necessarily to say that everything needs to be named the same, but ideally it should be possible to compose any given MaterialX node from, say, <5 three.js nodes. A lot of the standard nodes are actually covered already:

https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf

Note that this does not require supporting the MaterialX format; just having equivalent nodes seems like a good starting point. MaterialX provides open source OSL code for each standard node, as well. MaterialX has support in tools by Adobe, Autodesk, and Pixar — if there's a real chance at being able to exchange shader networks with 3D modeling tools in the long run, I'd be willing to bet that MaterialX is it.

sunag commented 3 years ago

Follow MaterialX a considerable design change in names would be PositionNode and NormalNode to GeometricNode( scope, space )? For example:

new PositionNode( PositionNode.WORLD ) new NormalNode( PositionNode.LOCAL ) ... to new GeometryNode( GeometryNode.POSITION, SpaceNode.WORLD ) new GeometryNode( GeometryNode.NORMAL, SpaceNode.LOCAL ) ...

What do you think?

sunag commented 3 years ago

It is also interesting to think... I too think that don't have to keep the same MaterialX names if there is no need although it wouldn't be bad...

sunag commented 3 years ago

Would it be good to have a position to start "here" perhaps with new names?

Mugen87 commented 3 years ago

Would it be good to have a position to start "here" perhaps with new names?

Sounds good! This is probably the easiest way to introduce new policies.

donmccurdy commented 3 years ago

If you'd rather not use the MaterialX names, that's OK with me — especially if you think our own names might be easier for users. The more important thing, I think, would be to try to fill in the gaps for nodes that are currently missing: remap, various noise nodes, etc. Unfortunately a few of them might be difficult, like the convolution nodes "blur" and "heighttonormal".

bhouston commented 3 years ago

I was just about to suggest this to Three.js! Thanks @donmccurdy for beating me to the punch. I suggested to the glTF committee back in 2019 that we add MaterialX support directly into the glTF format as well.

donmccurdy commented 3 years ago

I'd forgotten to include this at the time, but I also wrote a loader for the custom shader graph format used in Shade for iOS. Probably something similar could work for bringing MaterialX into THREE.NodeMaterial.

donmccurdy commented 3 years ago

Developers at Autodesk shared a (work-in-progress) WASM build of the MaterialX library, here:

https://github.com/autodesk-forks/MaterialX/tree/adsk_contrib/dev/source/JsMaterialX

The source library includes functionality to read/edit/write .mtlx files and node graphs, and to generate GLSL from those node graphs. I'm not sure how much of this functionality is in the WASM bindings today, but the unit tests give some idea of the API. They authors expressed interest in continuing the work, and would likely welcome help on that.

The MaterialX project also provides a collection of sample models, including some Standard Surface materials that might be a good fit for THREE.MeshStandardMaterial and THREE.MeshPhysicalMaterial with node systems.

EDIT: Example of customizing MaterialX GLSL output.

bhouston commented 3 years ago

MaterialX's native format is XML. I think that we should look to only support JSON representation in Three.js.

bhouston commented 3 years ago

One additional thing, MaterialX is in XML. I would suggest having an alternative MaterialX format that is in JSON. Why?

Some other sources for this: https://www.w3schools.com/js/js_json_xml.asp

donmccurdy commented 3 years ago

I think alternate serialization (i.e. not XML) is worth considering down the road, but would rather get something working with existing .mtlx files first. Creating a custom variant of MaterialX is not likely to help with interop, which is really the point of choosing an existing graph-based shader system. There are options to parse XML more cheaply than xml2js (https://developer.mozilla.org/en-US/docs/Web/Guide/Parsing_and_serializing_XML), and the MaterialX library (which might turn out to be fundamental here anyway?) embeds pugixml.

Simple example with no dependencies:

// content
const xml = `<?xml version="1.0"?>
<materialx version="1.38" colorspace="lin_rec709" fileprefix="../../../Images/">
  <nodegraph name="NG_Greysphere_Calibration">
    <texcoord name="texcoord1" type="vector2" />
    <place2d name="place2d" type="vector2">
      <input name="texcoord" type="vector2" nodename="texcoord1" />
      <input name="offset" type="vector2" value="-1.66, -0.49" />
      <input name="scale" type="vector2" value="0.21, 0.21" />
      <input name="pivot" type="vector2" value="0.5, 0.5" />
    </place2d>
    <image name="image1" type="color3">
      <input name="texcoord" type="vector2" nodename="place2d" />
      <input name="file" type="filename" value="greysphere_calibration.png" colorspace="srgb_texture" />
      <input name="uaddressmode" type="string" value="clamp" />
      <input name="vaddressmode" type="string" value="clamp" />
    </image>
    <output name="out1" type="color3" nodename="image1" />
  </nodegraph>
  <standard_surface name="SR_Greysphere_Calibration" type="surfaceshader">
    <input name="base" type="float" value="1.0" />
    <input name="base_color" type="color3" nodegraph="NG_Greysphere_Calibration" output="out1" />
    <input name="diffuse_roughness" type="float" value="0" />
    <input name="specular_roughness" type="float" value="0.7" />
    <input name="specular_IOR" type="float" value="1.5" />
  </standard_surface>
  <surfacematerial name="Greysphere_Calibration" type="material">
    <input name="surfaceshader" type="surfaceshader" nodename="SR_Greysphere_Calibration" />
  </surfacematerial>
</materialx>`;

// parse
const parser = new DOMParser();
const dom = parser.parseFromString(xml, 'application/xml');

// evaluate
dom.documentElement.getAttribute('colorspace'); // → "lin_rec709"
dom.querySelector('[name=image1]'); // → <image name="image1" type="color3">

At this point I am just envisioning a loader (i.e. THREE.MaterialXLoader) that creates a THREE.NodeMaterial from *.mtlx documents, did you have something else in mind @bhouston?

bhouston commented 3 years ago

I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?

Besides parsing XML, if we convert THREE.NodeMaterials convert to the MaterialX node definitions, what would be the usefulness of using MAterialX in WASM? I just want to understand.

donmccurdy commented 3 years ago

...what would be the usefulness of using MaterialX in WASM? I just want to understand.

I also want to understand this better, but the MaterialX library does more than just parse the XML. It can:

I'm not sure if the WASM bindings will support all that, or if we would even want generated GLSL — as opposed to maintaining our own GLSL code for each node. The size of the MaterialX WASM library might be a factor in this; I haven't tried compiling it myself yet.

donmccurdy commented 3 years ago

Attempting a task list:

bhouston commented 3 years ago

How large is the WASM runtime? I couldn't find that.

sunag commented 3 years ago

I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?

It sounds good. The use of WASM it's still unclear to me, seems to compete tasks that should be of NodeMaterial... I think that we can use a XML parser and extended the functionalities of NodeMaterial to meet the requirements of MaterialX. For example, Bake and GLSL generator are real things in NodeMaterial.

donmccurdy commented 3 years ago

So far, I'm not having much luck compiling MaterialX to WASM from the link above, which is still a work in progress admittedly.

❯ docker exec -it emscripten sh -c "cd build && cmake .. -DMATERIALX_BUILD_JS=ON -DMATERIALX_BUILD_RENDER=OFF -DMATERIALX_BUILD_TESTS=OFF -DMATERIALX_EMSDK_PATH=/emsdk_portable/ && cmake --build . --target install"
JS: Building Release
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build
[ 16%] Built target MaterialXCore
[ 21%] Built target MaterialXFormat
[ 51%] Built target MaterialXGenShader
[ 75%] Built target MaterialXGenGlsl
[ 79%] Built target MaterialXGenOsl
[ 90%] Built target MaterialXGenMdl
[ 94%] Built target MaterialXGenOgsXml
[ 97%] Built target MaterialXGenOgsFx
[ 99%] Built target MaterialXGenArnold
make[2]: *** No rule to make target '../source/JsMaterialX/JsMaterialXCore/JsMaterial.cpp', needed by 'source/JsMaterialX/JsMaterialX.js'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:640: source/JsMaterialX/CMakeFiles/JsMaterialX.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
donmccurdy commented 3 years ago

Trying to see how far we can get without the WASM bindings, just mapping MaterialX nodes to three.js nodes. Here's roughly how that would look:

https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader

Screen Shot 2021-05-04 at 9 21 45 PM

Just covering a small subset of the possible MaterialX nodes and parameters, and using THREE.MeshStandardNodeMaterial, so it's not going to allow all of the Standard Surface parameters. Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that.

Manual mappings seem workable, although there will be a fair bit of glue code involved. If the XML serialization changes from version to version (it looks like this happened with MaterialX v1.36 → v1.37 and v1.37 → v1.38) we may find ourselves wanting a pinned JSON serialization.

bhouston commented 3 years ago

"Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that."

Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.

bernardkwok commented 3 years ago

FYI: I've added a new github issue which gives an outline of proposed work. Autodesk will be taking on some of the work and we'll update status as we progress. (e.g. the build issues are being addressed with a current PR) .Volunteers are always welcome :). I have posted to the gltf-materialx Slack discussion as well. Thanks. https://github.com/autodesk-forks/MaterialX/issues/1184

donmccurdy commented 3 years ago

Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.

Mostly I just want to be clear that THREE.MeshStandardNodeMaterial can only support a very small subset of the Standard Surface properties today, and a majority of the Standard Surface samples require more than this. We'll be missing specular, transmission, subsurface, sheen, coat, volume, and thin film — these aren't in MeshStandardMaterial. Perhaps those will be added to MeshPhysicalMaterial eventually (its clearcoat already matches the glTF PBR BRDF pretty well), and perhaps we'll have a MeshPhysicalNodeMaterial to match, but this could be a long wait.

And even for the inputs we do support (base color, metalness, roughness, ...), mapping MaterialX nodes to three.js nodes will require a fair bit of code that may need to be updated with every MaterialX release. A lightweight WASM library that generates GLSL for each input appropriately would be ideal, I think, although I understand that optimizing an existing C++ library for WASM bundle size can be difficult.

sunag commented 3 years ago

If we think about Surface let's consider this implementation -> https://github.com/mrdoob/three.js/pull/21322 I think that we can fork the core code of transmission, subsurface, sheen and others and implement using FunctionNode for not use WASM in runtime, right? This would allow us to better adapt the code to threejs.

donmccurdy commented 3 years ago

On BRDFs: Perhaps let's focus on supporting the realtime glTF PBR BRDF — rather than custom surface shaders from MaterialX — in three.js. This probably means adding features to MeshPhysicalMaterial, and eventually creating a MeshPhysicalNodeMaterial to match it. Work is happening for this purpose https://github.com/mrdoob/three.js/issues/21000 and https://github.com/mrdoob/three.js/issues/16977.

On THREE.MaterialXLoader: I'd vote to start with getting procedural inputs working with our existing BRDF (i.e. MeshStandardNodeMaterial). That will be an awesome starting point for bringing in procedural materials from authoring software like Substance Designer.

donmccurdy commented 3 years ago

@sunag looking at the node list in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader#diff-1c2f8bf6e61d17450f8562c4102b8fd10fa99377ab8fcf1b6f4567b55fc65861R405, does adding nodes like RemapNode (https://github.com/mrdoob/three.js/pull/21793) sound okay to you? Might skip the "COMPOSITING" section for now but other than that would like to try to gradually cover these. The nodes and their inputs are defined in https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf.

donmccurdy commented 3 years ago

Some additions for NodeMaterial coming up:

Will probably create a PR for MaterialXLoader fairly soon. Parsing any particular MaterialX node has been painless, but could benefit from more samples to test, especially procedural node graphs as inputs to Autodesk Standard Surface materials.

bhouston commented 2 years ago

Myself and https://github.com/Threekit are pleased to put a bounty of $1000 USD on an accepted PR of an implementation that supports the import of core standard MaterialX nodes and the creation of a shader graph on top of the standard/physical materials.

donmccurdy commented 2 years ago

I have not been working on this recently, but would be glad to see someone pick up the work in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader, or feel free to take a different direction too. I can gladly try to answer questions where it's helpful.

Two suggestions that might improve probability of PR acceptance:

  1. Must be compatible with NodeMaterial (not just generating THREE.RawShaderMaterial GLSL with the MTLX SDK)
  2. If WASM dependencies are required for the MTLX SDK (although I suspect we can just parse the XML directly?) the size should be minimized as much as possible.
sunag commented 2 years ago

@donmccurdy I'll be working on it, thanks in advance for your work so far.

sunag commented 2 years ago

@donmccurdy Could we use the functions mx_* of GLSL from MaterialX on Three.js?

I say this because I have doubts about the license, maybe you know something about it?

mrdoob commented 2 years ago

@donmccurdy

although I suspect we can just parse the XML directly?

This would be ideal!

sunag commented 2 years ago

although I suspect we can just parse the XML directly?

I think the same thing... I only see some mx_ functions that we can bring in, although most are somewhat universal, and we can take from other sources if needed too.

bhouston commented 2 years ago

@jstone-lucasfilm, would it be legal for us to include snippets of MaterialX's shaders into ThreeJS? What are the legal ramifications? @sunag above is asking. We are trying to add support for parsing MaterialX's XML directly into Three.JS shader nodes.

I believe @mrdoob specifically wants to keep ThreeJS fully MIT licensed and thus doesn't have to have separate license agreements. Which makes sense, it keeps things simple.

bhouston commented 2 years ago

@jstone-lucasfilm, maybe you could release parts of MaterialX, like its core, under a more permissive license like MIT. I think it would help spread it around.

jstone-lucasfilm commented 2 years ago

@bhouston I believe the Apache 2.0 license in MaterialX is compatible with the MIT license in ThreeJS, since we place no requirements other than maintaining copyright and license details in code that is leveraged in other projects. So I believe you could copy a set of functions from MaterialX to ThreeJS, stating the copyright and license under which these functions are declared, and this should have no impact on the broader license of the ThreeJS project. That's my understanding, in any case, based on our work thus far with MaterialX and the ASWF.

donmccurdy commented 2 years ago

We are talking about .glsl files included in the AcademySoftwareFoundation/MaterialX repository, under Apache 2.0 license, correct? Agreed with @jstone-lucasfilm that to my understanding, Apache 2.0 is broadly considered compatible with the MIT license. I don't know that MaterialX dual-licensing under MIT would change anything — it is still an additional 'notice' license, and should be declared. Disclaimer: IANAL, etc.

We could also put these files under examples/jsm/libs to draw a clearer distinction here, if needed. Code in that folder is already under various additional licenses.

jo-chemla commented 4 months ago

In case it is useful to the conversation here to see where other rendering engines are regarding MaterialX support, here is the list of MaterialX nodes Blender supports, as well as a discussion on the MaterialX addon.

DennisSmolek commented 3 months ago

What is the current status of this? There was a lot of work in 2022 but I'm not sure where things sit. I'm interested in working on this, especially as we just dropped WebGLRenderer support.

Something to note: GLTF is adding support for procedural materials based on the MaterialX Spec. The idea being some sort of interoperability between GLTF and USD versions of files/materials. It's still in draft: https://github.com/KhronosGroup/glTF/pull/2381

@donmccurdy what's the priority of the procedural materials? Is it a main item or just a novelty extension?

donmccurdy commented 3 months ago

For three.js, procedural materials (implemented with TSL) continue to be a very high priority with lots of ongoing effort over the past several years.

For compatibility between TSL, NodeMaterial, and MaterialX in three.js ... I am not currently aware of a widely available, artist-friendly workflow to create and export MaterialX graphs, so it's hard to say. I'm keeping an eye on https://projects.blender.org/blender/blender/issues/112864 in the meantime. Research and improvements from other contributors would be welcome, but I am not actively working on this area.

For glTF, I can't comment on the priority of the draft extension.