Open donmccurdy opened 3 years ago
Follow MaterialX a considerable design change in names would be PositionNode
and NormalNode
to GeometricNode( scope, space )
? For example:
new PositionNode( PositionNode.WORLD )
new NormalNode( PositionNode.LOCAL )
...
to
new GeometryNode( GeometryNode.POSITION, SpaceNode.WORLD )
new GeometryNode( GeometryNode.NORMAL, SpaceNode.LOCAL )
...
What do you think?
It is also interesting to think... I too think that don't have to keep the same MaterialX names if there is no need although it wouldn't be bad...
Would it be good to have a position to start "here" perhaps with new names?
Would it be good to have a position to start "here" perhaps with new names?
Sounds good! This is probably the easiest way to introduce new policies.
If you'd rather not use the MaterialX names, that's OK with me — especially if you think our own names might be easier for users. The more important thing, I think, would be to try to fill in the gaps for nodes that are currently missing: remap, various noise nodes, etc. Unfortunately a few of them might be difficult, like the convolution nodes "blur" and "heighttonormal".
I was just about to suggest this to Three.js! Thanks @donmccurdy for beating me to the punch. I suggested to the glTF committee back in 2019 that we add MaterialX support directly into the glTF format as well.
I'd forgotten to include this at the time, but I also wrote a loader for the custom shader graph format used in Shade for iOS. Probably something similar could work for bringing MaterialX into THREE.NodeMaterial.
Developers at Autodesk shared a (work-in-progress) WASM build of the MaterialX library, here:
https://github.com/autodesk-forks/MaterialX/tree/adsk_contrib/dev/source/JsMaterialX
The source library includes functionality to read/edit/write .mtlx
files and node graphs, and to generate GLSL from those node graphs. I'm not sure how much of this functionality is in the WASM bindings today, but the unit tests give some idea of the API. They authors expressed interest in continuing the work, and would likely welcome help on that.
The MaterialX project also provides a collection of sample models, including some Standard Surface materials that might be a good fit for THREE.MeshStandardMaterial and THREE.MeshPhysicalMaterial with node systems.
EDIT: Example of customizing MaterialX GLSL output.
MaterialX's native format is XML. I think that we should look to only support JSON representation in Three.js.
One additional thing, MaterialX is in XML. I would suggest having an alternative MaterialX format that is in JSON. Why?
Some other sources for this: https://www.w3schools.com/js/js_json_xml.asp
I think alternate serialization (i.e. not XML) is worth considering down the road, but would rather get something working with existing .mtlx
files first. Creating a custom variant of MaterialX is not likely to help with interop, which is really the point of choosing an existing graph-based shader system. There are options to parse XML more cheaply than xml2js (https://developer.mozilla.org/en-US/docs/Web/Guide/Parsing_and_serializing_XML), and the MaterialX library (which might turn out to be fundamental here anyway?) embeds pugixml.
Simple example with no dependencies:
// content
const xml = `<?xml version="1.0"?>
<materialx version="1.38" colorspace="lin_rec709" fileprefix="../../../Images/">
<nodegraph name="NG_Greysphere_Calibration">
<texcoord name="texcoord1" type="vector2" />
<place2d name="place2d" type="vector2">
<input name="texcoord" type="vector2" nodename="texcoord1" />
<input name="offset" type="vector2" value="-1.66, -0.49" />
<input name="scale" type="vector2" value="0.21, 0.21" />
<input name="pivot" type="vector2" value="0.5, 0.5" />
</place2d>
<image name="image1" type="color3">
<input name="texcoord" type="vector2" nodename="place2d" />
<input name="file" type="filename" value="greysphere_calibration.png" colorspace="srgb_texture" />
<input name="uaddressmode" type="string" value="clamp" />
<input name="vaddressmode" type="string" value="clamp" />
</image>
<output name="out1" type="color3" nodename="image1" />
</nodegraph>
<standard_surface name="SR_Greysphere_Calibration" type="surfaceshader">
<input name="base" type="float" value="1.0" />
<input name="base_color" type="color3" nodegraph="NG_Greysphere_Calibration" output="out1" />
<input name="diffuse_roughness" type="float" value="0" />
<input name="specular_roughness" type="float" value="0.7" />
<input name="specular_IOR" type="float" value="1.5" />
</standard_surface>
<surfacematerial name="Greysphere_Calibration" type="material">
<input name="surfaceshader" type="surfaceshader" nodename="SR_Greysphere_Calibration" />
</surfacematerial>
</materialx>`;
// parse
const parser = new DOMParser();
const dom = parser.parseFromString(xml, 'application/xml');
// evaluate
dom.documentElement.getAttribute('colorspace'); // → "lin_rec709"
dom.querySelector('[name=image1]'); // → <image name="image1" type="color3">
At this point I am just envisioning a loader (i.e. THREE.MaterialXLoader) that creates a THREE.NodeMaterial from *.mtlx
documents, did you have something else in mind @bhouston?
I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?
Besides parsing XML, if we convert THREE.NodeMaterials convert to the MaterialX node definitions, what would be the usefulness of using MAterialX in WASM? I just want to understand.
...what would be the usefulness of using MaterialX in WASM? I just want to understand.
I also want to understand this better, but the MaterialX library does more than just parse the XML. It can:
*.mtlx
fileI'm not sure if the WASM bindings will support all that, or if we would even want generated GLSL — as opposed to maintaining our own GLSL code for each node. The size of the MaterialX WASM library might be a factor in this; I haven't tried compiling it myself yet.
Attempting a task list:
How large is the WASM runtime? I couldn't find that.
I am okay going with XML for now. THREE.MaterialXLoader sounds amazing. @sunag what do you think?
It sounds good. The use of WASM
it's still unclear to me, seems to compete tasks that should be of NodeMaterial
... I think that we can use a XML
parser and extended the functionalities of NodeMaterial
to meet the requirements of MaterialX
. For example, Bake
and GLSL
generator are real things in NodeMaterial
.
So far, I'm not having much luck compiling MaterialX to WASM from the link above, which is still a work in progress admittedly.
❯ docker exec -it emscripten sh -c "cd build && cmake .. -DMATERIALX_BUILD_JS=ON -DMATERIALX_BUILD_RENDER=OFF -DMATERIALX_BUILD_TESTS=OFF -DMATERIALX_EMSDK_PATH=/emsdk_portable/ && cmake --build . --target install"
JS: Building Release
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build
[ 16%] Built target MaterialXCore
[ 21%] Built target MaterialXFormat
[ 51%] Built target MaterialXGenShader
[ 75%] Built target MaterialXGenGlsl
[ 79%] Built target MaterialXGenOsl
[ 90%] Built target MaterialXGenMdl
[ 94%] Built target MaterialXGenOgsXml
[ 97%] Built target MaterialXGenOgsFx
[ 99%] Built target MaterialXGenArnold
make[2]: *** No rule to make target '../source/JsMaterialX/JsMaterialXCore/JsMaterial.cpp', needed by 'source/JsMaterialX/JsMaterialX.js'. Stop.
make[1]: *** [CMakeFiles/Makefile2:640: source/JsMaterialX/CMakeFiles/JsMaterialX.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Trying to see how far we can get without the WASM bindings, just mapping MaterialX nodes to three.js nodes. Here's roughly how that would look:
https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader
Just covering a small subset of the possible MaterialX nodes and parameters, and using THREE.MeshStandardNodeMaterial, so it's not going to allow all of the Standard Surface parameters. Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that.
Manual mappings seem workable, although there will be a fair bit of glue code involved. If the XML serialization changes from version to version (it looks like this happened with MaterialX v1.36 → v1.37 and v1.37 → v1.38) we may find ourselves wanting a pinned JSON serialization.
"Certainly if we were trying to support Standard Surface completely, we would need the MaterialX GLSL generation for that."
Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.
FYI: I've added a new github issue which gives an outline of proposed work. Autodesk will be taking on some of the work and we'll update status as we progress. (e.g. the build issues are being addressed with a current PR) .Volunteers are always welcome :).
I have posted to the gltf-materialx
Slack discussion as well. Thanks.
https://github.com/autodesk-forks/MaterialX/issues/1184
Why would we need that? Given that Standard Surface is basically equivalent to the glTF PBR Next BRDF, I do not know why we would need code generation.
Mostly I just want to be clear that THREE.MeshStandardNodeMaterial can only support a very small subset of the Standard Surface properties today, and a majority of the Standard Surface samples require more than this. We'll be missing specular, transmission, subsurface, sheen, coat, volume, and thin film — these aren't in MeshStandardMaterial. Perhaps those will be added to MeshPhysicalMaterial eventually (its clearcoat already matches the glTF PBR BRDF pretty well), and perhaps we'll have a MeshPhysicalNodeMaterial to match, but this could be a long wait.
And even for the inputs we do support (base color, metalness, roughness, ...), mapping MaterialX nodes to three.js nodes will require a fair bit of code that may need to be updated with every MaterialX release. A lightweight WASM library that generates GLSL for each input appropriately would be ideal, I think, although I understand that optimizing an existing C++ library for WASM bundle size can be difficult.
If we think about Surface let's consider this implementation -> https://github.com/mrdoob/three.js/pull/21322
I think that we can fork the core code of transmission
, subsurface
, sheen
and others and implement using FunctionNode
for not use WASM
in runtime, right? This would allow us to better adapt the code to threejs
.
On BRDFs: Perhaps let's focus on supporting the realtime glTF PBR BRDF — rather than custom surface shaders from MaterialX — in three.js. This probably means adding features to MeshPhysicalMaterial, and eventually creating a MeshPhysicalNodeMaterial to match it. Work is happening for this purpose https://github.com/mrdoob/three.js/issues/21000 and https://github.com/mrdoob/three.js/issues/16977.
On THREE.MaterialXLoader: I'd vote to start with getting procedural inputs working with our existing BRDF (i.e. MeshStandardNodeMaterial). That will be an awesome starting point for bringing in procedural materials from authoring software like Substance Designer.
@sunag looking at the node list in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader#diff-1c2f8bf6e61d17450f8562c4102b8fd10fa99377ab8fcf1b6f4567b55fc65861R405, does adding nodes like RemapNode (https://github.com/mrdoob/three.js/pull/21793) sound okay to you? Might skip the "COMPOSITING" section for now but other than that would like to try to gradually cover these. The nodes and their inputs are defined in https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf.
Some additions for NodeMaterial coming up:
Will probably create a PR for MaterialXLoader fairly soon. Parsing any particular MaterialX node has been painless, but could benefit from more samples to test, especially procedural node graphs as inputs to Autodesk Standard Surface materials.
Myself and https://github.com/Threekit are pleased to put a bounty of $1000 USD on an accepted PR of an implementation that supports the import of core standard MaterialX nodes and the creation of a shader graph on top of the standard/physical materials.
I have not been working on this recently, but would be glad to see someone pick up the work in https://github.com/mrdoob/three.js/compare/dev...donmccurdy:feat-materialxloader, or feel free to take a different direction too. I can gladly try to answer questions where it's helpful.
Two suggestions that might improve probability of PR acceptance:
@donmccurdy I'll be working on it, thanks in advance for your work so far.
@donmccurdy Could we use the functions mx_*
of GLSL
from MaterialX
on Three.js?
I say this because I have doubts about the license, maybe you know something about it?
@donmccurdy
although I suspect we can just parse the XML directly?
This would be ideal!
although I suspect we can just parse the XML directly?
I think the same thing... I only see some mx_ functions that we can bring in, although most are somewhat universal, and we can take from other sources if needed too.
@jstone-lucasfilm, would it be legal for us to include snippets of MaterialX's shaders into ThreeJS? What are the legal ramifications? @sunag above is asking. We are trying to add support for parsing MaterialX's XML directly into Three.JS shader nodes.
I believe @mrdoob specifically wants to keep ThreeJS fully MIT licensed and thus doesn't have to have separate license agreements. Which makes sense, it keeps things simple.
@jstone-lucasfilm, maybe you could release parts of MaterialX, like its core, under a more permissive license like MIT. I think it would help spread it around.
@bhouston I believe the Apache 2.0 license in MaterialX is compatible with the MIT license in ThreeJS, since we place no requirements other than maintaining copyright and license details in code that is leveraged in other projects. So I believe you could copy a set of functions from MaterialX to ThreeJS, stating the copyright and license under which these functions are declared, and this should have no impact on the broader license of the ThreeJS project. That's my understanding, in any case, based on our work thus far with MaterialX and the ASWF.
We are talking about .glsl
files included in the AcademySoftwareFoundation/MaterialX repository, under Apache 2.0 license, correct? Agreed with @jstone-lucasfilm that to my understanding, Apache 2.0 is broadly considered compatible with the MIT license. I don't know that MaterialX dual-licensing under MIT would change anything — it is still an additional 'notice' license, and should be declared. Disclaimer: IANAL, etc.
We could also put these files under examples/jsm/libs
to draw a clearer distinction here, if needed. Code in that folder is already under various additional licenses.
In case it is useful to the conversation here to see where other rendering engines are regarding MaterialX support, here is the list of MaterialX nodes Blender supports, as well as a discussion on the MaterialX addon.
What is the current status of this? There was a lot of work in 2022 but I'm not sure where things sit. I'm interested in working on this, especially as we just dropped WebGLRenderer support.
Something to note: GLTF is adding support for procedural materials based on the MaterialX Spec. The idea being some sort of interoperability between GLTF and USD versions of files/materials. It's still in draft: https://github.com/KhronosGroup/glTF/pull/2381
@donmccurdy what's the priority of the procedural materials? Is it a main item or just a novelty extension?
For three.js, procedural materials (implemented with TSL) continue to be a very high priority with lots of ongoing effort over the past several years.
For compatibility between TSL, NodeMaterial, and MaterialX in three.js ... I am not currently aware of a widely available, artist-friendly workflow to create and export MaterialX graphs, so it's hard to say. I'm keeping an eye on https://projects.blender.org/blender/blender/issues/112864 in the meantime. Research and improvements from other contributors would be welcome, but I am not actively working on this area.
For glTF, I can't comment on the priority of the draft extension.
I'd like to suggest that we work toward supporting equivalents of the MaterialX Standard Nodes within NodeMaterial. That is not necessarily to say that everything needs to be named the same, but ideally it should be possible to compose any given MaterialX node from, say, <5 three.js nodes. A lot of the standard nodes are actually covered already:
https://www.materialx.org/assets/MaterialX.v1.38D1.Spec.pdf
Note that this does not require supporting the MaterialX format; just having equivalent nodes seems like a good starting point. MaterialX provides open source OSL code for each standard node, as well. MaterialX has support in tools by Adobe, Autodesk, and Pixar — if there's a real chance at being able to exchange shader networks with 3D modeling tools in the long run, I'd be willing to bet that MaterialX is it.