NodeMaterial

[sunag]

Hi.

I started developing of a THREE.NodeMaterial to reconcile the differences materials between 3D authoring software. In SEA3D Studio has options to create layers in Albedo with mask and various blend modes, Rim shader and others without need custom shader code. I would like to bring this to Three.JS with node shader.

I think that MeshPhongMaterial, MeshPhysicalMaterial and others can easily be based on NodeMaterial through of a interface for backward compatibility or proxy only.

UPDATED http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_materials_nodes.html http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_postprocessing_nodes.html

Syntax example for uses UV1 or UV2 for texture:

var usesUv2 = true;
var isLightmap = false;

var t = new THREE.NodeTexture( texture, new THREE.NodeUV( usesUv2 ) );

var nodemat = new THREE.NodePhongMaterial();
if (isLightmap) nodemat.light = t;
else nodemat.color = t;
nodemat.build(); // build shader

I am making an editor too, currently this would be the interface. Color is albedo and transform is the vertex position.

I am also taking care that it can be used in a deferred shading. Now I will create reflection and refraction inputs.

Will be sharing the news to the PR, suggestions, tests and enhancement are welcome :+1:

[AndrewRayCode]

@bhouston oh nice, looks like we have a lot of overlapping requirements, aka default value, display name, human readable descriptions, etc. I don't know what the future holds but it would be a moderately easy change for me to use a format more like what you propose.

[GGAlanSmithee]

A formal shader format makes a lot of sense. It should live with the lib though, so that it can evolve with it easier. For someone that is rather new to three.js, is there any specific reason such a thing does not already exist? Should be interesting for the editor if nothing else.

*EDIT Maybe having the spec in three.js would defeat its purpose if it's meant to be consumed by other rendering pipes.

[sunag]

Right now it's probably more useful for me than for Three's core, since Three builds its own shaders its own way.

You can generate the same material in other language if you make modifications inside THREE.NodeGL.prototype.generate once a NodeMaterial(*Flow) is a visual language. The NodeBuilder can be the door to it. Currently is the intermediary of data between Nodes.

I cleaned an example of NodeMaterial "raw": It can still generate several bugs. https://github.com/sunag/sea3d/blob/gh-pages/Labs/Three.JS-NodeMaterial/index.html#L179 https://github.com/sunag/sea3d/blob/gh-pages/Labs/Three.JS-NodeMaterial/node/NodeMaterial.js#L9

I think to advance this with particles created root nodes. NodePass for Multi-Pass and NodeMaterialD after the PR what should I do in a few days.

I suggested a THREE.ShaderFrogLoader instead of a runtime shader.

[sunag]

In "Flow" the connectors don't pop to top level when dragging nodes over each other.

@MasterJames Thank you! I'll be looking at it.

[zz85]

:+1:

[sunag]

I started to create the examples. I hope to finish this week. :sweat_smile: http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_materials_nodes.html

Thoughts?

[WestLangley]

@sunag +1 : - )

[mrdoob]

+1!

[bhouston]

This is so awesome. I'd default on the plants+wall texture rather than the cloudy-like displacement. :) That is just beautiful.

[sunag]

I'd default on the plants+wall texture rather than the cloudy-like displacement.

You refer to the 'layers' example? Yet I will post more examples for us

gpu soft-body

[AndrewRayCode]

For comparison with a different approach, the Shader Frog graph editor is now live, and it's in the "kick the tires" phase. I'd like to underscore that it doesn't require any changes to Three.js core.

ShaderFrog currently has more power than traditional node editors. It has infinite node types to work with because it can parse and understand any shader. For example, to apply a reflection only to the edges of an object, you could simple take a glow shader, which is an entirely standalone shader with a fragment and vertex shader:

...and multiply it by a reflection shader...

...using the shader graph...

And ta-da! A reflection rim shader.

You can also, for example, mask any two shaders using any other shader, and so on:

Note that the only "node type" required by this process is a multiply node (there are others in ShaderFrog), which is almost entirely independent of GLSL. ShaderFrog can work with any shaders, which is why it has infinite node types. This advanced technology approach allows for this manipulation independent of the Three.js changes.

ShaderFrog has first class export support to Three.js, for free. All without modifying the Three.js source code. Sea3D is a third party product, like ShaderFrog. [the last part of this comment as the dumbest thing i've ever written, redacting!]

[richardanaya]

@DelvarWorld I really don't see a node-based shader construction abstraction as third-party specific technology. If any thing, it allows more competitors to become involved in node-based shader tech and to all benefit from the further benefit of each other's work ( more nodes, optimized construction, etc.).

As a graphics programmer like myself who understands shaders at a high level ( from tools like Blender 3D), but not at a low level, node based abstraction seems like a great way for me to programmatically interact with shaders using my 3D modeling knowledge. I understand the node setup for PBR for instance, but god help me if I ever wanted to write that in GLSL.

[bhouston]

@DelvarWorld Amazing. I love the new editor.

[WestLangley]

@DelvarWorld +1 Very nice! : - )

[AndrewRayCode]

Thanks! :)

I really don't see a node-based shader construction abstraction as third-party specific technology.

I understand the node setup for PBR for instance

Part of my counter example here is that you can get both of these things for free independent of Three.js. All ShaderFrog shaders are open source so any nodes that become part of a PBR implementation can also be learned from, edited and improved over time (as well as composed with any other shaders, not just some subset in Three.js core). To me, the fact that all of this can be accomplished without Three changes means that it's unnecessary, and blurs the line between Three.js being a webgl wrapper API and now driving implementation specific features. Storing shader node data is highly app specific and as far as I know there's currently no common node editor standard format for any software? Enforcing one might be harmful to other implementations.

[richardanaya]

@DelvarWorld As I understand this change, ShaderFrog will still be able to function and would continue to be able to go beyond GraphMaterial if it so desired. I looked at the example source code for GraphMaterial, and I was able to understand the simplicity of how it worked without even having to think about a third party solution/editor. As a graphics developer, I am concerned with how to programmatically make materials that impress quickly, I feel that these are a proper concern of ThreeJS.

[sunag]

I think that NodeMaterial and Flow has two different focuses. Flow and ShaderFrog are both visual editors third parties solution currently in closed source (I think mainly in artists for this) otherside NodeMaterial is an open-source shader editor for programmers done to Three.JS. I agree with @richardanaya in the sense that ShaderFrog can go beyond of NodeMaterial if desired. Just like I think Flow not need be just a material editor. I really hope great improvements of ShaderFrog in the future as usual.

wip: caustic-voronoi http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_materials_nodes.html

[bhouston]

@sunag I was going through your source code tonight. I really love it. Some quick feedback from my readings this evening:

• I would like to see a PR sooner rather than later. I can help. :)
• It may be cool to allow for arbitrary attributes rather than just the standard ones (uv, uv2, normal, position, etc.) I know that a lot of shaders can benefit from the tangent vec4s, more than just 2 uvs and also multiple color attributes. Thus I would generalize the needsPosition into a requestedAttributes array or something.
• I would suggest some organization of the nodes into the main NodeMaterial, the BRDFs (Phong, Standard, etc), texture nodes (NodeCubeTexture, NodeTexture), accessor (?) nodes (NodePosition, NodeNormal), math nodes (NodeOperator, etc..), utility nodes (NodeSwitch, ...), and extra (?) (NodeVelocity, ...) I think this would help a lot.
• It would be nice to have matrix accessors for Projection, View, and World matrices.
• You are missing some of the operators: % ^ & << >> ~ |
• I'm not sure i see the value in NodePhongMaterial when the class is so small, I'd just have users have to create it using the NodePhong node combined with a NodeMaterial (although I still do not like that name, GraphMaterial feels a lot more nature to me, but I can live with it.)
• I'd call NodeTimer -> NodeTime... but again personal preference.
• I wonder if there is some way to combine many of the variable nodes into one like you've done with NodeMath#, NodeOperator. These seem so similar: NodeViewPosition, NodeWorldPosition, NodeViewNormal, NodeTransformedNormal, NodeTransformedPosition, NodeProjectPosition...
• I would like to see the set of optional arguments for NodeMath#, NodeOperator to be in a form that can be a list, so that one can easily populate a dropdown box with it. Right now because they are members just capitalized on the classes, it is a bit harder to identify them. I am not sure of a good solution that is still convenient for coding. Maybe just collecting them into a list after their definitions as members of the class, so just add those lists?
• NodeTransformedNormal, NodeTransformedPosition are difficult for me to figure out what they are without looking at the code. Are they in world space, view space, object space? "Transformed" is a very generic term, which just means multiplied by a matrix. I'll look at the code for them now....
• I would allow for NodeUV and NodeColor to take an index rather than being limited to 2 and 1 channel respectively. Again I would like to see these made generic, really you pass in a string or one of a number of presets rather than having it hard coded to a set of specific channels. You can still have some dedicated code paths for some variables, but it would fall back to something generic for names it doesn't recognize for special case handling.

[bhouston]

More feedback:

• I would like to have a graph with more than one BRDF, basically two Standard BRDF and then do energy preservation between them. Once you get the PR in I would like to add this functionality. It will make the system more generic and more powerful.

[sunag]

@bhouston Wow! Thank you!

The request attributes, organization of folders, NodeTime, NodePosition, NodeNormal, NodeView was revised. Added +3 examples too.

in operators and matrix we have to add compatibility with integer and matrix conversions. For this to be done in automatic way as in float / vectors#. We have enough work for the next updates.. :sweat_smile:

Tomorrow I will make the PR. I have to review some things yet.

[bhouston]

@sunag:

I believe this is a bug:

addGui( 'metalnessA', roughnessA.number, function( val ) {

                        roughnessA.number = val;

                    }, false, 0, 1 );

[sunag]

I believe this is a bug:

Thanks!

[richardanaya]

I just got giddy going to the example page and seeing a PBR node graph <3

[sunag]

<3 Changes of the last revision https://github.com/sunag/sea3d/commit/a2e3813b53902331e91dbc0f6bae0136a384ad0c

[sunag]

NodeMaterial rev 4 + NodePass

http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_postprocessing_nodes.html

[mrdoob]

Niice!

[bhouston]

I was going to suggest using the nodes for post-processing but I was waiting for the first part to be done. Damn, that is beautiful!

[sunag]

NodeMaterial rev 5 + LightNode

http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_materials_nodes.html

Skin

Toon Shading

++ color-adjustment and plush examples.

[mrdoob]

Niiice!

[sunag]

subsurface scattering :sweat_smile:

This is a kind of shader that can work in future updates.

http://sunag.github.io/sea3d/Labs/Three.JS-NodeMaterial/webgl_materials_nodes.html

[mrdoob]

😮!

[dimarudol]

Thanks for awesome NodeMaterial!

I have several comments/bugs:

• trailing whitespace breaks function node

var n1 = new THREE.FunctionNode(" float mul_2_float(float x) { return x*2.0; }"); // problem
var n2 = new THREE.FunctionNode("float mul_2_float(float x) { return x*2.0; }"); // ok

• problem with int parameter in function node

working example with float parameter:

var floatFuncNode= new THREE.FunctionNode("float mul_2_float(float x) { return x*2.0; }");

var funcCallNode = new THREE.FunctionCallNode(floatFuncNode);
funcCallNode.inputs.x = new THREE.FloatNode(0.2);

var colorResNode = new THREE.OperatorNode(new THREE.ColorNode(0x00ff00),
    funcCallNode, THREE.OperatorNode.MUL);

var mat = new THREE.PhongNodeMaterial();
mat.color = colorResNode;

broken example with int parameter:

var intFuncNode= new THREE.FunctionNode("float mul_2_int(int x) { return float(x)*2.0; }");

var funcCallNode = new THREE.FunctionCallNode(intFuncNode);
funcCallNode.inputs.x = new THREE.IntNode(1);

var colorResNode = new THREE.OperatorNode(new THREE.ColorNode(0x00ff00),
    funcCallNode, THREE.OperatorNode.MUL);

var mat = new THREE.PhongNodeMaterial();
mat.color = colorResNode;

• suggestion: pass function parameters in FunctionCallNode constructor

//current:
var funcCallNode = new THREE.FunctionCallNode(floatFuncNode);
funcCallNode.inputs.param1 = new THREE.FloatNode(1);
funcCallNode.inputs.param2 = new THREE.ColorNode(0x0f0f0f);

// proposed:
var funcCallNode = new THREE.FunctionCallNode(floatFuncNode, 
    { param1: new THREE.FloatNode(1), param2: new THREE.ColorNode(0x0f0f0f) } );

// and, i think suitable in some cases: 
var funcCallNode = new THREE.FunctionCallNode(floatFuncNode, 
    [ new THREE.FloatNode(1), new THREE.ColorNode(0x0f0f0f) ] );

• does PhongNodeMaterial compatible with bump map?

[dimarudol]

few more notes:

• problem with PositionNode.LOCAL in fragment shader in "raw" NodeMaterial:

var material = new THREE.NodeMaterial(
        new THREE.RawNode( new THREE.PositionNode( THREE.PositionNode.PROJECTION ) ),
        new THREE.RawNode( new THREE.PositionNode( THREE.PositionNode.LOCAL ) )
    );

resulting shaders:

varying vec3 vPosition;
void main(){
gl_Position = (projectionMatrix * modelViewMatrix * vec4( position, 1.0 ));
vPosition = transformed;
}

varying vec3 vPosition;
void main(){
gl_FragColor = vec4(vPosition,0.0);
}

Probably "transformed" functionality should be moved from Phong/Standart node materials to NodeMaterial

• ConstNode usage

    var material = new THREE.NodeMaterial(
        new THREE.RawNode( 
            new THREE.OperatorNode(
                new THREE.PositionNode( THREE.PositionNode.PROJECTION ),
                new THREE.ConstNode("float TWO = 2.0;"),
                THREE.OperatorNode.MUL)
            ),
        new THREE.RawNode( new THREE.ColorNode( 0xff0000 ) )
    );

In that case result shader contains gl_Position = ((projectionMatrix * modelViewMatrix * vec4( position, 1.0 ))*TWO); but TWO constant is not declared. Is it my mistake or bug? Also, ConstNode is semicolon sensitive, so "float TWO = 2.0" is not parsed.

[sunag]

Thanks @dimarudol ! If you want to feel free with the notes as soon as possible will make the new revision.

but TWO constant is not declared. Is it my mistake or bug?

I will consider this way of use and in material for example material.include( node )

At the time try a global constant:

var TWO = new THREE.ConstNode("float TWO = 2.0;");
THREE.NodeLib.add( TWO ); // global

 var material = new THREE.NodeMaterial(
        new THREE.RawNode( 
            new THREE.OperatorNode(
                new THREE.PositionNode( THREE.PositionNode.PROJECTION ),
                TWO,
                THREE.OperatorNode.MUL)
            ),
        new THREE.RawNode( new THREE.ColorNode( 0xff0000 ) )
    );

[rraallvv]

Hi @sunag, I've been playing with Sea3d Flow and it looks pretty cool. Just wanted to ask whether there is an official repo somewhere for it. Regards.

[sunag]

Hi @rraallvv, not yet, it is already to be ready. As soon as possible I post the news.

[rraallvv]

Thanks @sunag, it’s very much appreciated.

[martinRenou]

Hi @sunag ! You did a really good job ! I've got a question : for my project it would be great to use your nodes structure but I need per vertex color (and custom per vertex float values), but in your nodes system you only got colorNode and floatNode applied to the entire mesh.. Do you think there is a way to easily implement something like a bufferColorNode (using THREE.BufferAttribute) ?

For example I've got this scene :

Each vertex have its own color. It is possible with THREE.BufferAttribute and a THREE.ShaderMaterial but there's no equivalent in your code.

I would need something like : `let material = new THREE.StandardNodeMaterial(); let texColorMap = new THREE.TextureNode( new THREE.TextureLoader().load("colorMap.jpg")); let customData = new THREE.BufferFloatNode(new Float32Array(...), "data"); // The second parameter is the name of the attribute

let colorMap = new THREE.FunctionNode([ "vec3 colorMap(sampler2D texColorMap, float data){", " return vec3(texture2D(texColorMap, customFunc(data)));", // customFunc return a vec2 depending on the data "}" ].join("\n"));

let colorMapCall = new THREE.FunctionCallNode(colorMap); colorMapCall.inputs.texColorMap = texColorMap;

material.color = colorMapCall; material.build();`

By the way, it seems like I can't use sampler2D parameters for the FunctionNode...

Am I missing something ? I think I could help if needed :)

[sunag]

@martinRenou Thanks! Hmm, maybe something like for now:

bufferGeometry.addAttribute( 'color', new THREE.BufferAttribute( colors, 4 ) );
...
colorMapCall.inputs.data = new THREE.ColorsNode(); // send color.x to float slot

I had not thought yet in assign a dynamic geometry attribute with node. But I think this be a good idea... maybe something like THREE.BufferAttributeNode...

[martinRenou]

@sunag thanks for your answer :) I'll try that. If it's ok for you I'll start work on a THREE.BufferAttributeNode, for now I'm still reading your code and trying to understand the structure.

1. Concerning THREE.FunctionNode, as I said, I really need to set a sampler2D in my parameters as it's used as a colorMap.. The pixel color is calculated with texture2D() and the second parameter of texture2D() is calculated using my custom data (which is for example a temperature). I understand that the idea of using vec3 parameters was for being friendly with users, but FunctionNode could also support sampler2D parameters, don't you think ?
2. Furthermore, THREE.FunctionNode only write in the fragment shader, but wouldn't it be nice to be able to write in the vertex shader too ?

This would be my use case, it's called IsoColor, because the color is corresponding to a data (temperature, pressure...) red ->high temperature, blue -> low temperature :

1. I'm thinking about writing a THREE.IsoColorNode (that would replace the FunctionNode of my diagram), and other nodes that would be very interesting for scientific visualizations. Are you interested in that ? @mrdoob would you be interested too ?

[sunag]

Concerning THREE.FunctionNode, as I said, I really need to set a sampler2D in my parameters as it's used as a colorMap.. The pixel color is calculated with texture2D() and the second parameter of texture2D() is calculated using my custom data (which is for example a temperature). I understand that the idea of using vec3 parameters was for being friendly with users, but FunctionNode could also support sampler2D parameters, don't you think ?

I understand your need now! I will create a PR for this and others notes of the @dimarudol ...

Furthermore, THREE.FunctionNode only write in the fragment shader, but wouldn't it be nice to be able to write in the vertex shader too ?

I do not tested this but I suppose so.

[sunag]

I added more two examples:

• sampler2D are in triangle-blur example
• custom BufferAttribute are in custom-attribute example

It is working but I will still give a polished in code...

https://github.com/mrdoob/three.js/pull/9636

[martinRenou]

Wow ! You really did a good job, it's working for me too :smile:

Thanks again for your work !

[martinRenou]

Hi @sunag,

For now we can't have FunctionNode with "void" type as we need to send it to the material as a parameter (color, alpha...). For example if I want to implement a clip plane FunctionNode, all I want to do is wright a shader part like this :

void clipPlane(vec4 plane){
 if(dot(position, plane.xyz) > plane.w) discard;
}

But that's not possible to give it to the material... A solution would be to return a float which is alpha canal, but that's not clean and optimized. Don't you think that it would be cool to add some void functions to a material like that:

var clipPlane = new FunctionNode([
"void clipPlane(vec 4 plane){",
" if (dot(position, plane.xyz) > plane.w) discard;",
"}"].join("\n"));
var clipPlaneCall = new FunctionCallNode(clipPlane);
clipPlaneCall.inputs.plane = myVec4Node;

var threshold = new FunctionNode([
"void threshold(float upperBound, float lowerBound, float data){",
" if(data < lowerBound) discard;",
" if(data > upperBound) discard;",
"}"].join("\n"));
var thresholdCall = new FunctionCallNode(threshold);
thresholdCall.inputs.upperBound = myFloatNode1;
thresholdCall.inputs.lowerBound = myFloatNode2;
thresholdCall.inputs.data = myAttributeNode;

var myMaterial = new StandardNodeMaterial();
myMaterial.color = ...
myMaterial.alpha = ...

// voidFunctions is not a good name I'm not inspired...
myMaterial.voidFunctions = [clipPlaneCall, thresholdCall];

For a more "design" example, if I want to make holes on my teapot with this texture: I would like to do something like that:

var holes = new FunctionNode([
"void holes(vec3 texColor){",
" if (/*texColor too much dark*/) discard;",
"}"].join("\n"));
var holesCall = new FunctionCallNode(holes);
holesCall.inputs.texColor = new TextureNode(LoadTexture("holes-text.jpg"));

var myMaterial = new StandardNodeMaterial();
myMaterial.voidFunctions = [holesCall];
myMaterial.side = THREE.DoubleSide;

[martinRenou]

Another thing is that we do not have control on the order of functions in shaders. If I've functions that are not commutative, I don't have control on the result...

let see this example:

var transparencyPlane = new FunctionNode([
"float transparencyPlane(vec 4 plane){",
" if (dot(position, plane.xyz) > plane.w) return 0.5.;",
" return 1.;",
"}"].join("\n"));
var transparencyPlaneCall = new FunctionCallNode(transparencyPlane);
transparencyPlaneCall.inputs.plane = myVec4Node;

var displacement = new FunctionNode([
"vec3 displacement(vec3 vector){",
" return position + vector;",
"}"].join("\n"));
var displacementCall = new FunctionCallNode(displacement);
displacementCall.inputs.vector = myVec3Node;

var myMaterial = new StandardNodeMaterial();
myMaterial.transform = displacementCall;
myMaterial.alpha = transparencyPlaneCall;

If a point is behind the transparency plane and comes beyond due to the displacement, in the case which "transparencyPlaneCall" is called before "displacementCall" alpha = 1., and in the case which it is called after alpha = 0.5. So I would like to set the order of function calls... Do you know what I mean ?

[sunag]

Hi @martinRenou

Hmm, about void functions maybe a ProxyNode to use in slot because this need follow a order or initialize at start or end of the code, maybe functionsStart or functionsEnd slots...

Using alpha slot if alpha is 0 is discartted automatically. https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/PhongNode.js#L180

So I would like to set the order of function calls...

Exist a sequence see here: https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/PhongNode.js#L122

The function order is done automatically using simple dependencies algorithm, should put the FunctionCall in a previous slot, color is first specular is second for example.

This example is two differente shader code: fragment to transparencyPlaneCall and vertex to displacementCall.

It makes me think use variables (varying and locals) to expand not inputs and const only will be very interesting. Maybe a VarNode...

[sunag]

Done r6 - https://github.com/mrdoob/three.js/pull/9636

[martinRenou]

Using alpha slot if alpha is 0 is discartted automatically.

Sorry for that, didn't see it.

The function order is done automatically using simple dependencies algorithm, should put the FunctionCall in a previous slot, color is first specular is second for example.

I understand, in fact I just wanted to be sure that function that discard pixels was called first. Sounds good to me to use alpha if you discard it just after my functions :smiley:

This example is two differente shader code: fragment to transparencyPlaneCall and vertex to displacementCall.

Right ! So that was not a good example. My mistake.

Done r6 - #9636

Cool ! Thanks for your answers.

[martinRenou]

It makes me think use variables (varying and locals) to expand not inputs and const only will be very interesting. Maybe a VarNode...

Sounds good too, it could be interesting to be able to explicitly wright a function in the vertex shader, use a varNode, and use it with a function in the fragment shader. Don't you think ?

[martinRenou]

@sunag I think there's an issue with the using of AttributeNode with FunctionNode in the vertex shader..

If I have a code like this :

var customData = new THREE.AttributeNode("data", "float");

var myFunc = new THREE.FunctionNode([
 "vec3 myFunc(float data){",
 " return vec3(data, 0., 0.);"
 "}"].join("\n"));
var myFuncCall = new THREE.FunctionCallNode(myFunc);
myFuncCall.inputs.data = customData;

material.transform = myFuncCall;

The shader is written in this order :

...
varying float nVdata;
attribute float data;
...
float myFunc(float data){
 return return vec3(data, 0., 0.);
}

void main(){
...
transformed = myFunc(nVdata); // We use nVdata but it is not initialized
...
nVdata = data;
...
}

A simple fix would be to initialize nVdata to data ?