Common Materials Extension

[tparisi]

https://github.com/KhronosGroup/glTF/tree/KHR_materials_common/extensions/Khronos/KHR_materials_common

Comments:

• The JSON examples are inconsistent. Both 1 and 1.0 are used. Arrays are sometimes on the same line and sometimes one element per line.
• index_of_refraction -> indexOfRefraction. glTF uses camel case for property names.
  • Same for constant_attenuation, linear_attenuation, falloff_angle, and falloff_exponent.
• Are we sure CONSTANT needs all those properties? How is each used?
• FLOAT_VEC4 or SAMPLER_2D -> In glTF, what is sampler 2D? Shouldn't this be a string (texture id)? We need to precisely define where the texture coordinates come from, e.g., if both diffuse and specular are texures, do they use the same or different set of texture coordinates?
• technique - why does this have a default? Perhaps it should be required. We should also make this separate from the table and say the table is for the values object.
• "relective": "texture_envmap" - glTF does not support cube maps.
• This needs to include the equations.
• This needs to include some discussion on how vertex attributes use semantics to map material inputs, e.g., normals, texture coordinates, etc.

To-do: Patrick need schema for each of the material types... can you help?

Maybe, but I did to finish the core spec reference doc first. Will let you know later this week.

[tparisi]

Related to #84

[tparisi]

@pjcozzi yes to all of the above. did we decide to punt on environment maps? guess i missed the memo.

for these two items, yes but it will take a little time to do these

• This needs to include the equations.
• This needs to include some discussion on how vertex attributes use semantics to map material inputs, e.g., normals, texture coordinates, etc.

[pjcozzi]

did we decide to punt on environment maps?

See #266

[pjcozzi]

@tparisi any update here?

[tfili]

@tparisi Do we really need a separate transparency value? Shouldn't that just be part of the colors?

[mlimper]

thanks, after reading over the draft I would say this is exactly the kind of compact, basic material definition which we have been missing wihin glTF so far... definitely very helpful, thanks a lot for the great work!

[pjcozzi]

@mlimper any chance you could implement this soon? Your feedback would be valuable to help tighten up the spec.

[mlimper]

What we would like to implement is a very simple material model to transport colors of the parts - here, the CONSTANT, BLINN, PHONG, and LAMBERT models seem like a perfect match.

I guess it could also be okay to use only PHONG or BLINN instead of using both, and maybe even LAMBERT could be left out, for the sake of simplicity, so that there is only CONSTANT and BLINN, for instance (since LAMBERT is just the diffuse part of PHONG / BLINN). Otherwise, if there are different models and some have a specular component, maybe these ones should not have zero specualr by default (so it makes a difference whether you use default LAMBERT or PHONG/BLINN)?

Apart from that, my main question would be how exactly the lights fit in. We'll not want to use them within our system (although that doesn't necessarily mean they shouldn't be there), we'd only transport material information. One obvious possibility could move the lights out of this proposal and put them into a separate extension, but maybe that's too much and it's okay if they are optional.

[pjcozzi]

We also need prose for generating shaders with skinning.

[tparisi]

@mlimper let's not separate out lights, too complex. Let's instead define that an implementation's use of lights is optional, but if it does use them it needs to implement them correctly (or words to that effect)

[tparisi]

@tfili no we want to keep transparency separate.

[tparisi]

@pjcozzi @mlimper all requested changes are in except the lighting calculations, that's next.

Please review the last few commits and LMK. I'm actually wondering about this 'optional lights' issue... it's sort of like cameras in the base glTF. What is a conforming implementation obligated to do? I feel like for both cameras and lights the use of them should be optional, but obviously, if they are used they need to be implemented to spec. Thoughts?

[tparisi]

First attempt at lighting calculations are in (see 4126edf) - please review

[mlimper]

What is a conforming implementation obligated to do? I feel like for both cameras and lights the use of them should be optional, but obviously, if they are used they need to be implemented to spec. Thoughts?

Well, to be honest, I guess we would not implement lights from glTF, or at least they would get a very low priority, since we usually integrate data from multiple sources to show everythin in a common context. If we would also import lights with every asset, it would all become a huge mess. So, that would probably mean our implementation would not be 100% spec conformant, I guess...

The conceptual problem here, and also with other things such as Cameras, is probably that glTF tries to be a format for both: Defining a single asset, and defining a complete scene setup. It's a problem of scope, lights and cameras often have a global scope, I guess, while meshes can be perfectly seen as something that can be integrated into an existing scene. However, it might not even be that clear for all cases: if a light is supposed to model the sun, it makes sense to ignore it when integrating an asset into an existing scene. In contrast, if a light is supposed to model one of the headlights of a car, attached to the model, it makes probably sense to load it, even if the current context already has lighting.

@pjcozzi What is your approach in Cesium? What would happen if I would try to import an asset that has lights into an existing world?

Possible conclusion: Some frameworks might want to ignore lights that have a global scope. Probably, all frameworks should be able to load lights that have a local scope, if they want to be spec-conformant. Therefore, one could add this scoping property to lights, to separate what is "scene data" and what is "asset data"... (maybe this also makes sense for other properties, such as camera viewpoints?)

Another thing that could make sense is that the glTF spec provides a list of all existing features, such as "meshes", "lights", "cameras", ... . An implementation can then provide a list of features it supports. On top of that, one could identify certain feature sets as "support level X" or "profile Y", to ease communication about the state of an implementation.

[tfili]

@tparisi Looking at technique.parameters shouldn't the node property be moved into an extension object. It looks like it was removed from the core spec, but I don't see it in the extension spec.

[tparisi]

@tfili good catch. Please make sure the converter does this and I will update the extension spec.

[tfili]

@tparisi @pjcozzi In the converter generated shaders we look in the collada extra tags for double sided lighting and we respect that. It's used by a bunch exporters. Do we want to add this in the extension spec so we can generate the correct shaders or should I output it in the extras tag of the extension?

[tparisi]

I personally think we should just put double-sided into the values. Extras is just another added level of nonsense. @pjcozzi what do you think?

[tfili]

I agree. Also one issue I'm running into is knowing if an image has transparency. When I get the glTF it just has the extension, so I have to generate techniques/shaders/etc, but I can't correctly detect the states at that point. I would have to wait for the image to load and figure it out. That is very difficult with our workflow. The converter however knows this already, so adding a property there would be much easier. For values that take a color or string, maybe have that be an color or object instead like this

"diffuse" : {
    "texture": "texture1",
    "transparent": true
}

[pjcozzi]

I personally think we should just put double-sided into the values. Extras is just another added level of nonsense. @pjcozzi what do you think?

OK.

[pjcozzi]

As for if lights should be required, I suggest that we allow an asset to not have any lights, in which case, the engine is free to use their own lights, e.g., in Cesium, it would be the sun. If an asset contains lights, then the engine is required to generate a shader that uses those lights and any engine-specific lights it has. The Cesium team can make an open-source library to generate the shaders based on @tfili's work.

In the short-term, lights in an asset are not a big deal for Cesium; longer term, we would use them for lights on satellites, aircrafts, trucks, etc.

@tfili what do you think based on your experience implementing this extension so far?

[pjcozzi]

The intro says "BLINN" means "Blinn-Phong" but the Blinn section says it means "Blinn-Torrance-Sparrow." Which is it? Why not just stock Blinn-Phong? @tfili is looking at Three, Babylon, etc. to see what they implement, e.g., they might need to implement another a new material if glTF uses Blinn-Torrance-Sparrow and their engine doesn't.

[pjcozzi]

More feedback:

• Rename constant_attenuation and linear_attenuation to constantAattenuation and linearAttenuation
• "relective": "texture_envmap" - glTF does not support cube maps.
• There are properties in the material table that are not used in the equations for any lighting model. I think this is because they are copied from COLLADA which has a more complicated lighting model. I suggest removing anything outside of the standard equations you provided.

I can help with the schema, but it will be a bit before I will have the time.

[tparisi]

Guys apologies on all the blinn-phong confusion. I cribbed language from COLLADA spec. Can someone suggest a path forward to resolve that?

@pjcozzi I will do the renames... and double-check the list of properties.

[pjcozzi]

I would stick just with the Blinn-Phong lighting model as you described in the equation and then remove the unused parameters from the table.

[tparisi]

Updated camel case in f594fc7... I'm stuck on this environment map thing, I know we're going to need them before long... @pjcozzi is there a holdup somewhere in the base spec regarding specifying cube- and sphere-map textures?

Regarding the lighting calculations and unused params, can you give me an example? I see that some spotlight values aren't taken into account but is there anything else? I believe this is a shortcoming of the equations I cribbed from the COLLADA spec...

[pjcozzi]

@pjcozzi is there a holdup somewhere in the base spec regarding specifying cube- and sphere-map textures?

They are not in the spec. Time is too tight to put them into 1.0. See #266.

Regarding the lighting calculations and unused params, can you give me an example?

The following parameters are not used in the lighting equations for any of the lighting models:

• indexOfRefraction
• reflective
• reflectivity

[tparisi]

r/e env maps - check.

r/e those 3 properties - roger that. nuking.

[tparisi]

Cleanup done, see 83161f7. Let me look at Blinn-Phong mess.

[tparisi]

Cleaned up Blinn-Phong in the intro see aef9485, sorry that was old stuff. I think it's consistent now please review @pjcozzi

[tfili]

@tparisi It looks like you missed a few columns in the materials property table.

[tfili]

@tparisi Any thoughts on specifying transparency if you are using a texture?

[tparisi]

Good catch fixed 76afd19

Not sure what you mean about transparency using a texture...

[tfili]

When implementing this extension I need to figure out whether a material has transparency so I can enable blending. It's easy without textures. With textures, loading all the images to check for transparent images before I process the model is very inconvenient.

[pjcozzi]

@tparisi the table of material parameters also needs to define the minimum and maximum value, e.g., see https://github.com/KhronosGroup/glTF/commit/4126edfeb13be38cff013c36dad473a2a286d654#commitcomment-13734984

[pjcozzi]

When the value of technique is BLINN, this defines a material shaded according to the Blinn-Torrance-Sparrow lighting model or a close approximation.

This should be "Blinn-Phong, not "Blinn-Torrance-Sparrow", right?

[pjcozzi]

I'm just not sold on the entire intro to the Blinn section. I find it unlikely that the typical WebGL engine implements Blinn-Torrance-Sparrow nor do I think the extension spec as currently written provides all the parameters for this lighting model.

Instead, we should use a stock Blinn-Phong like:

• Three.js - https://github.com/mrdoob/three.js/blob/master/src/materials/MeshPhongMaterial.js
• BabylonJS - https://github.com/BabylonJS/Babylon.js/blob/master/src/Materials/babylon.standardMaterial.ts

[pjcozzi]

@tparisi I think we should also explicitly list the attribute input to the material (which is a subset of the parameters) just like stock glTF techniques. Otherwise, our implementation has to search all the meshes/primitives for the attribute inputs. See https://github.com/AnalyticalGraphicsInc/cesium/pull/3039/files#r42259749

[tfili]

@tparisi We were looking at our implementation and it is annoying to determine the type of the values. I have to do something like this which is probably slower than it needs to be

var type = typeof value;
typeValue = -1;
switch (type) {
    case 'string':
        typeValue = WebGLConstants.SAMPLER_2D;
        break;
    case 'number':
        typeValue = WebGLConstants.FLOAT;
        break;
    default:
        if (Array.isArray(value)) {
            // 35664 (vec2), 35665 (vec3), 35666 (vec4)
            typeValue = 35662 + value.length;
        }
        break;
}  

My thought was to expand on the idea I had of making it an object for textures and always make it an object with a type like this. type and value would be required and transparent would be for textures only, optional and defaulting to false. Another option is we can throw the texture coordinate in there as well so we can support multiple ones.

"diffuse" : {
    "value": "texture1",
    "type": 35678,
    "transparent": true,
    "texcoord": "TEXCOORD_0"
}
"specular" : {
    "value": [0.2, 0.2, 0.2, 1.0],
    "type": 35666
}

CC @pjcozzi

[pjcozzi]

+1 for @tfili's suggestion. We could generalize "texcoord": "TEXCOORD_0" to "attribute" : "attribute-id" to also account for https://github.com/KhronosGroup/glTF/issues/424#issuecomment-148762276

[pjcozzi]

@tparisi I made a new empty branch from spec-1.0: KHR_materials_common. Can you please move this extension to that branch so we can cleanly merge spec-1.0 into master without this work-in-progress?

[tparisi]

will do - I'll get that out of the way soon and then we will get back to the list of oustanding issues

[tfili]

@tparisi Looking at the light properties, they seem a little weird. Generally lights have a constantAttenuation, linearAttenuation & quadraticAttenuation. This is how COLLADA them defined for common materials as well.

The spec is missing quadraticAttenuation and has a distance property. I'm not actually sure what to do with the distance property. I think we should have it the same way as it was in COLLADA, since that is such a well known technique for defining lights.

[tparisi]

sure makes sense adding to the list

[tfili]

@tparisi It also looks like we don't need direction either as direction and position are controlled by the node the light is attached to.

[tparisi]

not true; only position is controlled by the node the light is attached to. there is no direction property for a node.

[tfili]

Looking at the COLLADA spec it says lights point (0,0,-1) and the rotation of the node orients the light accordingly. Looking at the code in the gltf converter, this was how they were designed in the beginning as well.

Not saying it's correct, but that is how the code currently works.

[pjcozzi]

I agree with @tfili that the light should have a well-known direction; otherwise, it is redundant to have have the transform in the node and the direction in the light, e.g., there are two rotations when only one is needed.

[tparisi]

I suppose this is ok, we can go with it. Will remove direction.

[tparisi]

Removed direction and added language defining direction vector. Also added quadraticAttenuation. 8b07ba1