Design: Material Layering

[HifiExperiments]

Currently, when multiple materials are applied to an object, only the top/highest priority material is shown. I would love to collect thoughts on how to allow true material layering, with blending between multiple materials according to their priorities.

First, some basics:

• Materials Entities currently support 2 "material mapping modes": "uv" (textures use the UV coordinates from the model) and "projected" (textures are projected onto surfaces using the 3D transform of the material entity, e.g. decals)
• In the near future, I will add a "triplanar" mode, which uses triplanar mapping to apply textures, like voxels use (and voxels will support PBR materials). Without material layering, this refers to applying the same texture to all 3 directions
• There are hardware limits to how many textures can be bound at once, which means we will probably only be able to support a maximum 3 materials blending at a time. When 3 materials are used, certain textures, like occlusion and scattering maps, might not be able to be used.

I can think of different blending methods we might want:

• alpha blended: the materials are blended based on alpha, so you can apply a transparent material on top and it will show the second/third materials underneath based on alpha
• triplanar blending: specify one material for the x direction, one for y, and one for z
• splat mapping: an additional "splatMap" texture is supplied which uses the RGB colors to blend between the materials, e.g. the red channel specifies the magnitude of the first material, the green channel specifies the magnitude of the second material, and the blue channel specifies the magnitude of the third material

So my questions are: what do we want the material entity API for this to look like? Is it something like the top material specifies the "blending method" (alpha/triplanar/splatMap)? Or should each material only be responsible for itself somehow (if so, what does it mean to have mixtures of blending methods and how complicated can they be)? Should we be able to explicitly control the number of materials blended? How do other applications handle this? Are there gltf features we should be trying to match here?

[SilverfishVR]

This sounds like a great feature, but yea, it can get very complicated real fast. I think letting the top material set the blending mode is probably the easiest to understand and I struggle to come up with an example where a mix of blend modes, or more than 3 materials, would be needed.

Triplanar could just use the top 3 priority materials for X, Y, and, Z respectively. Or, if we don't care about using the existing material on the mesh in combination with material entities, it could just be priority 0, 1, and, 2.

I don't think material blending as such is a thing in GLTF but one thing that would be very useful is better handling of secondary (and tertiary too maybe) UV maps. A terrain for example, would use tiled materials for grass and dirt, but you really want the splat map texture to use a non tiled UV layout so You can.... well splat it . The UV map (texCoord) can be set for each texture in GLTF and defaults to 0 if not supplied.

Overte does not seem to read this attribute and instead applies some predefined rules, E.g. The Occlusion and light maps always use the second UV if it exists. Even if manual control and/or reading texCoord attribute from GLTF is not implemented at this point it might be useful if the splatmap texture specifically, would also use secondary UV if available.

[HifiExperiments]

yep I have a plan for how to support specifying per-texture texCoords/respecting the GLTF attribute and that will also be coming soon (https://github.com/overte-org/overte/issues/311). it might come after this feature though

keeping it to a single blend mode would certainly simplify things. so yeah, the top material could just specify it...although I wonder if it also needs to be able to specify the number of materials (e.g. just use the top 2, not all 3 if they exist)

for triplanar, I could imagine cases where you want to use one material for one direction, and a second for the other two (a la minecraft).