RFC: Materials And Limiting Render States

[santorac]

Summary:

The material system should generally not allow individual materials to change render states such as BlendState, DepthStencilState, or similar. It also should not allow individual materials to override a shader's draw list tag, because of its impact on render states.

What is the relevance of this feature?

Currently, materials can change any render state at any time, through both .material file settings and through dynamic changes at runtime. This will be problematic if O3DE is to support game consoles in the future, as some game consoles do not allow Pipeline State Objects (PSOs) to be compiled at runtime, and render states are part of the input structure for compiling PSOs. Game engines typically need to know ahead of time what pipeline states will be needed, compile them offline, and ship them with the game. If any material could potentially change render states at any time, it will be difficult (or impossible) for the build system to discover the full set of PSOs that will be needed. If we don't prevent materials from changing render states, then users may develop libraries of material types and materials that rely on this feature, making it harder for O3DE to eventually support game consoles. We should add new limitations soon, before much user content is built on the current system.

The details of how a future offline PSO compiler would work is outside the scope of this RFC. Right now we are mainly concerned about simplifying the design of the material system to mitigate future complications.

Feature design description:

Note: There is an alternative described at the end of this document (see "main alternative") that is pretty compelling, it might even be preferable to what's described in the main body of this RFC.

First, it's important to recognize that some render states must be set according to the needs of the overall renderer, rather than the needs of individual materials or material types. Even if we restrict materials to disallow setting render states, the system render states will need to be configured dynamically. There are two render states that are currently set outside the material system (see Scene::ConfigurePipelineState https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/RPI/Code/Source/RPI.Public/Scene.cpp#L923): RenderAttachmentConfiguration and MultisampleState. These are set according to the configuration of the render pipeline. (We might be able to simplify this as well, given the proposed material pipeline abstraction https://github.com/o3de/sig-graphics-audio/issues/68 but that is out of scope here). We are not proposing any changes to how these render states are configured at this time, but it's important to be aware of them.

There are currently only two cases of materials controlling render state

1. The opacity.mode property in StandardPBR and EnhancedPBR material types. When this property is set to Blended or TintedTransparent there is a lua material functor that converts the forward pass shader to work for transparent surfaces. To do this, it overrides the blend state and depth state, and changes the draw list tag to "transparent" to force it to run in the transparent pass. See https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/Feature/Common/Assets/Materials/Types/StandardPBR_HandleOpacityMode.lua#L58 . Instead, we can simply make two separate .shader files, one for the forward pass and one for the transparent pass, and each .shader file can configure the render states separately. Then the material type will just switch to the appropriate shader as needed.
2. The doubleSided property in StandardPBR and EnhancedPBR material types. When this property is set, there is a lua material functor that overrides the CullMode. See https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/Feature/Common/Assets/Materials/Types/StandardPBR_HandleOpacityDoubleSided.lua#L25 . There isn't really any way around this, we'll need to make an exception to allow materials to set the cull mode.

Although we need to allow dynamic configuration of CullMode (as well as system controlled render states such RenderAttachmentConfiguration and MultisampleState), removing access to the other render states should help simplify the material system design, and mitigate the risk of future complications for PSO baking.

Finally, we should also remove the ability to override the draw list tag because of its relationship to render states. The draw list tag controls which pass is used to run a shader, and the pass dictates which RenderAttachmentConfiguration and MultisampleState is applied. So allowing materials to override the draw list tag will complicate the determination of which PSOs need to be compiled offline. Also, the opacity.mode setting described above is the only use case for this feature, and can be handled by simply switching to a different .shader asset.

Technical design description:

Change StandardPBR and EnhancedPBR to use separate shaders for forward and transparent passes.

1. Duplicate StandardPBR_ForwardPass_EDS.shader as StandardPBR_BlendedTransparent.shader
   1. Change its blend state and depth state to match what's set in ConfigureAlphaBlending https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/Feature/Common/Assets/Materials/Types/StandardPBR_HandleOpacityMode.lua#L25
   2. Change its draw list tag to "transparent"
2. Duplicate StandardPBR_ForwardPass_EDS.shader as StandardPBR_TintedTransparent.shader
   1. Change it's blend state and depth state to match what's set in ConfigureDualSourceBlending https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/Feature/Common/Assets/Materials/Types/StandardPBR_HandleOpacityMode.lua#L34
   2. Change its draw list tag to "transparent"
3. Edit StandardPbr.materialtype and EnhancedPbr.materialtype to include the new shaders.
4. Edit StandardPBR_HandleOpacityMode.lua to remove all the code for overriding blend state and draw list tag.
5. Edit StandardPBR_ShaderEnable.lua to switch between these three shaders instead of using the forward shader for all opacity modes.

Replace the CullMode overrides in StandardPBR and EnhancedPBR with a specialized double-sided mechanism.

Each material property in a material type file can be connected to certain outputs. This is controlled using a MaterialPropertyOutputType enum which can be set to either "ShaderInput" for ShaderResourceGroup fields, or "ShaderOption" for connection to shader options. For example, see https://github.com/o3de/o3de/blob/2a734b4e02755e0315faa9866245c4d52291f90b/Gems/Atom/Feature/Common/Assets/Materials/Types/MaterialInputs/BaseColorPropertyGroup.json#L12 . We will add a new special connection type that will allow the doubleSided material property to be connected to a special "DoubleSided" setting instead of being controlled by a lua functor. The setup will look like this:

{
    "name": "doubleSided",
    "displayName": "Double-sided",
    "description": "Whether to render back-faces or just front-faces.",
    "type": "Bool",
    "connection": {
        "type": "Special",
        "name": "DoubleSided"
    }
},

Here are the specific changes we'll need to make:

1. Update ShaderCollection::Item class to include a DoubleSided setting.
2. Add a new MaterialPropertyOutputType "Special", which can be used for bespoke features like setting a double-sided mode. The field name will indicate the type of special setting, in this case the field name value will be "DoubleSided".
   1. Add a new MaterialTypeAssetCreator::ConnectMaterialPropertyToSpecialSetting function
   2. Update Material::SetPropertyValue to pass the value into the ShaderCollection::Item
3. Update MeshDrawPacket to set the CullMode to None, if ShaderCollection::Item::GetDoubleSided is true.
4. Update the doubleSided property in GeneralCommonPropertyGroup.json to use this feature.

Delete a bunch of code

1. ShaderCollection::Item::Get/SetRenderStatesOverlay and all code that references it.
2. ShaderCollection::Item::Get/SetDrawListTagOverride and all code that references it.
3. Render state utility functions like MergeStateInto,MergeValue, GetInvalidRasterState. I'm not sure if these might be used in other places though, we need to check. They were originally added to support ShaderCollection::Item, but I see some of this code referenced in PipelineStateForDraw
4. All the render state code in LuaMaterialFunctor like LuaMaterialFunctorShaderItem::GetRenderStatesOverride and the LuaMaterialFunctorRenderStates class.
5. LuaMaterialFunctorShaderItem::SetDrawListTagOverride

(Optional) Optimize by removing redundant shader code compilation

At this point, we'll have three separate .shader files for the forward pass, for blended transparency, and for tinted transparency. However these three files could still be using the same .azsl source file. This would mean the shader asset builder is compiling the exact same shader code three times. Instead, we could improve the .shader file to allow it to reference another .shader file and provide its own blend states (or any settings not related to shader compilation). The shader system would then be able to reuse the compiled shader bytecode from the other shader asset instead of compiling it again. Probably the cleanest way to achieve this would be to separate the bytecode into a ShaderBytecodeAsset rather than storing it directly in a ShaderAsset. That way, multiple ShaderAssets could reference the same bytecode.

However, this improvement is optional, and maybe not even necessary if the shader code diverges anyway. Once we split the forward pass, blended transparency, and tinted transparency into separate .shader assets, we might find that it's cleaner to have separate .azsl files with code that's specific to each use case and the expected render targets.

What are the advantages of the feature?

• Mitigates the complexity of eventually supporting offline PSO compilation.
• Users will be less likely to use the material system in unexpected ways, creating unforeseen maintenance burdens.
• Simplifies the material system design.
• Material types will be easier to understand if we remove the complexity of changing a shader's draw list and render states dynamically.

What are the disadvantages of the feature?

• It could result in extra shaders to compile, unless we take steps to support reuse of identical compiled shader bytecode.

How will this be implemented or integrated into the O3DE environment?

• No special integration procedure is needed, beyond the implementation details described above.

Are there any alternatives to this feature?

The system could do some static analysis of the material type to determine which render states could change to which values. That way, the (future) offline PSO compiler can compile all possible PSOs for each shader. But this would be difficult to implement, it's easier to provide structured data rather than conjure reflection data.

Or render state information could be reflected by the material type, just listing alternate render state combinations that the lua functors might achieve. This is similar to the proposed solution of having multiple .shader files that configure different render states; in both cases the material type author has to manually provide separate render state values for each possible configuration. The advantage here is that material types could reuse shader bytecode, avoiding redundant compilation, without any changes to the shader system. But the potential problem is that material type authors could create some complex unforeseen scenario where there are many combinations of different render states, so the list becomes unwieldy and difficult to align with what is implemented in the lua material functor.

Main alternative: The .shader file could define a named list of alternate render state configurations that are available. The material would not be allowed to set individual render states, but it would be allowed to select one of the shader asset's published configurations. For the transparency mode thing, the forward shader would have render state configurations for "Default", "BlendedTransparent", and "TintedTransparent". There would only be one .azsl file, the render states could be changed as a predefined unit, and the offline PSO compiler would have information about what PSOs are needed, to support any dynamic changes coming from the material. The shader code would only be compiled once, unlike the original proposal. We could also combine CullMode into this solution, so there would be an expansion of the configurations like "TintedTransparent_SingleSided" and "TintedTransparent_DoubleSided" ... or CullMode could still have special handling like described above, this would need more thought. The approach still has the disadvantage of requiring the draw list override feature, which impacts RenderAttachmentConfiguration and MultisampleState and therefore complicates the determination of which PSOs need to be compiled offline. We could do a hybrid approach where draw list override is disallowed, so the same shader can't be used for both forward and transparent, but at least the multiple render state configurations would allow the same shader asset to be used for BlendedTransparent and TintedTransparent.

We thought about moving the double-sided setting to be controlled through the MeshComponent rather than on a per-material basis. That would make CullMode be a "system render state" similar to RenderAttachmentConfiguration and MultisampleState, and use special code paths outside the material system. This would allow us to avoid having a special case in the material system for CullMode. But there are some problems with this idea. It would complicate the MeshComponent because it would need a double-sided setting for each mesh in the model. Also, users have come to expect double-side to be a per-material setting based on how other engines expose this.

How will users learn this feature?

The material system documentation will be updated to describe the "Special" "DoubleSided" connection that's available to material properties, and to reflect the changes to the lua material functor API.

Are there any open questions?

• The "main alternative" listed above is pretty compelling. Should we switch over to an alternate RFC where that alternative is the primary proposal?

[jeremyong-az]

For either the primary method proposed or the alternative, the main idea of moving all PSO state away from runtime mutable material state to a statically configured location strikes me as a good idea.

Configuring that state in the shader file seems pretty convenient, and makes me think variants should be considered in the same way. Previously, there were restrictions that prevented this consolidation (as I understand it), but with intermediate assets, perhaps we should consolidate both variants and material PSO state under the singular shader asset and simplify the process of shader creation in general