Create and expose a `layers` property on `BaseMaterial3D`

[thompsop1sou]

Describe the project you are working on

I've been working on an example project that showcases how to use cameras/viewports to create custom screen buffers: https://github.com/thompsop1sou/custom-screen-buffers

Note: I know this isn't the best solution for several reasons (performance, viewport texture precision, etc). And I know the team is already hard at work on the rendering compositor. However, I think this is a good enough workaround while we wait for the compositor to be implemented.

Describe the problem or limitation you are having in your project

I'd like to be able to make this project more accessible for users that aren't as familiar with shaders. I may even try to turn it into a plugin that's really straightforward to use. To that end, I want users to be able to use StandardMaterial3D or ORMMaterial3D as the base material for their 3D objects.

Note: To be clear, this proposal is in reference to the "modular" approach that is used in my project. In that approach, I have it so that the shaders that display depth info and normal info are chained on the next_pass property of the base material. This means that the base material can theoretically be switch out for any other material.

The issue is that, currently, there is no way to get around having to use shader materials for every object in the game. This is because the workaround requires using CAMERA_VISIBLE_LAYERS to determine when to render the regular, base material versus when to render depth info, normal info, or some other custom buffer.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

All that would be required to remedy this issue is to create and expose a layers property (or something similarly named) on BaseMaterial3D (which would also affect StandardMaterial3D and ORMMaterial3D, if I understand the inheritance correctly). This would allow for rendering that material only on the indicated layers.

I also imagine that this could be a useful property to have in other visual effects scenarios. For example, if someone was implementing an x-ray or thermal camera (like Metroid Prime's visors), they could simply use different materials on different visual layers.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

From an end-user perspective, all that would change is that there would be a new layers property available to edit in StandardMaterial3D and ORMMaterial3D. I imagine it would look just like a copy of the layers property that currently exists on VisualInstance3D nodes:

Since there would be a performance cost with enabling this property, there would probably need to be a flag to enable it first (much like many other properties in BaseMaterial3D).

Once enabled, the native shader code would need to change. I'm not sure what that would require in the C++ code for BaseMaterial3D. But the end result should be that there are a few extra lines of code in the generated fragment shader. I believe something like this would work to discard the fragment when there is no overlap between the camera's layers and the material's layers:

if ((CAMERA_VISIBLE_LAYERS & layers) == 0u) {
    discard;
}

Note: Alternately, perhaps the material could be discarded in the vertex shader with something like this:

if ((CAMERA_VISIBLE_LAYERS & layers) == 0u) {
    VERTEX = vec3(1.0 / 0.0);
}

I don't know if this is a well supported way to hide geometry in a vertex shader, but it appears to work in my own shaders. I suggest it because it seems that it would be more performant than discarding every fragment.

Edit: Looking at the source code for BaseMaterial3D, I realized _update_shader() creates Godot shader code. So I've updated the code above to reflect that.

If this enhancement will not be used often, can it be worked around with a few lines of script?

It can only be worked around by writing a custom shader material. Not only is this not very accessible, but using a custom shader material is also problematic for those who are importing materials from other programs. For example, I know that Blender imports use StandardMaterial3D by default. As far as I know, it would require writing a post-import script to change that.

Is there a reason why this should be core and not an add-on in the asset library?

I don't know of any way to extend BaseMaterial3D (or its child classes) to add this feature. This is because the methods for updating the native shader code are not exposed to scripting.

[tetrapod00]

I'm very sympathetic to the problem being solved here but I feel like this likely falls into the class of things that https://github.com/godotengine/godot-proposals/issues/8366 and https://github.com/godotengine/godot/pull/94427 will solve. My guess is that if shader templates are implemented, new features for BaseMaterial3D will be a bit more disincentivized than they currently are.

This proposal is also essentially doing "culling" in the vertex or fragment shader, which strikes me as a bit of an antipattern especially if it's exposed with the same layer mask UI that actual culling uses.

[thompsop1sou]

Oh, yes, that PR probably would be a good solution. Sorry, i tried to look through open issues to see if there was one relevant to what i'm doing, but i missed that one. It would be great to have the layers functionality on the default material that everyone uses, but i can also understand not wanting to add too many features to BaseMaterial3D.

Regarding culling, i think you are probably right. I'm not very familiar with the rendering pipeline, so this is the only way i know of to cull on a per-material basis (rather than on a per-visual instance basis). That's the only reason i suggested it, and i'd be totally happy if there was a better way to cull per-material.

[tetrapod00]

Oh, it's perfectly fine to suggest alternate solutions to the same problem, and don't feel bad about not being able to find existing proposals among the 5000 open ones 😄 . A third approach is to what I believe is the same problem would be https://github.com/godotengine/godot-proposals/issues/10546, which would do the conditional rendering for different layers by adding a material override on a viewport or environment (they end up functionally very similar).

[thompsop1sou]

I went ahead and tried implementing this here: https://github.com/godotengine/godot/pull/99565

Totally understand if this is not something that makes sense to add to BaseMaterial3D. But I thought it would at least be helpful to have a working PR/branch.