Alpha to Coverage implementation needs work

[clayjohn]

Godot version

4.2 beta 4

System information

N/A

Issue description

While fixing a regression in A2C I noticed a few things that are implemented sub-optimally and should ideally be fixed.

1. A2C has enabled blending despite using A2C for alpha. The blending operation is likely ignored by the driver, but it might not be, so we might be spending performance on something that doesn't change the visual output.
2. A2C meshes currently rely on a conservative depth prepass (the exact same as deapth_prepass_alpha) which discards everything with less than 0.99 alpha. Then they are rendering in the alpha pass with A2C enabled. This means that we only get per-pixel depth sorting for the fully opaque sections and the partial coverage pixels don't get depth sorted at all. This partially defeats the purpose of A2C. A huge benefit of A2C is that the mesh can be rendered in the opaque pass and have proper per-pixel depth sorting, including the coverage samples. There is no reason to use the transparent pipeline at all. (There are technical limitations to fixing this that I will explain below)

With respect to number 2, the problem is that alpha to coverage requires us to output an alpha value. It piggybacks on the alpha value that we use to write to coverage instead. Currently this doesn't work because the depth prepass doesn't have a color attachment, so there is no alpha to write to! I can think of two ways to solve number 2, each comes with pros and cons.

Depth prepass

Maintaining the depth prepass would be nice as it means that A2C materials can be included in SSAO, GI, etc.

We would need to attach a dummy color attachment to the depth prepass in location 0. Then when using alpha to coverage we would write out the alpha to that dummy color attachment. When not using alpha to coverage, we would bind the color attachment, but it would never be written to, so the shader will optimize it out.

A benefit of this is we might be able to only do the coverage calculation in the depth prepass. During regular rendering we only render when the depth is equal to the depth sample, so invalid pixels/samples will be rejected by the depth prepass.

No depth prepass

We could also just never pass A2C materials to the depth prepass and force depth writing on them always. This would solve the depth sorting issue, but it would mean that A2C materials would not be included in SSAO or GI. We would have to force the forward GI path for A2C materials (like we do for transparents) to include them in GI. They would, however, still work with SSS and SSR. So it's just SSAO that would be problematic.

Give up

Its possible that neither of the above "correct" solutions are preferable. In which case, we can modify our current approach to be better.

Even if we can't properly fix 2, using the depth_alpha_prepass with a threshold of 0.99 means that any portion of the mesh that is between 0.99 alpha and whatever alpha is used in the alpha scissor / alpha hash will be rendered in the transparent pass only and won't be included in the depth test. Meaning they will not have per-pixel sorting. This is really bad for foliage like trees and bushes which require per-pixel sorting for internal depth conflicts. If we can't implement 2 properly, then we at least need to use a less conservative depth prepass and capture all pixels with full coverage.

This option loses the benefit of alpha to coverage as the coverage samples won't have corresponding depth samples. But it does properly support SSAO. GI will run in the forward path still so it will work, but it will be needlessly costly. Plus this comes with the downside of being part of the transparent pass.

Steps to reproduce

N/A

Minimal reproduction project

N/A

[Calinou]

Maintaining the depth prepass would be nice as it means that A2C materials can be included in SSAO, GI, etc.

Does this also include their ability to cast shadows? Given A2C is often used for foliage, we want to ensure it's possible for it to cast shadows.

[clayjohn]

Does this also include their ability to cast shadows? Given A2C is often used for foliage, we want to ensure it's possible for it to cast shadows.

Ya, writing to depth is necessary for shadows. Currently it does cast shadows, but they don't match the size of what the user sees (similar to depth_prepass_alpha)