Antialiasing with HDR light values produces incorrect result

[cheetahspots01]

Godot version

v4.0.alpha6.official [e4f0fc50f]

System information

any hardware, any backend

Issue description

Issue:

With HDR lighting values with intensity over 1, 3D antialiasing stops working. I believe this is because antialiasing is performed before tonemapping / clamping of values (it is performed scene-referred, rather than display-referred). Because this happens, very high values (>1) are averaged with lower ones (0-1), resulting in a value that is still very high and gets clipped. This issue happens in both Godot 3.x and the 4.0 alpha.

Examples:

The only difference between these two scenes is the intensity of the directional light. They both use ACES tonemapping. Both scenes have 8x MSAA enabled, although the issue occurs with all types of antialiasing.

Light intensity 1: Since all values in this image are within 0-1, all of the edges appear properly antialiased.

Light intensity 10: Note the correct antialiasing where the shadowed part of the cube touches the sky, since both parts were already within the 0-1 range. However, on the lit parts of the cube and the plane, the edges appear to have very little antialiasing due to the >1 values.

Steps to reproduce

1. Create a scene with some geometry and a light.
2. In the project settings, enable advanced settings and then turn on MSAA or FXAA.
3. Observe the edges of the geometry with the light at its default settings.
4. Set the light's energy value to something greater than 1.
5. Observe the edges of the geometry. They appear pixelated, as if there was no antialiasing.

Minimal reproduction project

No response

[Calinou]

I can confirm this on 4.0.alpha e9699dca0 (Linux, NVIDIA).

From what I've read, this is a difficult problem to solve as multiple tonemapping steps may be required (with a noticeable performance impact). See also the discussion starting from the linked comment: https://github.com/godotengine/godot/pull/52476#issuecomment-1081267814

PS: Please upload a minimal reproduction project to make this easier to troubleshoot (ideally one for master, and one for 3.x).

[cheetahspots01]

According to this article, it seems like MSAA is part of rasterization, in which case it would indeed be hard to apply tonemapping beforehand. As far as I can tell though, FXAA is applied very close to the end of the process, so it should be able to easily be applied after tonemapping.

[cheetahspots01]

If you're willing to sacrifice a bit of performance, it seems like there is a technique that works for MSAA. I found it in this blog post, but the basic idea is that you tonemap each MSAA fragment sample, and then undo the tonemapping after the samples have been resolved.

The problem is that, depending on the tonemapping operator used, the inverse operator may be computationally expensive, and lose precision in the bright parts. That blog post solves the precision issue by simply inverting the image before and after the MSAA resolve. A solution to the second problem (and perhaps the first one too), proposed in this thread, suggests using a simpler tonemap operator in the MSAA step, since the tonemap will be undone immediately after so it won't affect the final image's colors too much.

[clayjohn]

It looks like the popular solution used to be to do the tonemap/inverse tonemap in a custom resolve function. Godot 3.x just uses hardware resolve, but a custom resolve could be added (for desktop, not for mobile or web) in theory if there was significant demand. We could also run a separate tonemapping and reverse tonemapping pass before and after the resolve, but that would likely be prohibitively slow.