Golden image test: basics/decals fails randomly

[ghost]

Expected image:

This test usually passes, but every once in a while this image gets rendered instead:

This started happening after #440, but only on Github Actions. On M2 Max the decals example works as expected.

[FredrikNoren]

@VinhTruongAmbient Hm that's so strange with the weird grey box in the background in the second screenshot. Seems like it's failing almost every time on main now

[ghost]

I was able to reproduce this on my local Docker container which runs LLVMPipe. Investigating.

[ghost]

As a test I removed all entities, except for the camera and the decal. The result looks like this:

The gray box looks to be oriented by the decal, which makes sense. Removing orientation yields:

[ghost]

There seem to be two separate issues:

1. Our decals use CubeMesh to draw the context in which the decal is rendered. This CubeMesh is actually twice as large in every dimension. The shader expects a cube with min=[-0.5,-0.5,-0.5], max=[0.5,0.5,0.5] but our CubeMesh is actually min=[-1.0,-1.0,-1.0], max=[1.0,1.0,1.0]. 1.1. The above statement is incorrect because the transparent green cube uses the same cube as the decal. Something else however does double the size of the decal volume. 1.2. Actually, both extent=1 and extent=2 cubes are in play here. The green cube is the cube() component, which is extent=1 or the "unit cube". However, the decals code uses the regular "cube", which defaults to the extent=2 cube, but thinks it's using the extent=1 cube. This confusion of cubes caused the mismatch between the two.
2. get_decal() in decal shader calls discard; if the projected point does not land inside the unit cube. This is incorrect because in this case the fragment never gets written. So the gray background just happens to be some uninitialized render target data.

Working on a fix.

[ghost]

The uninitialized render target explains the random behavior we have seen in the CI.

[ghost]

Looks like the discard theory doesn't tell the whole story. This bug actually happens on my Mac too, you just had to change the clear color from black to something else:

We should change our default clear color to something else than pitch black. Helps reveal bugs like this.

[ghost]

Fixed in https://github.com/AmbientRun/Ambient/pull/473.

A summary of the problem:

• First, the decal shader expects a cube of extent=1, but instead got a cube of extent=2. This was caused by confusion about the two ways to define a "unit cube" inside the engine. This issue actually helped to reveal the more fundamental problem about the shader which is:
• The decal shader assumes that every pixel occupied by the decal projection volume (gray box) has a depth value that corresponds to some surface. This can be incorrect, if the projection volume is rendered on top of a freshly cleared depth buffer, where all depth values are cleared to 0. This assumption causes the "decal-insideness" test to pass, but then it leads to the sampling of non-existing normals from the screen_normal_quat render target. The correct behavior is to discard; these pixels.
• Now, since the decals test has passed incorrectly, the PBR shader moves on and takes this zero-length normal and shades the projection volume like a regular surface.
• Finally, you get the end result of the decal projection volume being shaded with incorrect shading parameters.