Normal Maps with large height gradients are not getting applied correctly

[fuzzypurplepixel]

Godot version: Godot 3.2.3 and 3.2.4-beta3

OS/device including version: Windows 10, Nvidia GTX 960 GLES2 GLES3

Issue description: The normal file in question is labeled Asteroid_DeepNormal.png as that has some height gradients. I also have an Asteroid_SoftNormal.png file that I made with softer gradients. The issue is I want some large height gradients for my normal maps to give an effect of deeper indents in my asteroids. I could see this have a use for other game objects as well.

The issue exists for GLES3 and GLES2 for both Sprite, Sprite3D, and MeshInstance. Parts of the sprite get shaded incorrectly when using normal maps that have large ranges in values. The issue does diminish when using the 3D methods and setting the normal scale to 0.5 or less. I have confirmed my normal map files are valid to work by testing on the Unity game engine.

Note about Godot Master branch: this issue is only for GLES2 and GLES3 when using a MeshInstance/Sprite3D as Sprite2D does not have a normal map option when making the issue using Godot Master branch with Volkan. I also can't seem to test the GLES2 on the master branch at this moment.

Steps to reproduce: 3DScene and 2DScene files are provided and should be runnable. It does seem that if the normal scale is set to 0.5 most of the issue is no longer visible.

Minimal reproduction project: TestNormals.zip

[clayjohn]

How are you generating the normal maps? Also are they HDR? it looks like the deep_normal texture might be HDR.

Also note it could be a precision related issue. The normal maps looks significantly better when they are uncompressed.

[fuzzypurplepixel]

How are you generating the normal maps? Also are they HDR? it looks like the deep_normal texture might be HDR.

Also note it could be a precision related issue. The normal maps looks significantly better when they are uncompressed.

The normal maps are generated with tweaking in the Laigter tool. If you know how to use the tool this is a copy of my presets Presets.txt

I also have made a new version of the normals. I do run my images through a compressor before using them. This one I got right out of Laigter. This issue was still occurring before I was using the compression tool.

Another note. The images I provided are for testing with Godot only(don't redistribute them for any other purpose). I have designed them and plan on using them in my own commercial product.

[fuzzypurplepixel]

Forgot to turn on the Invert y axis

[lawnjelly]

Sounds similar to #43309? (not that it got fully investigated)

It does seem a lot more pronounced with diffuse mode burley, with lambert you can't see it much. Perhaps it is just lack of precision in an area that is emphasised by burley. It would be nice to compare it with a normal map that isn't compressed with godots 2 channel compression, just to check that isn't introducing the problem.

There could be a threshold value somewhere in the shader where the lighting normal is causing one path or another, which is causing this dithering instead of smooth change.

[lawnjelly]

I made a test doing normal mapping type operation manually (just using a dot product of a light vector and the texture normal), and there were no problems with the texture. So it does suggest either the compression or the math in the shader.

normal_map_manual.zip

[lawnjelly]

Looks like it is the texture compression method. Same as before, except this time the shader loads the normal compressed as RG8, and uses the same math to derive the z of the normal as in the standard shaders. Result - horrible dithering.

[lawnjelly]

This can be fixed in the shader. The correct version should be:

    normal.z = sqrt(1.0 - clamp(dot(normal.xy, normal.xy), 0, 1));

or

    normal.z = sqrt(max (0.0, 1.0 - dot(normal.xy, normal.xy)));    

may be marginally cheaper.

I may be able to do a quick PR before I have to leave today.

The result is not perfect but is a lot better. It may be that it is not possible to improve this more without having an option for uncompressed normal map.

[fuzzypurplepixel]

This image is how it is rendered in Unity. Unity does have some more complex lighting options for 2D like a spotlight so it does have a higher contrast from the light source

[lawnjelly]

Interestingly, a lot of the dithering nastiness due to compression disappears when adding a bodge factor to the minimum. I'm not exactly sure why.. when x and y are 0, the tangent normal is pointing to the bottom left (or some corner) and the upward component is zero. Perhaps there's a large variation in normal direction in this region due to the sqrt function, leading to more pronounced dithering:

Using 0.0: Using 0.01:

normal_map_manual_test2.zip

[fuzzypurplepixel]

normal.z = sqrt(max (0.03, 1.0 - dot(normal.xy, normal.xy))); removes visible noise clumps. I have not been coding many shaders, is it possible to print the values of normal.z?

[lawnjelly]

normal.z = sqrt(max (0.03, 1.0 - dot(normal.xy, normal.xy))); removes visible noise clumps. I have not been coding many shaders, is it possible to print the values of normal.z?

It's not easy getting data out of a shader. Often all you can do is write data as colour, then screenshot and analyse the colours (while taking into account any colour changes due to tone mapping etc). Perhaps there is some debugger that would help with this with e.g. a software rasterizer but I'm not aware of one.

You can duplicate the shader on the CPU, but it is hard to emulate the effects of precision (I kept meaning to try and have a go adding this to swiftshader or ask them to add it).

[lawnjelly]

In your example project to minimize problems you would select the import tab for the normal map and change compress mode to lossless. Texture compression (in addition to the fixed 2 channel compression) makes the artifacts worse. I'm not sure where the sprite 3d normal maps are converted, ah yes, here:

#if defined(ENABLE_NORMALMAP)

    normalmap.xy = normalmap.xy * 2.0 - 1.0;
    normalmap.z = sqrt(max(0.01, 1.0 - dot(normalmap.xy, normalmap.xy))); //always ignore Z, as it can be RG packed, Z may be pos/neg, etc.

    normal = normalize(mix(normal, tangent * normalmap.x + binormal * normalmap.y + normal * normalmap.z, normaldepth));

#endif

I just tried applying the bodge here in the scene shader, and what do you know, it also clears the artifacts. I'll take a screenshot.

I'll talk this over with @clayjohn. It is possible we could just apply this bodge in the shaders to deal with this problem. It's a little bit naughty, as it is not 100% physically correct, but hey ho. On the other hand there may be a bug in the compressor which is causing all this which would be a better fix, if that were the case. Or even the input data. If the input normal were pointing downward I don't know what would happen.

[fuzzypurplepixel]

I was able to build godot 3.2 and it seems like 2D is working. Are you still waiting to push the 3D fix

[lawnjelly]

Are you still waiting to push the 3D fix

Yes, I have been away from home. It's not just making a PR, it's discussing it with clayjohn and reduz and checking over the compression code to check there isn't a bug there, or a more appropriate way to address this. In general we don't like to make bug fixes until we are sure it is the right fix. Better to be patient and get the right fix than to rush something through and find it breaks something else.

[clayjohn]

I wonder if using a more correct encoding/decoding method would be enough. see https://aras-p.info/texts/CompactNormalStorage.html

[clayjohn]

I took another look at this as I was porting the fix to Godot 4.0. The source normalmap is not normalized. So at certain points the length of the normal far exceeds unit length. To save on bandwidth, Godot ignores the z channel and instead reconstructs it from the x and y channels. I'm not sure that this optimization makes sense as for many drivers a texture read will read all 4 channels from the source texture anyway. In 3D, we optimize the normal map to an RG texture, so we know we aren't wasting bandwidth (we also validate the normals at convert time). For 2D we just use the source normalmap and trust that it is proper - thus leading to issues like this one.

As a workaround for users facing this issue you can insert the following into your fragment shader:

NORMAL = texture(NORMAL_TEXTURE, UV).xyz;
NORMAL.xy = NORMAL.xy * 2.0 - 1.0;

This will read your source normals directly and should result in nicer looking results.

Godot 3.5 Godot 3.5 with a bodge of 0.3 as described above _Godot 3.5 reading z from the NORMALTEXTURE as described here