Normal maps render black at grazing angles

[chrisoffner3d]

Summary

Mitsuba shows black for surface points at which the (normal map-induced) shading normal faces away from the incoming (camera) ray.

System configuration

OS: macOS Ventura 13.0.1 (22A400) CPU: 2,4 GHz 8-Core Intel Core i9 GPU: AMD Radeon Pro 5500M 8 GB, Intel UHD Graphics 630 1536 MB

Mitsuba version: Mitsuba version 3.0.2 (master[https://github.com/mitsuba-renderer/mitsuba3/commit/47a3958efe01eb93e2499ab91522da5e151e4360], Mac OS, 64bit, 16 threads, 8-wide SIMD)

Description

It seems that Mitsuba 3 shows black for surface points at which the (normal map-induced) normal faces away from the incoming (camera) ray, possibly because we shouldn't be able to see a surface point with such a normal in the first place. The following images are all rendered with Mitsuba 3 with a normal map applied to the mirror sphere and the floating plane.

In Blender Cycles this happens only at a much steeper angle:

https://user-images.githubusercontent.com/18167754/207460094-398a3dfc-b17c-42e3-a513-4f4ef77bb982.mp4

Does anybody know why this behaves differently, and how Blender seems to handle this more robustly?

The example image in Mitsuba 3's documentation doesn't seem to suffer from these artefacts, so I'm wondering if I'm doing something wrong in applying the normal map. 🤔

(Example image for normal maps from Mitsuba 3's documentation)

[njroussel]

Hi @chrisoffner

Could you share the scene you have in Mitsuba? Specifically the texture and the BSDF of the rectangle.

[Vrroom]

Does anyone have the normal map for the Mitsuba example? Trying to reproduce it. Thanks!!!

[Vrroom]

@njroussel @chrisoffner I tried a trivial example with the following normal map (the result should be as if the normal map is not there). But my renders have this same problem.

Render:

Normal Map (trivially Blue everywhere)

Scene XML.

<scene version="3.0.0">
<default name="spp" value="64"/>
<default name="width" value="683"/>
<default name="height" value="512"/>
<default name="max_depth" value="-1"/>
<integrator type="path">
<integer name="max_depth" value="$max_depth"/>
</integrator>
<sensor type="perspective" id="camera">
<string name="fov_axis" value="smaller"/>
<float name="focus_distance" value="6.0"/>
<float name="fov" value="28.8415"/>
<transform name="to_world">
<lookat target="3.04072, -2.85176, 2.80939" origin="3.69558, -3.46243, 3.25463" up="-0.317366, 0.312466, 0.895346"/>
</transform>
<sampler type="independent">
<integer name="sample_count" value="$spp"/>
</sampler>
<film type="hdrfilm" id="film">
<integer name="width" value="$width"/>
<integer name="height" value="$height"/>
<string name="pixel_format" value="rgb"/>
<rfilter type="gaussian"/>
</film>
</sensor>
<emitter type="envmap" id="emitter-envmap">
<string name="filename" value="envmap.exr"/>
<transform name="to_world">
<rotate y="1" angle="-180"/>
<matrix value="-0.224951 -0.000001 -0.974370 0.000000 -0.974370 0.000000 0.224951 0.000000 0.000000 1.000000 -0.000001 8.870000 0.000000 0.000000 0.000000 1.000000"/>
</transform>
<float name="scale" value="3"/>
</emitter>
<bsdf type="diffuse" id="bsdf-diffuse">
<rgb name="reflectance" value="0.18 0.18 0.18"/>
</bsdf>
<texture type="checkerboard" id="texture-checkerboard">
<rgb name="color0" value="0.4"/>
<rgb name="color1" value="0.2"/>
<transform name="to_uv">
<scale x="8.000000" y="8.000000"/>
</transform>
</texture>
<bsdf type="diffuse" id="bsdf-plane">
<ref name="reflectance" id="texture-checkerboard"/>
</bsdf>
<!--  <bsdf type="tinted_dielectric_bsdf" id="bsdf-matpreview">
        <vector name="tint" value="0.240, 0.771, 0.361" />
        <float name="eta" value="1.33" />
    </bsdf>  -->
<bsdf type="normalmap" id="bsdf-matpreview">
<texture name="normalmap" type="bitmap">
<boolean name="raw" value="true"/>
<string name="filename" value="identity_1k.jpg"/>
</texture>
<bsdf type="roughplastic"/>
</bsdf>
<shape type="serialized" id="shape-plane">
<string name="filename" value="matpreview.serialized"/>
<integer name="shape_index" value="0"/>
<transform name="to_world">
<rotate z="1" angle="-4.3"/>
<matrix value="3.38818 -4.06354 0 -1.74958 4.06354 3.38818 0 1.43683 0 0 5.29076 -0.0120714 0 0 0 1"/>
</transform>
<ref name="bsdf" id="bsdf-plane"/>
</shape>
<shape type="serialized" id="shape-matpreview-interior">
<string name="filename" value="matpreview.serialized"/>
<integer name="shape_index" value="1"/>
<transform name="to_world">
<matrix value="1 0 0 0 0 1 0 0 0 0 1 0.0252155 0 0 0 1"/>
</transform>
<ref name="bsdf" id="bsdf-diffuse"/>
</shape>
<shape type="serialized" id="shape-matpreview-exterior">
<string name="filename" value="matpreview.serialized"/>
<integer name="shape_index" value="2"/>
<transform name="to_world">
<matrix value="0.614046 0.614047 0 -1.78814e-07 -0.614047 0.614046 0 2.08616e-07 0 0 0.868393 1.02569 0 0 0 1"/>
<translate z="0.01"/>
</transform>
<ref name="bsdf" id="bsdf-matpreview"/>
</shape>
</scene>

[Vrroom]

@njroussel @wjakob,

Following up on this issue. Please let me know if I can provide any more information to help debug this issue? I'm using the mitsuba version available through pip (v3.5.x).

[njroussel]

Hi @Vrroom

I always thought this was just a expected behavior because the shading frame normal could define a back-facing surface in some cases. However, as you pointed out, an all-blue normalmap should have no effect. This should be enough to start debugging this issue, thanks. It will take some time until we get around to this, we're currently working on a large update to the backed & python bindings.

[joeylitalien]

+1 for this, I'm also experiencing the same issues in my scenes. I think it might also apply to bumpmap as I see similar black artifacts when using a displacement texture.

[Vrroom]

@joeylitalien Can you also check on an identity operation (a bump map setting that doesn't bump)? BTW you neural LOD work looks very cool!

[vid8687]

@njroussel @Vrroom I've been observing the same issue and noticed that mitsuba treats (0.5,0.5,1) i.e., x->(x+1)/2 mapping as the trivial blue solution everywhere (with normal texture set to raw=true in the scene description, and a tensorxf updated to have those values setup in the normal texture data). What mapping does mitsuba expect on its raw normal data ?

[BoyeGuillaume]

I ran into the same issue when writing my own renderer (for a course given by @wjakob). The issue arise from the fact that normal map are faked texture that act as geometry. Due to this, it can sometimes happens that the perturbed normal is facing in the opposite direction than the original unperturbed ray. This would not happen in a "perfect" physically based ray tracer because those piece of geometry would be occluded by other piece of geometry facing the correct way.

The fix I've camed up with (for my renderer) was to invert the Z axis of the local frame when applying the normal map flip the backface (aka. was not a backace become a backface after normal map). This very hacky fix was motivated by the fact that, as I said earlier such events should not be possible in a purely physically based renderer.

The cycle implementation of blender perform a similar "trick" to fix backfacing normal map.

Sadly I wasn't able to port my fix to mitsuba for when I try to compile it I get El-Famoso blue screen of death (for some reason).