NRC Only Results (Directly Visualized Prediction) Appear Much Darker Than the Paper Suggested?

[ChikaYan]

Hi there, thank you so much for your great work!

I have a question when looking into the implementation of NRC, specifically, the immediate prediction of NRC network obtained via the "directly visualized prediction" option seems to result in a much darker rendering compared to the path tracing/path tracing + NRC. For example, below is the result of path tracing + NRC:

And the result of NRC only:

While according to the figure 4 of the original paper, visualizing NRC output at first vertex would give frequency artifacts in early frames (seems to be resolved with hash grid encoding tho), but the overall brightness should be the same:

When looking into the details of the implementation, I realized that the path tracing + NRC result is obtained by summing the direct light sampling through the next event estimation with the predicted radiance of NRC. However, the training target of NRC seems to already include both (direct contribution + indirect contribution from next NRC output). I'm quite confused by this, because if my understanding is correct, then it would be unnecessary to sum direct light sampling contribution to the rendering again and the NRC output should be much brighter. May I ask if you could provide any insight on this? Any comments would be much appreciated!

Many thanks.

[shocker-0x15]

Hello, thanks for having interest in my work.

I tried with two scenes. The following is combined screen shot for each scene, where the left and right 1/3 part is the direct visualization of prediction with tri-wave encoding (the encoding in the original paper) and hash-grid encoding (the one in instant-NGP paper) respectively and the center 1/3 part is a pure path tracing result. (Note that NRC doesn't include the emitting term in the rendering equation, it approximates the scattered radiance term.) I have to admit that there is some fluctuation in color and brightness, but basically the direct visualization of NRC seems to have similar brightness as the pure path tracing's one.

Regarding implementation, for example in the simplest case where NRC is used at the second vertex, path tracing + NRC result is obtained by summing the NEE at the first vertex and the NRC at the second vertex (divided by a probability density). NRC indeed includes a "direct" contribution, but that at the second vertex here, not at the first vertex. So this should not be double counting.

[shocker-0x15]

Note that the amplitude of scene radiance is important for faster and stable training because of the value range of half float used in tiny-cuda NN and particularly because of the epsilon term in relative L2 luminance loss function. If training seems unstable or too slow, try radiance scale slider in the application.

[shocker-0x15]

I guess you are using the model from McGuire Computer Graphics Archive. If so, look at the .mtl file. Basically every material uses a reflectance texture as map_Ke that is emittance texture. As the paper says, NRC approximates only the scattered radiance term and doesn't include the emitted radiance term. This is the reason why the direct visualization is much darker than the path tracing result.

Additionally, this scene's textures seem to be lit already. Therefore I would say this scene is not good for testing global illumination.

[ChikaYan]

Thank you so much for your quick replies and yes, you are exactly correct! The scene I tested the NRC on indeed includes many emitting textures, which NRC does not learn by design.

Again, thank you so much for your helpful insight :)