Question about Neural Radiance Cache

[ib00]

Thanks for your implementation of this algorithm!

Beginner's question: I am trying to understand the training part of the NRC algorithm. I have read the paper several times and it's still not clear to me.

Say, you have a path x0 x1 x2 (x0 - camera, x1 - surface, x2 - surface). How do you train the cache?

1. Do you explicitly sample lights at each intermediate vertex, to get paths x0 x1 l2 (where l2 is now vertex on a light source) and x0 x1 x2 l3 and then put end vertices (in this case x1, and x2) and direction (x2 -> x1, and x1 -> x0) into cache?
2. How does the "path extension" work?
3. There was an issue with "flickering" that was solved by temporal smoothing. How do you solve this problem?

[shocker-0x15]

1. Yes, it does NEE at each intermediate vertex to compute "direct lighting" component for the vertex. Additionally it uses implicit light sampling with MIS. For example, for x1, the program computes direct lighting contribution for this vertex from x0 x1 l2 (NEE, l.185- and l.531- in optix_pathtracing_kernels.cu) and x0 x1 x2 (implicit light sampling, when x2 is emissive, l.400- and l.638- in optix_pathtracing_kernels.cu). It does the same thing for the subsequent vertices x2, x3 and so on (i.e. for x2, direct lighting contribution from x1 x2 l3 and x1 x2 x3). After path tracing pass, NRC queries are issued for the single end vertex for each path, the result is back propagated (The term back propagation is not related to the same term used in a neural network context) through the vertices of each path to complete each vertex data's "indirect lighting" component. Once we have both direct and indirect components for each vertex, this is the target value used for training. Note that the end vertex of each path is not used for training since it consists of only NN output and useless for training NN.
2. Are you referring path extension loop? This works basically the same as normal path tracing (a path ends with Russian roulette), but it additionally possibly ends with path spread heuristic.
3. As the paper mentions, it uses EMA (l.54 in network_interface.cu).

line numbers are based on the commit 24827a47e5501f490b86f4e57b0f5d98d8d4557e

[ib00]

Thanks for clarification.

2. Path extension. Yes, are you terminating path based on spread?

A few more questions: a) In the new paper ("instant primitives"), they have some new encodings. I am wondering if it would make sense to store somehow entire angular distribution?

For example, if you shoot a ray from x0 to x1, instead of a single radiance value, you get distribution.

b) Could this radiance cache idea be extended to improve caustics rendering?

[shocker-0x15]

are you terminating path based on spread?

I just did what the paper explains. You can easily find in my code if you read.

a. I'm not sure I understood your question, NRC already represents radiance distribution. We can query at some point with various directions.

b. By using NRC at primary hit, it seems conceptually possible to return radiance field from caustics but we need to feed good training data than normal path tracing and need to develop a better heuristic. And directly representing high frequency caustics by NRC seems challenging. Something like path guiding instead of radiance cache would be useful.

[ib00]

NRC already represents radiance distribution. We can query at some point with various directions.

Correct me if I am wrong, but a query NRC(x, dir) gives you radiance at point X towards 'dir'.

My question was whether NRC can be changed so that you can get queries like NRC(x, dir, another_dir) -- give me radiance at x from direction 'another_dir' towards 'dir'.

Maybe this is a stupid idea, but then you may be able to quickly get full radiance distribution at x (for importance sampling -- by using RIS or something else).

[shocker-0x15]

give me radiance at x from direction 'another_dir' towards 'dir'.

You should train the cache which returns incident radiance for "another_dir" at the query point "x". "radiance at x from direction 'another_dir' towards 'dir'.": This can be obtained by multiplying BSDF and the cosine factor to the result. Moreover, this seems essentially the same as the NRC in the paper in the end by replacing the query point by a point found in "another_direction" from the "x".

[ib00]

I guess I am referring to the video (https://tom94.net/data/publications/mueller21realtime/mueller21realtime-gtc.mp4 at 24:20) when it's mentioned about using ReSTIR to importance sample radiance cache.

Not sure exactly what that means. Any thoughts?

[shocker-0x15]

I guess they talks about combining the NRC with something like ReSTIR GI (or ReSTIR PT for today). The original ReSTIR, say ReSTIR DI samples a good direction toward a light from which direction we can sample larger direct illumination contribution based on RIS. ReSTIR GI/PT (I haven't finished to read PT paper yet though) samples a good direction from which we can sample larger indirect illumination. DI is for many lights and GI/PT is for indirect illumination, those seems somewhat different but if we see indirect illumination as the infinite number of many lights, they look similar.

[ib00]

Another approach could using world-space ReSTIR (https://dl.acm.org/doi/abs/10.1145/3478512.3488613). The paper claims that it works better for indirect illumination than screen-space.