Investigate TRAA for retaining samples on change

[bhouston]

Oops I created a duplicate -- I will merge that new issue into this one: https://github.com/gkjohnson/three-gpu-pathtracer/issues/191

[bhouston]

Is your feature request related to a problem? Please describe.

Temporal reproduction anti-aliasing is a method of reusing samples from previous frames on the current frame but warping them based on the scene/camera movement that occurred. While this renderer currently does accumulate samples across frames, it can not handle motion in those frames. Implementing TRAA will enable it to handle motion.

Describe the solution you'd like

Something akin to this: https://www.youtube.com/watch?v=2XXS5UyNjjU

I believe naive approaches do not handle transparent backgrounds because those are not required in games. This is okay for the first implementation, but a further improvement would be to handle transparent backgrounds.

[bhouston]

I can offer a bounty of $500 USD for a merged PR that implements this feature.

[richardassar]

I remember this talk and I'd been thinking about this in recent days, for static scenes an idea came to mind which I'm not sure has been tried before: output from the path tracer can be used to train a NeRF network, baking the scene into the network weights and hash table.

Online demo of NeRF networks running in the browser

Although that's not what this issue requests I think it would be interesting to try. There's no need for camera rays to be coherent also since they're independent, so you can just sample random rays within a given volume and get good scene coverage.

This approach is somewhat related to the much more complex Real-time Neural Radiance Caching for Path Tracing which makes use of Q-learning, probably a step too far for WebGL.

For retargeting I think having an approximate differentiable renderer could be an interesting approach, as you can quickly reject samples that fall within some bound of imperceptibility given the (possibly approximate) partial derivative of the final output with respect to all scene parameters.

Differentiable Monte Carlo Ray Tracing through Edge Sampling

Differentiable Path Tracing on the GPU/TPU

Path-Space Differentiable Rendering

Differentiable renderers aren't very fast but there may be ways to approximate this alongside the main monte carlo integration.

It should be the case that a return; from a fragment shader yields relatively quickly to other fragments, better would be parallel reduction over the thresholded gradient image that is then read by a compute shader that individually dispatches only those rays, writing out only those pixels with imageStore.

Lacking imageStore, which can be used from other pipeline stages, in WebGL means having to hack it with GL_POINTS, I used this trick in the Electric Sheep IFS renderer I'd written many years ago and I think that approach could be useful here for the scattered writes required in the reprojection step.

[bhouston]

Well, I would love a differentiable renderer. :). But I think that this is a longer term goal.

I've implemented classic TRAA before and I just had a full screen quad which used a velocity field buffer to do look ups backwards to where things used to be and to bring those forward. So it isn't a scattered write, but rather just a structured write with a texture lookup into the previous frame buffer via an offset (the per pixel velocity.)

I have a PR on Three.js that creates a full screen quad of per object velocities for rigid objects + camera motion. It can be extended to deformable objects, but I think that is out of scope for the first pass: https://github.com/mrdoob/three.js/pull/23784

[bhouston]

I love the IFS renderer - this is beautiful: https://richardassar.github.io/ElectricSheep_WebGL/. I did some similar things a decade ago for some Hollywood films via a renderer I wrote called Krakatoa in combo with Doc Bailey's fractal system: http://www.imagesavant.com

I think that future steps should be NeRF or similar. Did you see this new paper that shows non-NN radiance field approaches are even faster https://alexyu.net/plenoxels/

I view this as a much more advanced feature that can wait a few months. I think we still need to get this to production grade first for just images and then take it to the next level -- differentiable rendering + radiance fields, etc. would be absolutely amazing, because that is what the future needs.

[richardassar]

That looks awesome, I've been using differentiable rendering to generate mipmapped multichannel signed distance fields for font rendering (for VR) in a non-NN approach. MSDFGEN uses a median filter over the R,G,B channels but it's wasteful and uses only one texture lod, using differentiable rendering I was able to propagate gradients into a mip-pyramid, so you get sharp text at grazing angles and the memory coherence benefits. I've got some nice results, as yet unpublished, but I think it's promising. The problem there is how to scale "training" up to large character sets e.g. Chinese. The only annoying thing there is that no spec or vendor explicitly define their anisotropic filtering approach, so nobody has bothered to incorporate that into the model, it would improve things further but the only recourse there is just reverse engineering it. Anyway, fonts is a bit off-topic, but I was just speaking to the validity of non-NN approaches.

I think a finite difference approximation to dBuffer/dParameters could be used to dispatch "difference rays" which sample and apply the delta, not the objective value itself as in standard path tracing. That way if you have a static scene but change e.g. the directional light color you can update the affected pixels within perceptual threshold. Overall the number of pixels required to update through a discontinuity is low so shouldn't contribute much to the cost.

Some other ideas come to mind involving ideas from texel-space shading, using a giant UV unwrap of the scene as an irradiance/reflectance cache and storing information that can contribute to earlier ray termination. The UV scaling could be reduced by some heuristic involving roughness (ray incoherence), number of bounces etc. and dynamically updated each frame as cameras and objects within the scene move. In this way, when you're reprojecting or trying to fill holes when a foreground object moves, you can just integrate a small number of cached values quickly.

[richardassar]

I love the IFS renderer - this is beautiful: https://richardassar.github.io/ElectricSheep_WebGL/. I did some similar things a decade ago for some Hollywood films via a renderer I wrote called Krakatoa in combo with Doc Bailey's fractal system: http://www.imagesavant.com

The guy behind JWildfire contacted me to ask if he could use ElectricSheep_WebGL, and he made this with it https://ancient-fortress-28026.herokuapp.com/gallery (on linux this only works for me in Chrome btw)

IFS are an interesting candidate for denoising and I'd tried some approaches, prior to NeRF but in the same spirit, for denoising / caching / compressing / approximating them for realtime rendering.

[bhouston]

On a related note, I've put out a bounty of $1000 USD for an implementation of TRAA there as an accepted PR. Maybe TRAA on Three.js could share some implementation elements to with this project... https://github.com/mrdoob/three.js/issues/14050

[gkjohnson]

Just to cut down on redundant work - @0beqz has been in touch is working on this! I'm very excited for the work.