search

BachiLi / redner

Differentiable rendering without approximation.
https://people.csail.mit.edu/tzumao/diffrt/
MIT License
1.37k stars 137 forks source link

Use samplers as input, not textures #118

Open nkyriazis opened 4 years ago

nkyriazis commented 4 years ago

It would be nice if the user could provide a (differentiable) sampler rather than a texture, wherever required. This way, the user can control the level of detail along with the embedding required. Environment maps and texture maps, when required to be mapped to 2d surfaces, cannot accommodate uniform resolution, which might require too much memory to counter with textures.

BachiLi commented 4 years ago

This requires a rather significant change to render, but perhaps we can make the texture access code in C++ more modular and people can write their samplers in C++ (or provide dynamic libraries). I need to think about how to implement this. Suggestions are welcome.

nkyriazis commented 4 years ago

This was based on the assumption that there is a single point where the relevant textures are sampled, in batches. The idea is for this sampling to become a callback, instead, with input the relevant information, e.g. the incoming ray and the intersection point.

The callback's default implementation is the existing code the callback would replace. For pytorch, providing a differentiable callback should be straightforward, as the caller needs to set-up the tensors and call the python code to get the samples back. For the C++ I'm not sure.

I have not seen the code in the relevant points. Perhaps a few pointers from your side on where to look could help me get a better understanding and perhaps propose something better.

nkyriazis commented 4 years ago

Again, this is towards uniformity of resolution and memory usage rationalization. Perhaps there are other ways to achieve those I'm not aware of.

A relevant question is, why use mipmapping in textures?

BachiLi commented 4 years ago

The code that does texture sampling is in src/texture.h which is mostly used by src/material.h. We're using mipmapping because prefiltering is important in both forward/differentaible rendering. Consider a case where a pixel contains a surface with 4k by 4k texture -- you'll need significantly more Monte Carlo samples to render that pixel if you don't use prefiltering. This is even more important in gradient computation.