search

pmh47 / dirt

DIRT: a fast differentiable renderer for TensorFlow
MIT License
313 stars 63 forks source link

Order independent transparency? #9

Open ziriax opened 5 years ago

ziriax commented 5 years ago

First for all thanks for making this open source! For a 3D veteran like me, these differentiable rendering engine are very exciting. I'm a compleet newbie in AI, so I apologize for asking stupid questions.

I don't understand exactly how DIRT works - I most likely need to re-read your paper and all references 100 times, meaning I need another life ;-) And I have absolutely no idea how to integrate this module into a tensorflow graph, but I guess this will become clear the more I learn about Tensorflow. I do have a very specific problem that I want to solve, so that is a good starting point.

I was wondering if it would be possible to incorporate order-independent-transparency in this renderer?

So basically instead of a pixel being obscured, every pixel is a linear combination of N weighted depth sorted points on the ray through that pixel, but done in hardware (DirectX has rasterizer order views for this, and OpenGL might support this with extensions, but basically the GPU hardware can deal with this)

Would this approach make a DR more suitable for back-propagation? Because after watching the OpenDR presentation, it seemed the edge pixels are problematic because you need to track the pixel underneath. Order independent transparency could help here?

Another interesting approach would be statistical moment-based shadow mapping and order independent transparency, but I don't know enough of these new techniques.

And of course supporting Physically based rendering would be insane ;-)

pmh47 commented 5 years ago

So far as I understand it, order-independent transparency requires keeping track of all the fragments hitting a pixel, including their occlusion ordering, in order to calculate the blending correctly; as such, it is strictly more complex than regular occlusion with z-buffering. One could modify DIRT to support this, but it'd be quite complex! If one uses a non-physical light-transmission model for the non-opaque surfaces that does not depend on the z-order (e.g. just multiply all the transmissivities together along a ray as in volume rendering), then it becomes significantly easier, but one still has to define derivatives at the edge of triangles.

As for differentiable PBR, redner already does this!