search

NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.37k stars 146 forks source link

Request for custom OpenGL shaders #96

Closed jenkspt closed 1 year ago

jenkspt commented 1 year ago

I would like to be able to pass my own shaders to OpenGL. This is a feature request for lower level access to the rendering pipeline -- similar to: https://github.com/tensorflow/graphics/blob/ed5c9acd2e7b8d44bb23cc0120acea74fdbb77ea/tensorflow_graphics/rendering/opengl/rasterization_backend.py#L183-L194

s-laine commented 1 year ago

Similar to TF Graphics, nvdiffrast always uses its internal set of shaders for which it has the gradient implementations available. User-specified shaders would need to provide the gradient code as well, so that the rendering can be differentiated. This would complicate the interface enormously, which is why the idea in nvdiffrast is that all shading is done after the fixed-function rasterization op using differentiable PyTorch/TensorFlow ops.

If your use case doesn't require computing gradients, i.e., you're just rendering and not backpropagating all the way to vertex positions etc., then it's better to use OpenGL directly instead of nvdiffrast.

jenkspt commented 1 year ago

Thanks for the reply. I'm interested in the use case where I would provide my own gradient code. From my very basic understanding, it's tricky to share memory between CUDA and OpenGL -- and I figured nvdiffrast could be a good solution for general OpenGL/PyTorch/Tensorflow interop.

s-laine commented 1 year ago

It's not that tricky to share data between OpenGL and CUDA — it's all done on the CUDA side, and there are only a handful of functions related to it. There are many reasons for nvdiffrast to not have general OpenGL shader support, one being that the CUDA-based rasterizer (that cannot support OpenGL shaders) is about equally fast and better supported across computing platforms.

However, you're welcome to extend the rasterization op in your own project to support the kind of shading your use case needs, possibly along with some helpers to facilitate OpenGL/CUDA interop for data sharing. This may be somewhat complicated, but still much simpler than designing a general API for shader support. If you have specific questions about the design of the rasterization op, I'll be happy to help.

jenkspt commented 1 year ago

Thanks for the feedback! I'll close this now and open a new issue if I have any more questions.