Closed jakobtroidl closed 9 months ago
Hi @jakobtroidl
Out-of-the box, I'd say that there are a few things missing in Dr.Jit to really squeeze out every bit of performance of something like this.
As you suggested, in your current approach there is some overhead of the data transfer. Fundamentally, this problem lies on the display tool/framework you want to use: find on which will accept CUDA arrays through the dlpack
interface. (There are some ongoing discussions about the necessity to synchronize when using that interface https://github.com/mitsuba-renderer/drjit/issues/198). Alternatively, the Texture2f
/Texture3f
classes use CUDA textures, I'd expect some framework to maybe accept these directly. However their respective handles aren't exposed through Python so you'd need to add that yourself.
Another big overhead is the tracing runtime of executing the Python interpreter through your render()
function. Although Dr.Jit will cache its kernels and re-use them, it still has to "read" through your code entirely to realize that you're executing some piece of code that it has already seen. There isn't much you can do to alleviate this.
I think it should still be possible to access the underlying Dr.Jit evaluated CUDA buffers directly using the data_()
call. E.g.,
a = dr.linspace(dr.cuda.Float32, 0, 1, 1024)
print(a.data_())
If I recall correctly, this will be a simple pointer directly to the CUDA memory. So if you then have a CUDA kernel that draws that to screen/a texture, that might be quite fast. I agree with Nicolas that the tracing is likely of significant overhead.
Thank you so much for your answers.
Another big overhead is the tracing runtime of executing the Python interpreter through your render() function.
I am wondering how I could work around this issue. Would it make sense to compile CUDA kernels once from a Python implementation and then invoke them from a C++ based rendering loop? There must be a way around this issue since the paper mentioned above is so incredibly fast and it seems like they implemented their forward pass in drjit.
This paper uses a custom Dr.Jit version with many project-specific modifications. It's our goal to make something like this possible in mainline Dr.Jit in the future. Right now it is not possible due to the tracing overheads mentioned above. It will take a long time, you may want to pursue other other options if your goal is to do this right now.
ok, thanks for the heads-up.
I am trying to use drjit to build a forward-only renderer that should run in real-time after being inspired by this amazing new paper.
I wonder how to best transfer frames rendered by a kernel to the screen as fast as possible. I am first converting the data into a numpy array and then render it using the PyGame window library - this runs at ~20 FPS on my LLVM-accelerated Intel MacPro (see code below). This seems inefficient because there's a lot of unnecessary data transfer (GPU->CPU->GPU for CUDA backend) happening for each frame. Do you have recommendations on more clever design choices when using drjit for real-time rendering?