CUDA_malloc out of memory!

[jakoaltrd]

I successfully built mitsuba2 with the gpu_autodiffspectral variants: "enabled": [

The "scalar_rgb" variant must be included at the moment.

    "scalar_rgb",
    "scalar_spectral",
    "gpu_autodiff_spectral",
    "gpu_autodiff_rgb"
],_

and run it with the command: _mitsuba -m gpu_autodiff_spectral cbox.xml -o autodiff.jpeg_then get the error: Caught a critical exception: cuda_malloc(): out of memory! cuda_shutdown() then I change the film width to a small integer 300 integer name="width" value="300" integer name="height" value="300" it works well, so how to solve this issue , specifications: win10, cmake, python3.8 cuda11.4 rtx3070 vs2019 optix7.1

there are 8 gigabytes VRAM available, so gpu memory should not be the bottleneck.

[merlinND]

Hi @jakoaltrd,

Please have a look at previous issues related to this problem: https://github.com/mitsuba-renderer/mitsuba2/issues?q=is%3Aissue+out+of+memory

The solution is to render with fewer samples per pixel, lower resolution, or to use smaller value of samples_per_pass (it can be added to the integrator's properties in the XML scene file).

The upcoming version of Mitsuba will allow rendering with a much smaller GPU memory footprint.

[MarisaKirisame]

Hi, I work on memory saving for autodiff, in particular gradient checkpointing. Are you interested in hearing more about how it work? I could give a talk on my work, dynamic tensor rematerialization, if you are interested about it.

[merlinND]

Hi @MarisaKirisame,

In the context of differentiable rendering, we were able to modify the algorithm so that it inherently uses less memory (Radiative Backpropagation) and relies on AD only very locally, which means that we don't have to create large graphs.

That being said it's always interesting to read about other techniques, would you have a paper or writeup available on that topic?

[MarisaKirisame]

Yes. marisa.moe/dtr.html is the project's blog post, and https://arxiv.org/abs/2006.09616 is the paper. I skimmed the paper. It seems like there are connection to the age old AD technique - adjoint of calling a solver (e.g. ODE solver, linear solver, or even just a fixpoint function) amount to calling that solver on adjointed equation. In such a case there's indeed little need for checkpointing.