CUDA_ERROR_OUT_OF_MEMORY

[riponazad]

Hi,

Is there any limitation on the training image resolution?

I used just 12 images of resolution 4640x3472, and got this error though I successfully generated mesh for custom images with lower resolution. I am using Nvidia RTX A6000 with 50GB, so memory shouldn't be an issue.

Are there some parameters in the config that I can tweak to occupy less memory with compromising mesh quality?

[jmunkberg]

We rely on nvdiffrast for differentiable rasterization, and rasterizing full screen buffers, so for very high image resolution, the memory cost is significant in the public code base. Easiest way to reduce memory (at the cost of quality) is to reduce the image resolution and/or lower the batch size.

In your case, I would recommend to downscale the images a factor 2x (2320x1736) or 4x (1160x868) to get around the memory issue. Also, 12 images is likely too few to get high quality results. Most of our test use 100+ views, preferably nicely distributed around the object.

[jmunkberg]

Also, see https://github.com/NVlabs/nvdiffrec/issues/15

[riponazad]

@jmunkberg Thanks for your quick response.

I reduced the number of training images to check if the memory issues go away as it was the case for instance nerf. However, I am trying to compare with the instant nerf in terms of fewer training images and resolution.

I will check #15 for further understanding. Thanks again.

[jmunkberg]

The number of training images is not an issue, you can have 100s of them. The problem is that in each training iteration & minibatch, we need memory for the forward and backward passes for full size images x batch size.

This can be worked around by, like (instant)NeRF training, sample random rays from random views instead of rasterizing full views, but that requires replacing the differentiable rasterizer with a differentiable ray caster, which we haven't done in this code release.

[riponazad]

@jmunkberg Yeah, I understood it now completely. Thanks for the enlightenment.