search

NVlabs / nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks
Other
455 stars 30 forks source link

CUDA out of memory. #16

Open d-sharafeldeen opened 2 years ago

d-sharafeldeen commented 2 years ago

RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 8.00 GiB total capacity; 7.11 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

d-sharafeldeen commented 2 years ago

Anyone have a solution for this problem

jmunkberg commented 2 years ago

Our algorithm is unfortunately quite memory hungry, so to run on GPUs with only 8GB of memory, you need to reduce the rendering resolution (to 512 or even 256) and/or the batch size. In the config: 

    "train_res" : 256, 
    "batch": 1,

This will reduce the quality of the resulting models, but could be used for faster iterations and experiments on smaller GPUs

X1aoyueyue commented 1 year ago

Could I deal with this problem with two GPUs? Load ref mesh to one GPU, and load base mesh to another?

X1aoyueyue commented 1 year ago

And how could I do that?

jmunkberg commented 1 year ago

Hello @X1aoyueyue

You can look at https://github.com/NVlabs/nvdiffrec#examples which is a similar code base. There, we used PyTorch DDP for multiGPU support (Linux only).

I think the nvdiffmodeling code base could be adapted similarly.

The way we distribute the work is to split the minibatches over multiple GPUs, so if you have more GPUs, you can run with a larger batch size.