Open d-sharafeldeen opened 2 years ago
Anyone have a solution for this problem
Our algorithm is unfortunately quite memory hungry, so to run on GPUs with only 8GB of memory, you need to reduce the rendering resolution (to 512 or even 256) and/or the batch size. In the config:
"train_res" : 256,
"batch": 1,
This will reduce the quality of the resulting models, but could be used for faster iterations and experiments on smaller GPUs
Could I deal with this problem with two GPUs? Load ref mesh to one GPU, and load base mesh to another?
And how could I do that?
Hello @X1aoyueyue
You can look at https://github.com/NVlabs/nvdiffrec#examples which is a similar code base. There, we used PyTorch DDP for multiGPU support (Linux only).
I think the nvdiffmodeling code base could be adapted similarly.
The way we distribute the work is to split the minibatches over multiple GPUs, so if you have more GPUs, you can run with a larger batch size.
RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 8.00 GiB total capacity; 7.11 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF