search

alexanderbergman7 / metanlrpp

Official Implementation for Fast Training of Neural Lumigraph Representations using Meta Learning.
41 stars 5 forks source link

CUDA out of memory error #2

Open svsambandam opened 2 years ago

svsambandam commented 2 years ago

Hello :)

When running python experiment_scripts/train_sdf_ibr.py --config_filepath configs/nlrpp_dtu.txt I run out of memory quickly. If I understand correctly, I am running on a GeForce RTX 2080 SUPER GPU with about 6000MiB available. Is there a way to reduce the batch_size? I think the dataloader in train_sdf_ibr.py already has a batch_size of 1. Thank you in advance!!

alexanderbergman7 commented 2 years ago

Yes, so 6G of VRAM is going to be quite low for MetaNLR++ (we trained on GPUs with either 24G or 48G). The simplest place to start is to decrease the size of models, especially the image encoder / decoder (the MLP is not that large in comparison). Operating on lower resolution images is also a possibility to significantly reduce the computational overhead; you can change this parameter (load_im_scale) in the config to automatically load in the DTU at a lower resolution.

alexanderbergman7 commented 2 years ago

I'll leave it open in case anyone else has this same question and would like to see the discussion.

ghost commented 2 years ago

I have a question I have eight gpus and each 32G memory, also have the error "RuntimeError:CUDA out of memory, thied to allocate 352.00MB(GPU 0; 31.75GiB total capacity:29.93GiB already allocated;83.69MiB free; 30.19GiB reserved in total by Pytorch)", It look like doesn't using other Gpus @alexanderbergman7