search

ethanweber / nerfiller

NeRFiller project https://ethanweber.me/nerfiller/
Apache License 2.0
96 stars 3 forks source link

OOM Problem #5

Closed aiyb1314 closed 8 months ago

aiyb1314 commented 8 months ago

Inpainting batch of images... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% 0:00:03 torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 9.38 GiB (GPU 0; 23.70 GiB total capacity; 8.04 GiB already allocated; 9.27 GiB free; 12.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.

Environment: GPU RTX 3090 24GB

ethanweber commented 8 months ago

Hm, could you provide more details leading to this error? Are you running in the ns-launch command or individually? Running the command by itself will help to debug where it's running out of memory and why. Maybe you have something else running on the same machine or GPU?

aiyb1314 commented 8 months ago

When I use python scripts/inpaint_nerfstudio_dataset.py grid-prior --nerfstudio-dataset data/nerfstudio/materials to generate inpainting image, torch.cuda. OutOfMemoryError: insufficient CUDA memory. tried to allocate 9.38 GiB (GPU 0; 23.70 GiB total capacity; 7.97 GiB already allocated; 7.68 GiB free; 8.23 GiB reserved in total by PyTorch) If the reserved memory is larger than the allocated memory, try setting max_split_size_mb to avoid fragmentation. See the memory management and PYTORCH_CUDA_ALLOC_CONF documentation.

ethanweber commented 8 months ago

I see! Try running python scripts/inpaint_nerfstudio_dataset.py grid-prior --help if you want to see things you can modify.

Maybe make this 20 and 10, respectively. It would be inpainting less images at a time in a batch, but it might let you get around this memory issue. https://github.com/ethanweber/nerfiller/blob/c59164a0206b44b7ae8fc6b7033070a4dd558498/nerfiller/configs/inpaint.py#L40-L43