search

PJLab-ADG / neuralsim

neuralsim: 3D surface reconstruction and simulation based on 3D neural rendering.
MIT License
582 stars 31 forks source link

CUDA out of memory #15

Closed npcdna closed 11 months ago

npcdna commented 11 months ago

after running train.py, i ran render.py after that, but i meet cuda memory error like this:

RuntimeError: CUDA out of memory. Tried to allocate 3.78 GiB (GPU 0; 15.74 GiB total capacity; 8.81 GiB already allocated; 228.75 MiB free; 13.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.

But the magic is that sometimes I turn off the terminal that was running python before, and I can run render.py successfully, and sometimes it doesn’t work

ventusff commented 11 months ago

Hi, Have you tried a smaller --rayschunk value ? The original default 65536 might be too large.

If you're running experiments on your daily-use machine, apps like browsers can consume GPU memory. It's possible that GPU memory usage is right at the edge of being fully occupied, which could lead to the phenomenon you've observed.

npcdna commented 11 months ago

thanks, i change this config to finish render task, but i got a terrible render result !