search

sunset1995 / DirectVoxGO

Direct voxel grid optimization for fast radiance field reconstruction.
https://sunset1995.github.io/dvgo
Other
1.05k stars 110 forks source link

torch.cuda.OutOfMemoryError: CUDA out of memory #57

Open panxkun opened 1 year ago

panxkun commented 1 year ago

Hi, this work is amazing. But I have encountered a strange problem. On desktop computer with 2080 8g gpu, it works well. However, when I run it on the server with 3090 24G gpu, it crashed with oom: I think it maybe the problem with the version of cuda and pytorch. so I kept the environment on server same with it on desktop. It still cannot run normally. Do you have any suggestion?

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.77 GiB (GPU 0; 23.70 GiB total capacity; 19.12 GiB already allocated; 3.93 GiB free; 19.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

kts707 commented 1 year ago

Hi, I also encountered the same issue with RTX 3090 GPU when I tried to run the training for LLFF datasets. Have you resolved this issue? Thanks!