JunyuanDeng
commented
6 months ago 6G显存的的确确太少了。我们测试用的都是24G的显卡，这段代码如果没有大量优化，确实很难在6G卡上运行。最好能有16G的显存。
Open Brosy-qj opened 6 months ago
JunyuanDeng
commented
6 months ago 6G显存的的确确太少了。我们测试用的都是24G的显卡，这段代码如果没有大量优化，确实很难在6G卡上运行。最好能有16G的显存。
PanWenbo
commented
6 months ago 你好，我使用的是24G的GPU，运行“python demo/run.py configs/kitti/kitti_01.yaml” 依然报RuntimeError: CUDA out of memory的错误，请问还有什么解决方案吗？
post-processing steps: 100%|███████████████████| 37/37 [02:43<00:00, 4.43s/it] (601, 4, 4) tracking frame: 55%|██████████▍ | 602/1101 [1:56:27<10:48:41, 78.00s/it]Process Process-2: Traceback (most recent call last): File "/home/panwb/anaconda3/envs/LOAM/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/panwb/anaconda3/envs/LOAM/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/home/panwb/NerfLOAM/NeRF-LOAM/src/mapping.py", line 112, in spin self.do_mapping(share_data, tracked_frame) File "/home/panwb/NerfLOAM/NeRF-LOAM/src/mapping.py", line 179, in do_mapping bundle_adjust_frames( File "/home/panwb/NerfLOAM/NeRF-LOAM/src/variations/render_helpers.py", line 394, in bundle_adjust_frames final_outputs = render_rays( File "/home/panwb/NerfLOAM/NeRF-LOAM/src/variations/render_helpers.py", line 211, in render_rays intersections, hits = ray_intersect( File "/home/panwb/anaconda3/envs/LOAM/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(args, *kwargs) File "/home/panwb/NerfLOAM/NeRF-LOAM/src/variations/voxel_helpers.py", line 534, in ray_intersect pts_idx, min_depth, max_depth = svo_ray_intersect( File "/home/panwb/NerfLOAM/NeRF-LOAM/src/variations/voxel_helpers.py", line 108, in forward children = children.expand(S G, *children.size()).contiguous() RuntimeError: CUDA out of memory. Tried to allocate 5.51 GiB (GPU 0; 23.64 GiB total capacity; 2.38 GiB already allocated; 10.22 GiB free; 5.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
JunyuanDeng
commented
5 months ago 看上去完全是有空间的，报错地方显示你有10GB的剩余空间，但是只需要分配5.5GB，因此你需要检查一下有没有别的进程在跑，或者已经结束但没退出的进程。
你好，我使用的是6G的GPU，我修改了chunk_size为chunk_size//10 ~ chunk_size//10000之后依然报错，请问还有其他解决方案吗？
PatchWorkpp::PatchWorkpp() - INITIALIZATION COMPLETE Decoder( (pe): Same() (pts_linears): ModuleList( (0): Linear(in_features=16, out_features=256, bias=True) (1): Linear(in_features=256, out_features=256, bias=True) ) (sdf_out): Linear(in_features=256, out_features=1, bias=True) ) ******* initializing first_frame: 0 initializing the first frame ... mapping process started!!!!!!!!! frame id 1 trans tensor([0., 0., 0.], device='cuda:0') insert keyframe PatchWorkpp::PatchWorkpp() - INITIALIZATION COMPLETE ******* tracking process started! ******* tracking frame: 0%| | 0/99 [00:00<?, ?it/s]Process Process-2: Traceback (most recent call last): File "/home/brosy/anaconda3/envs/NERF_LOAM/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/brosy/anaconda3/envs/NERF_LOAM/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/mapping.py", line 112, in spin self.do_mapping(share_data, tracked_frame, selection_method='current') File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/mapping.py", line 184, in do_mapping bundle_adjust_frames( File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/variations/render_helpers.py", line 395, in bundle_adjust_frames final_outputs = render_rays( File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/variations/render_helpers.py", line 211, in render_rays intersections, hits = ray_intersect( File "/home/brosy/anaconda3/envs/NERF_LOAM/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/variations/voxel_helpers.py", line 534, in ray_intersect pts_idx, min_depth, max_depth = svo_ray_intersect( File "/home/brosy/projects/NERF_LOAM/NeRF-LOAM/src/variations/voxel_helpers.py", line 108, in forward children = children.expand(S * G, *children.size()).contiguous() RuntimeError: CUDA out of memory. Tried to allocate 516.00 MiB (GPU 0; 5.81 GiB total capacity; 189.92 MiB already allocated; 519.38 MiB free; 204.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF tracking frame: 1%|▎ | 1/99 [00:07<11:55, 7.30s/it]