graphdeco-inria / gaussian-splatting

Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Other
14.85k stars 1.94k forks source link

cuda out of memory #1009

Open datouzhan opened 1 month ago

datouzhan commented 1 month ago

(gaussian_splatting) root@autodl-container-32ed4aa578-4afa071a:~/gaussian-splatting# python train.py -s /root/autodl-tmp/pazhouta11 -r 1600 Optimizing Output folder: ./output/9d30c127-6 [12/10 16:26:13] Tensorboard not available: not logging progress [12/10 16:26:13] Reading camera 2927/2927 [12/10 16:26:29] Loading Training Cameras [12/10 16:26:30] Traceback (most recent call last): File "train.py", line 219, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 35, in training scene = Scene(dataset, gaussians) File "/root/gaussian-splatting/scene/init.py", line 73, in init self.train_cameras[resolution_scale] = cameraList_from_camInfos(scene_info.train_cameras, resolution_scale, args) File "/root/gaussian-splatting/utils/camera_utils.py", line 58, in cameraList_from_camInfos camera_list.append(loadCam(args, id, c, resolution_scale)) File "/root/gaussian-splatting/utils/camera_utils.py", line 52, in loadCam image_name=cam_info.image_name, uid=id, data_device=args.data_device) File "/root/gaussian-splatting/scene/cameras.py", line 46, in init self.original_image *= torch.ones((1, self.image_height, self.image_width), device=self.data_device) RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 47.50 GiB total capacity; 43.46 GiB already allocated; 18.56 MiB free; 46.35 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 企业微信截图_17287227908894 But when I look at the graphics card usage information, there is no process occupying it.I have set export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128.Is there any solution? Attached is my training configuration 企业微信截图_1728723045131 企业微信截图_1728723139733

hjh530 commented 1 month ago

You can try take images to cpu.

datouzhan commented 1 month ago

You can try take images to cpu. Thanks for your answer. How can I transfer the image to the cpu?

datouzhan commented 1 month ago

I tried to transfer the image to the cpu. But it still failed. 企业微信截图_17289978893391

hjh530 commented 1 month ago

Maybe your images is too larger and the number of them is too big.The first measure is that cut down the number of pictures.The second is to reduce the image resolution

datouzhan commented 1 month ago

可能是你的图片太大,图片数量太多,第一点是减少图片数量,第二点是降低图片分辨率

Thank you for your answer, it works fine.