search

ingra14m / Deformable-3D-Gaussians

[CVPR 2024] Official implementation of "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction"
https://ingra14m.github.io/Deformable-Gaussians/
MIT License
931 stars 53 forks source link

RuntimeError: CUDA error: out of memory #52

Open RuiqingTang opened 7 months ago

RuiqingTang commented 7 months ago

My GPU is 4060 with 8GB of VRAM. Is it too small? However, I saw someone using a 3090 and still encountering errors. https://github.com/ingra14m/Deformable-3D-Gaussians/issues/38 Here is the info: Training progress: 12%|█▎ | 5000/40000 [06:40<48:38, 11.99it/s, Loss=0.0467618]Traceback (most recent call last): File "E:\Projects\Python_projects\3D_Vsion\Deformable-3D-Gaussians\train.py", line 274, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations) File "E:\Projects\Python_projects\3D_Vsion\Deformable-3D-Gaussians\train.py", line 132, in training dataset.load2gpu_on_the_fly, dataset.is_6dof) File "E:\Projects\Python_projects\3D_Vsion\Deformable-3D-Gaussians\train.py", line 221, in training_report images = torch.cat((images, image.unsqueeze(0)), dim=0) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Training progress: 12%|█▎ | 5000/40000 [06:40<46:44, 12.48it/s, Loss=0.0467618]

RuiqingTang commented 7 months ago

The dataset is NeRF-DS.

ingra14m commented 7 months ago

Hi, thanks for your interest in this work.

In my experiments, NeRF-DS will not encounter oom on 3090. Can you provide the command leading to this problem?

RuiqingTang commented 7 months ago

The command I executed is as follows: python train.py -s G:/dataset/3Dvision/real-world/NeRF-DS/cup -m output/exp-ds1 --eval

yangbaoquan commented 6 months ago

Hi, thanks for your interest in this work.

In my experiments, NeRF-DS will not encounter oom on 3090. Can you provide the command leading to this problem?

I have encountered the same issue on RTX4090.

RuiqingTang commented 6 months ago

@yangbaoquan, I uniformly sampled 52 images from the original dataset and fixed this error, but the reconstruction was poor.

yangbaoquan commented 6 months ago

@yangbaoquan, I uniformly sampled 52 images from the original dataset and fixed this error, but the reconstruction was poor.

@RuiqingTang As a temporary solution, I comment the following lines and run training code successfully.

# Log and save
# cur_psnr = training_report(tb_writer, iteration, Ll1, loss, l1_loss, iter_start.elapsed_time(iter_end),
#                            testing_iterations, scene, render, (pipe, background), deform,
#                            dataset.load2gpu_on_the_fly, dataset.is_6dof)
# if iteration in testing_iterations:
#     if cur_psnr.item() > best_psnr:
#         best_psnr = cur_psnr.item()
#         best_iteration = iteration