search

sxyu / svox2

Plenoxels: Radiance Fields without Neural Networks
BSD 2-Clause "Simplified" License
2.79k stars 360 forks source link

How to use 2 GPUs to train? #64

Closed Miles629 closed 2 years ago

Miles629 commented 2 years ago

I just set the as "0,1", but it seems just gpu0 was used. ./launch.sh -c

When I set the as "0", I got

Traceback (most recent call last):
  File "opt.py", line 605, in <module>
    train_step()
  File "opt.py", line 595, in train_step
    grid.optim_sh_step(lr_sh, beta=args.rms_beta, optim=args.sh_optim)
  File "/home/mcy/.local/lib/python3.8/site-packages/svox2/svox2.py", line 2024, in optim_sh_step
    self.sh_rms = torch.zeros_like(self.sh_data.data) # FIXME init?
RuntimeError: CUDA out of memory. Tried to allocate 1.53 GiB (GPU 0; 15.78 GiB total capacity; 10.66 GiB already allocated; 1.13 GiB free; 13.49 GiB reserved in total by PyTorch)

And when I set "0,1" here (try to use 2 gpus) : ./launch.sh <exp_name> <GPU_id> <data_dir> -c <config> It seems that only one gpu was used.

Traceback (most recent call last):
  File "opt.py", line 605, in <module>
    train_step()
  File "opt.py", line 595, in train_step
    grid.optim_sh_step(lr_sh, beta=args.rms_beta, optim=args.sh_optim)
  File "/home/mcy/.local/lib/python3.8/site-packages/svox2/svox2.py", line 2024, in optim_sh_step
    self.sh_rms = torch.zeros_like(self.sh_data.data) # FIXME init?
RuntimeError: CUDA out of memory. Tried to allocate 1.55 GiB (GPU 0; 15.78 GiB total capacity; 10.69 GiB already allocated; 1.12 GiB free; 13.50 GiB reserved in total by PyTorch)

I want to know what is the correct way to use two gpus in this work. Thanks for your help~

sarafridov commented 2 years ago

Currently only one GPU is supported. If your GPU does not have sufficient memory, you can try reducing the resolution, increasing the sparsity-inducing regularizer, or switching to optimizer to SGD.

Miles629 commented 2 years ago

Currently only one GPU is supported. If your GPU does not have sufficient memory, you can try reducing the resolution, increasing the sparsity-inducing regularizer, or switching to optimizer to SGD.

Thanks for your reply, I will try as you suggest. I recommend considering multiple Gpus support because, many people don't have enough memory on one GPU

city19992 commented 2 years ago

Thank you for your advice. I have another question which is how to increase the sparsity-inducing regularizer. There are so many configs that I am confused about them.