RuntimeError: No CUDA GPUs are available

[wowo0709]

Hi. Thank you for your great work!

I encountered a Runtime error while executing reconstruction with my custom yaml file.

Below is my custom yaml file. I changed N_voxel_init, time_grid_init/final, data_dir, and downsample. I guess the error would raise also with given dnerf_slim.yaml config file because I changed few configs.

systems:
  basedir: "./logs"

model:
  N_voxel_init: 32768 # Paper: 32^3

  # We heuristically set initial time_grid and final time grid. 
  # A common strategy to set these two parameters for DNeRF dataset is:
  # We empirically set time_grid_final = int(0.24 * N_frames), and time_grid_init = int(0.5 * time_grid_final)
  # We show N_frames for each video 
  # "standup": 150; "jumpingjacks": 200; "hook"   : 100; "bouncingballs": 150
  # "lego"   :  50; "hellwarrior" : 100; "mutant" : 150; "trex"         : 200
  time_grid_init: 6  
  time_grid_final: 12
  upsampling_type: unaligned
  upsample_list: [3000, 6000, 9000]
  update_emptymask_list: [4000, 8000, 10000] # Paper: 4k, 10k

data:
  datadir: "./data/lego"
  scene_bbox_min: [-1.5, -1.5, -1.5]
  scene_bbox_max: [1.5, 1.5, 1.5]
  downsample: 2.5 # assert (img_res / ds) % 16 (macro_block_size) = 0

optim:
  n_iters: 25000

render_test: True
render_path: True
expname: dnerf_slim_lego

And below is the Runtime error I encountered. The error raised after 10k iterations.

██████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:41<00:00,  8.22s/it]
PSNR: 25.67093115332822

Iteration 09990: train_psnr = 33.24 test_psnr = 0.00 mse = 0.000485:  40%|███▉      | 9999/25000 [2:27:09<3:40:46,  1.13it/s]
Traceback (most recent call last):
  File "main.py", line 199, in <module>
    reconstruction(cfg)
  File "main.py", line 121, in reconstruction
    trainer.train()
  File "/mnt/c/Repositories/2023-1/TextTo4D/hexplane/hexplane/render/trainer.py", line 444, in train
    torch.cuda.synchronize()
  File "/home/ywjeon990709/anaconda3/envs/hexplane/lib/python3.8/site-packages/torch/cuda/__init__.py", line 494, in synchronize
    _lazy_init()
  File "/home/ywjeon990709/anaconda3/envs/hexplane/lib/python3.8/site-packages/torch/cuda/__init__.py", line 217, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

OS: WSL2 Ubuntu 20.04 GPU: RTX 2050 4GB Please tell me if you need any other information. Thanks.

[Caoang327]

Hi:

I am trying to replicate this issue. I just want to double-check several facts, if possible. How many GPUs are available in your environment? Do you see the same issue with the other data, like standup with the same setting? Thanks

[wowo0709]

Thank you for your reply.

I'm using only a single GPU. And the error also raises after 10k iterations when I run with standup data under same setting.

[Caoang327]

Hi:

Thanks for the info. I didn't see the problem using the provided config. Could you please try GPUs with bigger memory, like 2080TI?

[wowo0709]

It seems the error raised because of the lack of my GPU memory.

I'll try with the bigger one and reopen the issue if the error still occurs.

Thanks

[HajunKim]

Have you tried with a larger one?

[Caoang327]

Have you tried with a larger one?

Do you meet the same problem as well?

[HajunKim]

nah I am good

[HajunKim]

Have you tried with a larger one?

Do you meet the same problem as well?

nah I am good