search

primecai / Pix2NeRF

271 stars 33 forks source link

when use 2 Tesla V100 GPU cards, it gives this below error which said No locks available, what do you think the issue is ? #25

Open dellshan opened 1 year ago

dellshan commented 1 year ago

(Pix2NeRF) bash-4.4$ CUDA_VISIBLE_DEVICES=0,1 python3 train_con.py --curriculum=celeba --output_dir='/data/yshan/data/Img/pix2NerfOutput' --dataset_dir='/data/yshan/data/Img/img_align_celeba' --encoder_type='CCS' --recon_lambda=5 --ssim_lambda=1 --vgg_lambda=1 --pos_lambda_gen=15 --lambda_e_latent=1 --lambda_e_pos=1 --cond_lambda=1 --load_encoder=1 Namespace(n_epochs=3000, sample_interval=1000, output_dir='/data/yshan/data/Img/pix2NerfOutput', load_dir='/data/yshan/data/Img/pix2NerfOutput', curriculum='celeba', eval_freq=5000, port='12354', set_step=None, model_save_interval=200, pretrained_dir='', wandb_name='', wandb_entity='', wandb_project='', recon_lambda=5.0, ssim_lambda=1.0, vgg_lambda=1.0, dataset_dir='/data/yshan/data/Img/img_align_celeba', pos_lambda_gen=15.0, sn=0, lambda_e_latent=1.0, lambda_e_pos=1.0, encoder_type='CCS', cond_lambda=1.0, ema=1, load_encoder=1) Lock not found Traceback (most recent call last): File "/data/yshan/Pix2NeRF/train_con.py", line 684, in mp.spawn(train, args=(num_gpus, opt), nprocs=num_gpus, join=True) File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes while not context.join(): File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/data/yshan/Pix2NeRF/train_con.py", line 85, in train setup(rank, world_size, opt.port, opt.output_dir) File "/data/yshan/Pix2NeRF/train_con.py", line 46, in setup dist.init_process_group('gloo', init_method=file_lock, rank=rank, world_size=world_size) File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 907, in init_process_group default_pg = _new_process_group_helper( File "/home/yshan/anaconda3/envs/Pix2NeRF/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1009, in _new_process_group_helper backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout) RuntimeError: flock: No locks available

dellshan commented 1 year ago

while the 2 GPU cards are all available: bash-4.4$ nvidia-smi Sat Sep 16 13:25:43 2023
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 | | N/A 33C P0 26W / 250W | 71MiB / 32768MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-PCIE... Off | 00000000:D8:00.0 Off | 0 | | N/A 33C P0 26W / 250W | 8MiB / 32768MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 2179 G /usr/libexec/Xorg 39MiB | | 0 N/A N/A 3079 G /usr/bin/gnome-shell 26MiB | | 1 N/A N/A 2179 G /usr/libexec/Xorg 4MiB | +-----------------------------------------------------------------------------+

dellshan commented 1 year ago

While when I use only one GPU card, then it suddenly works !!!! why it can not work on 2 cards ?

primecai commented 1 year ago

Hi,

Please feel free to delete the lines related to lock. Those were meant to work my own training infrastructure which apparently don't generalize. It should be fine directly deleting them.

Best regards, Shengqu