Closed Kana-alt closed 3 years ago
There seems to be a mismatch between number of gpus you have and requested here.
If there is less than two gpus, you could modify the script to use one gpu and larger batch size, which may take longer to train.
thank you.
I used the docker to build the environment. I prepared the DAVIS data and tried to run Optimize on camel observations.
Then, I got a CUDA error and the execution did not proceed.
Can you tell me the cause?
docker run -v $(pwd):/lasr --gpus all lasr bash -c 'cd lasr; source activate lasr; bash scripts/template.sh camel' Jitting Chamfer 3D Jitting Chamfer 3D Loaded JIT 3D CUDA chamfer distance Loaded JIT 3D CUDA chamfer distance Traceback (most recent call last): File "optimize.py", line 59, in
app.run(main)
File "/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "optimize.py", line 40, in main
torch.cuda.set_device(opts.local_rank)
File "/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/cuda/init.py", line 263, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal