How to solve the error reported during the training trial

[nss6planner]

on3d$ bash tools/dist_train.sh configs/fcaf3d/fcaf3d_8x2_sunrgbd-3d-10class.py 2***** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmdet/utils/setup_env.py:49: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. f'Setting MKL_NUM_THREADS environment variable for each process ' /home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmdet/utils/setup_env.py:49: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. f'Setting MKL_NUM_THREADS environment variable for each process '

[filaPro]

Hi @nss6planner ,

I don't see any errors here, just warnings. Can you provide the full log?

[nss6planner]

Sorry, the complete error report is as follows：

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmdet/utils/setup_env.py:49: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. f'Setting MKL_NUM_THREADS environment variable for each process ' /home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmdet/utils/setup_env.py:49: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. f'Setting MKL_NUM_THREADS environment variable for each process ' Traceback (most recent call last): Traceback (most recent call last): File "tools/train.py", line 263, in File "tools/train.py", line 263, in main() File "tools/train.py", line 171, in main main() File "tools/train.py", line 171, in main init_dist(args.launcher, cfg.dist_params) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 40, in init_dist init_dist(args.launcher, cfg.dist_params) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 40, in init_dist _init_dist_pytorch(backend, kwargs) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 63, in _init_dist_pytorch _init_dist_pytorch(backend, kwargs) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 63, in _init_dist_pytorch dist.init_process_group(backend=backend, kwargs) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group dist.init_process_group(backend=backend, kwargs) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group barrier() File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier barrier() File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier work = _default_pg.barrier() RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8 work = _default_pg.barrier() RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8 Traceback (most recent call last): File "/home/air/anaconda3/envs/mmde/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/air/anaconda3/envs/mmde/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in main() File "/home/air/anaconda3/envs/mmde/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/air/anaconda3/envs/mmde/bin/python', '-u', 'tools/train.py', '--local_rank=1', 'configs/fcaf3d/fcaf3d_8x2_sunrgbd-3d-10class.py', '--seed', '0', '--launcher', 'pytorch']' returned non-zero exit status 1.

[filaPro]

Looks like the problem not with fcaf3d, but with your pytorch installation... Can you check that pytorch and for example mmcv works fine on your machine?