RuntimeError: SigmoidFocalLoss is not compiled with GPU support

[Morizb]

Hello, when I download the fusion_voxel0075_R50.pth you provided, and run sh . /tools/dist_train.sh . /configs/MSMDFusion_nusc_voxel_LC.py 2 for the 2-nd stage training, the error is reported as follows, tried some solutions on the Internet still did not get a solution, I hope you can point out, thank you!

2023-09-14 10:43:15,801 - mmdet - INFO - Start running, host: xzluo@b5163d5d11c9, work_dir: /public/home/xzluo/zc/MSMDFusion-main/work_dirs/MSMDFusion_nusc_voxel_LC 2023-09-14 10:43:15,801 - mmdet - INFO - workflow: [('train', 1)], max: 6 epochs Traceback (most recent call last): File "./tools/train.py", line 283, in main() File "./tools/train.py", line 272, in main train_detector( File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector runner.run(data_loaders, cfg.workflow) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run epoch_runner(data_loaders[i], kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 46, in train_step output = self.module.train_step(inputs[0], kwargs[0]) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 247, in train_step losses = self(data) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func return old_func(args, kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/base.py", line 58, in forward return self.forward_train(kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/MSMDFusion.py", line 534, in forward_train losses_pts = self.forward_pts_train(pts_feats, img_feats, gt_bboxes_3d, File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/MSMDFusion.py", line 574, in forward_pts_train losses = self.pts_bbox_head.loss(loss_inputs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func return old_func(*args, kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/dense_heads/transfusion_head.py", line 1260, in loss layer_loss_cls = self.loss_cls(layer_cls_score, layer_labels, layer_label_weights, avg_factor=max(num_pos, 1)) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, *kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/losses/focal_loss.py", line 170, in forward loss_cls = self.loss_weight calculate_loss_func( File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/losses/focal_loss.py", line 85, in sigmoid_focal_loss loss = _sigmoid_focal_loss(pred.contiguous(), target, gamma, alpha, None, File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/ops/focal_loss.py", line 54, in forward ext_module.sigmoid_focal_loss_forward( RuntimeError: SigmoidFocalLoss is not compiled with GPU support Traceback (most recent call last): File "./tools/train.py", line 283, in main() File "./tools/train.py", line 272, in main train_detector( File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector runner.run(data_loaders, cfg.workflow) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run epoch_runner(data_loaders[i], kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 46, in train_step output = self.module.train_step(inputs[0], kwargs[0]) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 247, in train_step losses = self(data) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func return old_func(args, kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/base.py", line 58, in forward return self.forward_train(kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/MSMDFusion.py", line 534, in forward_train losses_pts = self.forward_pts_train(pts_feats, img_feats, gt_bboxes_3d, File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/detectors/MSMDFusion.py", line 574, in forward_pts_train losses = self.pts_bbox_head.loss(loss_inputs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func return old_func(*args, *kwargs) File "/public/home/xzluo/zc/MSMDFusion-main/mmdet3d/models/dense_heads/transfusion_head.py", line 1260, in loss layer_loss_cls = self.loss_cls(layer_cls_score, layer_labels, layer_label_weights, avg_factor=max(num_pos, 1)) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, kwargs) File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/losses/focal_loss.py", line 170, in forward loss_cls = self.loss_weight * calculate_loss_func( File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmdet/models/losses/focal_loss.py", line 85, in sigmoid_focal_loss loss = _sigmoid_focal_loss(pred.contiguous(), target, gamma, alpha, None, File "/public/home/xzluo/anaconda3/envs/zc/lib/python3.8/site-packages/mmcv/ops/focal_loss.py", line 54, in forward ext_module.sigmoid_focal_loss_forward( RuntimeError: SigmoidFocalLoss is not compiled with GPU support ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 29983) of binary: /public/home/xzluo/anaconda3/envs/zc/bin/python ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group

[SxJyJay]

How do you set up the mmcv library? If you compile it locally, please check up whether your cuda/nvcc is enabled during compiling.

[Morizb]

Thanks for your reply, I found the problem, when I run python mmdet3d/utils/collect_env.py, it shows TorchVision: 0.10.0+cu111 OpenCV: 4.8.0 MMCV: 1.2.7 MMCV Compiler: GCC 8.4 MMCV CUDA Compiler: not available MMDetection: 2.10.0 MMDetection3D: 0.11.0+

[Morizb]

Hi, I modified the previous bug,

but when I continue to run sh . /tools/dist_train.sh . /configs/MSMDFusion_nusc_voxel_LC.py 2, it reports the following error:

The environment for installation is as follows: (msmd) xzluo@d037a065fa35:~/zc/MSMDFusion-main$ conda list

packages in environment at /public/home/xzluo/anaconda3/envs/msmd:

#

Name Version Build Channel

_libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main _sysroot_linux-64_curr_repodata_hack 3 haa98f57_10 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main absl-py 1.4.0 addict 2.4.0 aiofiles 22.1.0 aiosqlite 0.19.0 anyio 3.7.1 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 astor 0.8.1 attrs 23.1.0 Babel 2.12.1 backcall 0.2.0 beautifulsoup4 4.12.2 binutils_impl_linux-64 2.35.1 h27ae35d_9 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main binutils_linux-64 2.35.1 h454624a_30 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main black 23.3.0 blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bleach 6.0.0 ca-certificates 2019.11.28 hecc5488_0 moussi cached-property 1.5.2 cachetools 4.2.4 ccimport 0.4.2 certifi 2019.11.28 py37_0 moussi cffi 1.15.1 charset-normalizer 3.2.0 click 8.1.7 comm 0.1.4 cumm-cu117 0.4.11 cycler 0.11.0 Cython 3.0.2 dataclasses 0.6 debugpy 1.7.0 decorator 5.1.1 defusedxml 0.7.1 deprecation 2.1.0 descartes 1.1.0 entrypoints 0.4 exceptiongroup 1.1.3 fastjsonschema 2.18.0 fire 0.5.0 flake8 5.0.4 fonttools 4.38.0 fqdn 1.5.1 future 0.18.3 gast 0.2.2 gcc_impl_linux-64 8.4.0 he7ac559_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gcc_linux-64 8.4.0 he201b7d_30 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main google-auth 1.35.0 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.58.0 gxx_impl_linux-64 8.4.0 h9ce2e92_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gxx_linux-64 8.4.0 h85ed34b_30 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main h5py 3.8.0 idna 3.4 imageio 2.27.0 importlib-metadata 4.2.0 importlib-resources 5.12.0 iniconfig 2.0.0 intel-openmp 2022.0.1 h06a4308_3633 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ipdb 0.13.13 ipykernel 6.16.2 ipython 7.34.0 ipython-genutils 0.2.0 ipywidgets 8.1.1 isoduration 20.11.0 jedi 0.19.0 Jinja2 3.1.2 joblib 1.3.2 json5 0.9.14 jsonpointer 2.4 jsonschema 4.17.3 jupyter 1.0.0 jupyter-console 6.6.3 jupyter-events 0.6.3 jupyter-server 1.24.0 jupyter-ydoc 0.2.5 jupyter_client 7.4.9 jupyter_core 4.12.0 jupyter_packaging 0.12.3 jupyter_server_fileid 0.9.0 jupyter_server_ydoc 0.8.0 jupyterlab 3.6.5 jupyterlab-pygments 0.2.2 jupyterlab-widgets 3.0.9 jupyterlab_server 2.24.0 Keras-Applications 1.0.8 Keras-Preprocessing 1.1.2 kernel-headers_linux-64 3.10.0 h57e8cba_10 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main kiwisolver 1.4.5 lark 1.1.7 ld_impl_linux-64 2.35.1 h7274673_9 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libffi 3.2.1 he1b5a44_1007 moussi libgcc-devel_linux-64 8.4.0 hd257e2f_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgcc-ng 9.1.0 hdf63c60_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran-ng 7.3.0 hdf63c60_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgomp 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-devel_linux-64 8.4.0 hf0c5c8d_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-ng 9.1.0 hdf63c60_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main llvmlite 0.31.0 lyft-dataset-sdk 0.0.8 Markdown 3.3.4 MarkupSafe 2.1.3 matplotlib 3.5.2 matplotlib-inline 0.1.6 mccabe 0.7.0 mistune 3.0.1 mkl 2019.4 243 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl-service 2.3.0 py37he8ac12f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft 1.0.14 py37hd81dba3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r mkl_random 1.0.4 py37hd81dba3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r mmcv-full 1.2.7 mmdet 2.10.0 mmdet3d 0.11.0 mmpycocotools 12.0.3 mypy-extensions 1.0.0 nbclassic 1.0.0 nbclient 0.7.4 nbconvert 7.6.0 nbformat 5.8.0 ncurses 6.3 h7f8727e_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.7 networkx 2.2 ninja 1.11.1 notebook 6.5.5 notebook_shim 0.2.3 numba 0.48.0 numpy 1.17.0 py37h7e9f1db_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r numpy 1.19.5 numpy-base 1.17.0 py37hde5b4d6_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r nuscenes-devkit 1.1.10 oauthlib 3.2.2 open3d 0.13.0 opencv-python 4.5.5.64 openssl 1.1.1e h516909a_0 moussi opt-einsum 3.3.0 packaging 23.1 pandas 1.3.5 pandocfilters 1.5.0 parso 0.8.3 pathspec 0.11.2 pccm 0.4.8 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.5.0 pip 22.3.1 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pkgutil_resolve_name 1.3.10 platformdirs 3.10.0 plotly 5.16.1 pluggy 1.2.0 plyfile 0.8.1 portalocker 2.7.0 prometheus-client 0.17.1 prompt-toolkit 3.0.39 protobuf 4.24.3 psutil 5.9.5 ptyprocess 0.7.0 pyasn1 0.5.0 pyasn1-modules 0.3.0 pybind11 2.11.1 pycodestyle 2.9.1 pycparser 2.21 pyflakes 2.5.0 Pygments 2.16.1 pyparsing 3.1.1 pyquaternion 0.9.9 pyrsistent 0.19.3 pytest 7.4.2 python 3.7.7 hcf32534_0_cpython https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2023.3.post1 PyWavelets 1.3.0 PyYAML 6.0.1 pyzmq 24.0.1 qtconsole 5.4.4 QtPy 2.4.0 readline 8.1.2 h7f8727e_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main requests 2.31.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rsa 4.9 scikit-image 0.19.3 scikit-learn 1.0.2 scipy 1.4.1 Send2Trash 1.8.2 setuptools 65.6.3 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main Shapely 1.8.5 six 1.16.0 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sniffio 1.3.0 soupsieve 2.4.1 spconv-cu117 2.3.6 sqlite 3.38.5 hc218d9a_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sysroot_linux-64 2.17 h57e8cba_10 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tenacity 8.2.3 tensorboard 2.1.1 tensorflow-estimator 2.1.0 tensorflow-gpu 2.1.0 termcolor 2.3.0 terminado 0.17.1 terminaltables 3.1.10 threadpoolctl 3.1.0 tifffile 2021.11.2 tinycss2 1.2.1 tk 8.6.12 h1ccaba5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tomli 2.0.1 tomlkit 0.12.1 torch 1.7.0+cu110 torch-scatter 2.0.7 torchaudio 0.7.0 torchvision 0.8.1+cu110 tornado 6.2 tqdm 4.66.1 traitlets 5.9.0 trimesh 2.35.39 typed-ast 1.5.5 typing_extensions 4.7.1 uri-template 1.3.0 urllib3 2.0.4 waymo-open-dataset-tf-2-1-0 1.2.0 wcwidth 0.2.6 webcolors 1.13 webencodings 0.5.1 websocket-client 1.6.1 Werkzeug 2.2.3 wheel 0.38.4 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main widgetsnbextension 4.0.9 wrapt 1.15.0 xz 5.2.5 h7f8727e_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main y-py 0.6.0 yapf 0.40.1 ypy-websocket 0.8.4 zipp 3.15.0 zlib 1.2.12 h7f8727e_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

Do you know what the problem is, please?

[SxJyJay]

Error "numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject" indicates that your numpy version is not compatible with another library, to solve this problem, you can refer to this site. However, since numpy is a foundation library of other libraries like torch, scipy, etc, modifying the numpy version will arouse more version conflicts. Therefore, I suggest you find the library incompatible with the current numpy version, or setup a new environment by referring to my environment details.

[Morizb]

What is your graphics card model and memory? I can only apply two cards, the model is GeForce RTX 2080 Ti, the video memory is 11G, when I set samples_per_gpu=2, workers_per_gpu=2, it will report error when I run the code:

Do you know how to solve this issue?

[SxJyJay]

We use RTX3090 with 24G memory. You can try some techniques (like fp16, pytorch checkpoint, etc.) for saving the GPU memory.