search

libuyu / GHM_Detection

The implementation of “Gradient Harmonized Single-stage Detector” published on AAAI 2019.
MIT License
618 stars 110 forks source link

ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE #16

Open Vipermdl opened 5 years ago

Vipermdl commented 5 years ago

Hi, I`m facing the problem with training: (py36) [madongliang@compute-0-5 experiments]$ sh train.sh Traceback (most recent call last): Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): from mmdet.apis import (train_detector, init_dist, get_root_logger, from mmdet.apis import (train_detector, init_dist, get_root_logger, File "../mmdetection/tools/train.py", line 8, in File "../mmdetection/tools/train.py", line 8, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in from mmdet.apis import (train_detector, init_dist, get_root_logger, from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import # noqa: F401, F403 from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper from mmdet.ops.nms import nms_wrapper from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align from .roi_align import RoIAlign, roi_align from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .roi_align import RoIAlign, roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .functions.roi_align import roi_align ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .. import roi_align_cuda from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE

I`m using CUDA 9.0, pytorch 1.0.1.post2, python 3.6 Everything compiled well during installation.

Thanks!

libuyu commented 5 years ago

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment. For more suggestions, you can refer to https://github.com/open-mmlab/mmdetection/issues/385 .

ruiyuanlu commented 5 years ago

Try 

import torch

at the very first in your code. If it doesn't work, maybe this comment can help you.

Vipermdl commented 5 years ago

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment. For more suggestions, you can refer to open-mmlab/mmdetection#385 .

thanks, this is working.