search

fundamentalvision / Deformable-DETR

Deformable DETR: Deformable Transformers for End-to-End Object Detection.
Apache License 2.0
3.14k stars 513 forks source link

undefined symbol: _Z27ms_deform_attn_cuda_forwardRKN2at6TensorES2_S2_S2_S2_i #225

Closed bshakya77 closed 6 months ago

bshakya77 commented 6 months ago

Hello,

I am trying to train the FasterViT_DINO model for object detection. I installed all the required packages mentioned in the repository. But, when I run the test.py. I get the following error.

ImportError:../.local/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/MultiScaleDeformableAttention.cpython-310-x86_64-linux-gnu.so: undefined symbol: _Z27ms_deform_attn_cuda_forwardRKN2at6TensorES2_S2_S2_S2_i

System configuration:

CUDA Version (torch): 12.1 CUDA Version (system): 12.2 Torch Version: 2.2.0+cu121 Python 3.10.13 IDE: Miniconda 3 OS: Ubuntu

I installed MultiScaleDeformableAttention 1.0 using sh make.h command. I think it ran sucessfully and I can see the folder MultiScaleDeformableAttention created. The log is attached below.

/usr/local/cuda
12.1
running build
running build_py
running build_ext
/home/ren-admin/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
/home/ren-admin/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:415: UserWarning: The detected CUDA version (12.2) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
  warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/ren-admin/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no g++ version bounds defined for CUDA version 12.2
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
running install
/home/ren-admin/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/ren-admin/.local/lib/python3.10/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing manifest file 'MultiScaleDeformableAttention.egg-info/SOURCES.txt'
running install_lib
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/functions
creating build/bdist.linux-x86_64/egg/modules
byte-compiling build/bdist.linux-x86_64/egg/functions/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/functions/ms_deform_attn_func.py to ms_deform_attn_func.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/modules/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/modules/ms_deform_attn.py to ms_deform_attn.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/MultiScaleDeformableAttention.py to MultiScaleDeformableAttention.cpython-310.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying MultiScaleDeformableAttention.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MultiScaleDeformableAttention.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MultiScaleDeformableAttention.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MultiScaleDeformableAttention.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
__pycache__.MultiScaleDeformableAttention.cpython-310: module references __file__
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
removing '/home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg' (and everything under it)
creating /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg
Extracting MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg to /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages
byte-compiling /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/MultiScaleDeformableAttention.py to MultiScaleDeformableAttention.cpython-310.pyc
byte-compiling /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/functions/__init__.py to __init__.cpython-310.pyc
byte-compiling /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/functions/ms_deform_attn_func.py to ms_deform_attn_func.cpython-310.pyc
byte-compiling /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/modules/__init__.py to __init__.cpython-310.pyc
byte-compiling /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg/modules/ms_deform_attn.py to ms_deform_attn.cpython-310.pyc
MultiScaleDeformableAttention 1.0 is already the active version in easy-install.pth

Installed /home/ren-admin/miniconda3/envs/research-env/lib/python3.10/site-packages/MultiScaleDeformableAttention-1.0-py3.10-linux-x86_64.egg
Processing dependencies for MultiScaleDeformableAttention==1.0
Finished processing dependencies for MultiScaleDeformableAttention==1.0

The training works in Google Collab Pro and the configuration looks similar to my local setup.

CUDA Version (torch): 12.1 CUDA Version (system): 12.2 Torch Version: 2.1.0+cu121 Python 3.10.12

If anyone has tried object detection using Deformable DETR or FasterViT_DINO and has faced a similar issue, please provide me with the solution.

Your suggestions are greatly appreciated.

Regards, Bijay

TinyAway commented 6 months ago

Hi, have you solve this problem. I met the same issue with you.

bshakya77 commented 6 months ago

Yes, I resolved it. It was some CUDA and Pytorch version incompatibility.

Follow instructions in GitHub repo

https://github.com/fundamentalvision/Deformable-DETR

In my case, the following Pytorch version worked. Replace pytorch=1.7.1 !conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch

TinyAway commented 6 months ago

Yes, I resolved it. It was some CUDA and Pytorch version incompatibility.

Follow instructions in GitHub repo #https://github.com/fundamentalvision/Deformable-DETR

In my case, the following Pytorch version worked. Replace pytorch=1.7.1 !conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch

Thanks a lot. I solved it too!