[Bug] Unable to run BEVFusion Demo

Torreskai0722 commented 10 months ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0: NVIDIA GeForce RTX 3080 CUDA_HOME: /usr/local/cuda-12 NVCC: Cuda compilation tools, release 12.1, V12.1.66 GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.1.0+cu121 PyTorch compiling details: PyTorch built with:

GCC 9.3
C++ Version: 201703
Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 12.1
NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
CuDNN 8.9.2
Magma 2.6.1
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.16.0+cu121 OpenCV: 4.8.1 MMEngine: 0.9.1 MMDetection: 3.2.0 MMDetection3D: 1.2.0+12b595c spconv2.0: False

Reproduces the problem - code sample

(openmmlab) hydrapc@hydrapc:~/projects/DROI-BEV/mmdetection3d$ python projects/BEVFusion/demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py bevfusion_converted.pth --cam-type all --score-thr 0.2 --show

Reproduces the problem - command or script

(openmmlab) hydrapc@hydrapc:~/projects/DROI-BEV/mmdetection3d$ python projects/BEVFusion/demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py bevfusion_converted.pth --cam-type all --score-thr 0.2 --show

Reproduces the problem - error message

Traceback (most recent call last):
  File "/home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/utils/misc.py", line 77, in import_modules_from_strings
    imported_tmp = import_module(imp)
  File "/home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/__init__.py", line 1, in <module>
    from .bevfusion import BEVFusion
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/bevfusion.py", line 16, in <module>
    from .ops import Voxelization
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/ops/__init__.py", line 1, in <module>
    from .bev_pool import bev_pool
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/ops/bev_pool/__init__.py", line 1, in <module>
    from .bev_pool import bev_pool
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/ops/bev_pool/bev_pool.py", line 3, in <module>
    from . import bev_pool_ext
ImportError: /home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/bevfusion/ops/bev_pool/bev_pool_ext.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 462, in fromfile
    import_modules_from_strings(**cfg_dict['custom_imports'])
  File "/home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/utils/misc.py", line 84, in import_modules_from_strings
    raise ImportError(f'Failed to import {imp}')
ImportError: Failed to import projects.BEVFusion.bevfusion

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "projects/BEVFusion/demo/multi_modality_demo.py", line 78, in <module>
    main(args)
  File "projects/BEVFusion/demo/multi_modality_demo.py", line 42, in main
    model = init_model(args.config, args.checkpoint, device=args.device)
  File "/home/hydrapc/projects/DROI-BEV/mmdetection3d/mmdet3d/apis/inference.py", line 59, in init_model
    config = Config.fromfile(config)
  File "/home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 474, in fromfile
    raise ImportError(err_msg) from e
ImportError: Failed to import custom modules from {'imports': ['projects.BEVFusion.bevfusion'], 'allow_failed_imports': False}, the current sys.path is: 
    /home/hydrapc/projects/DROI-BEV/mmdetection3d/projects/BEVFusion/demo
    /home/hydrapc/anaconda3/envs/openmmlab/lib/python38.zip
    /home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8
    /home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/lib-dynload
    /home/hydrapc/anaconda3/envs/openmmlab/lib/python3.8/site-packages
    /home/hydrapc/projects/DROI-BEV/mmdetection3d
    /tmp/tmp_f0g2gch
You should set `PYTHONPATH` to make `sys.path` include the directory which contains your custom module

Additional information

I follow the installation of mmdetection3d but cannot run the demo for BEVFusion.

ranai-srivastav commented 10 months ago

I have the same issue as well.

Tried adding init.py to some folders and added the BEVFusion path to system path as well

Machine-NO-Learning commented 10 months ago

any solutions?

Machine-NO-Learning commented 10 months ago

there is one possible sol, config.py raise error is based on your import_module, but import_module raise error is not reason of wrong sys path, maybe you need install einops, so just pip install einops.

FraMarotta commented 9 months ago

I have the same problem, anyone has solved?

qiliang72 commented 9 months ago

Possible problem might be compatibility of gcc, g++, cuda version. Maybe you can try to use GCC 9.3 which is your pytorch compiler version. Also make sure to delete all your temporary files of previous builds. (This takes me too much time ><)

razgzy commented 9 months ago

I have the same problem

sss-stu-stupid commented 9 months ago

I have import error too, when from . import bev_pool_ext; but it describes the problem as {ImportError: cannot import name 'bev_pool_ext' from partially initialized module 'projects.BEVFusion.bevfusion.ops.bev_pool' (most likely due to a circular import) (mmdetection3d/projects/BEVFusion/bevfusion/ops/bev_pool/init.py)} This is the only difference in our logs.

BeMuCa commented 9 months ago

I have import error too, when from . import bev_pool_ext; but it describes the problem as {ImportError: cannot import name 'bev_pool_ext' from partially initialized module 'projects.BEVFusion.bevfusion.ops.bev_pool' (most likely due to a circular import) (mmdetection3d/projects/BEVFusion/bevfusion/ops/bev_pool/init.py)} This is the only difference in our logs.

This Solution from the OG BEVFusion Git solved it for me:

https://github.com/mit-han-lab/bevfusion/issues/504

ranai-srivastav commented 9 months ago

@BeMuCa I don't have a mmdetection3d/mmdet3d/ops folder. This is my folder structure

~/software/mmdetection3d/mmdet3d$ ls
apis  configs  datasets  engine  evaluation  __init__.py  models  __pycache__  registry.py  structures  testing  utils  version.py  visualization

Torreskai0722 commented 8 months ago

My issue is resolved after I uninstall and reinstall numba:

pip uninstall numba
pip install numba

Besides, I also installed spconv which causes another cuda runtime error:

pip install spconv-cu120

seven-linglx commented 8 months ago

cd mmdetection3d python projects/BEVFusion/setup.py develop

open-mmlab / mmdetection3d