NVIDIA-AI-IOT / Lidar_AI_Solution

A project demonstrating Lidar related AI solutions, including three GPU accelerated Lidar/camera DL networks (PointPillars, CenterPoint, BEVFusion) and the related libs (cuPCL, 3D SparseConvolution, YUV2RGB, cuOSD,).
Other
1.35k stars 239 forks source link

Core dumped when trying the CUDA-BEVFusion #60

Closed nguyenlam185 closed 1 year ago

nguyenlam185 commented 1 year ago

I am trying build the BEVFusion base on the guidelines in README.md. Everythings seem find until the Compiling process has finished, the bevfusion cannot run. Could anybody please give me some clues to get it work?

bash tool/run.sh 
==========================================================
||  MODEL: resnet50int8
||  PRECISION: int8
||  DATA: /home/docker_user/shared_dir/Lidar_AI_Solution/CUDA-BEVFusion/example-data
||  USEPython: ON
||
||  TensorRT: /usr/lib/x86_64-linux-gnu/
||  CUDA: /usr/local/cuda-11.8
||  CUDNN: /home/docker_user/.local/lib/python3.8/site-packages/torch/lib
==========================================================
Find Python_Inc: /usr/include/python3.8
Find Python_Lib: /usr/lib
Find Python_Soname: libpython3.8.so
Try to get the current device SM
Current CUDA SM: 86
Configuration done!
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda-11.8 (found version "11.8") 
-- Found Protobuf: /usr/local/lib/libprotobuf.so;-lpthread (found version "3.21.12") 
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/docker_user/shared_dir/Lidar_AI_Solution/CUDA-BEVFusion/build
[  4%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/common/bevfusion_core_generated_tensor.cu.o
[  8%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-normalization.cu.o
[ 12%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-bevpool.cu.o
[ 16%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-backbone.cu.o
[ 25%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-vtransform.cu.o
[ 25%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-depth.cu.o
[ 29%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_camera-geometry.cu.o
[ 33%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_lidar-voxelization.cu.o
[ 41%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_transfusion.cu.o
[ 41%] Building NVCC (Device) object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion_core_generated_head-transbbox.cu.o
[ 45%] Building CXX object CMakeFiles/bevfusion_core.dir/src/bevfusion/bevfusion.cpp.o
[ 50%] Building CXX object CMakeFiles/bevfusion_core.dir/src/bevfusion/lidar-scn.cpp.o
[ 54%] Building CXX object CMakeFiles/bevfusion_core.dir/src/common/tensorrt.cpp.o
[ 58%] Linking CXX shared library libbevfusion_core.so
[ 58%] Built target bevfusion_core
[ 62%] Building NVCC (Device) object CMakeFiles/bevfusion.dir/src/common/bevfusion_generated_visualize.cu.o
[ 66%] Building NVCC (Device) object CMakeFiles/bevfusion.dir/__/libraries/cuOSD/src/bevfusion_generated_cuosd_kernel.cu.o
[ 70%] Building CXX object CMakeFiles/pybev.dir/src/python.cpp.o
[ 75%] Building CXX object CMakeFiles/bevfusion.dir/home/docker_user/shared_dir/Lidar_AI_Solution/libraries/cuOSD/src/cuosd.cpp.o
[ 79%] Building CXX object CMakeFiles/bevfusion.dir/home/docker_user/shared_dir/Lidar_AI_Solution/libraries/cuOSD/src/textbackend/backend.cpp.o
[ 83%] Building CXX object CMakeFiles/bevfusion.dir/home/docker_user/shared_dir/Lidar_AI_Solution/libraries/cuOSD/src/textbackend/stb.cpp.o
[ 91%] Building CXX object CMakeFiles/bevfusion.dir/src/main.cpp.o
[ 91%] Building CXX object CMakeFiles/bevfusion.dir/home/docker_user/shared_dir/Lidar_AI_Solution/libraries/cuOSD/src/textbackend/pango-cairo.cpp.o
[ 95%] Linking CXX shared library libpybev.so
[ 95%] Built target pybev
[100%] Linking CXX executable bevfusion
[100%] Built target bevfusion
tool/run.sh: line 47: 42789 Segmentation fault      (core dumped) ./build/bevfusion $DEBUG_DATA $DEBUG_MODEL $DEBUG_PRECISION
hopef commented 1 year ago

You can try debugging this program to find out where the error is. Usually, it is caused by a file configuration error, or a problem with lfs.

nguyenlam185 commented 1 year ago

@hopef Thanks for your suggestion. I have tried debugging and find out that the problem may come from the protobuf library. Do I have to use a specific version of protobuf? or just go with the latest one? My environment now installed with protobuf v3.21.12

hopef commented 1 year ago

libprotobuf-dev == 3.6.1

nguyenlam185 commented 1 year ago

thank you so much @hopef It now works well with protobuf v3.6.1

hunagjingwei commented 1 year ago

hi, how do you fix this problem , i try to compile the protobuf v3.6.1 ,but it can't to solve problem .do you apt to install protobuf? I can't install success

nguyenlam185 commented 1 year ago

@hunagjingwei I built protobuf from source

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git checkout v3.6.1
git submodule update --init --recursive
./autogen.sh
./configure
make -j$(nproc)
make -j$(nproc) check
sudo make install