Megvii-BaseDetection / YOLOX

YOLOX is a high-performance anchor-free YOLO, exceeding yolov3~v5 with MegEngine, ONNX, TensorRT, ncnn, and OpenVINO supported. Documentation: https://yolox.readthedocs.io/
Apache License 2.0
9.29k stars 2.18k forks source link

Segmentation Fault (core dumped) occurs when YOLOX is converted to TensorRT. #916

Open Rehide opened 2 years ago

Rehide commented 2 years ago

I tried to convert YOLOX-S to TensorRT with Jetson. python3 tools/trt.py -n yolox-s -c yolox_s.pth

However, I did not succeed, just getting Segmentation Fault. Does anyone know how to solve this problem?

My environment is as follows.

Machine: Jetson Xavier NX, Jetson AGX Xavier OS: Ubuntu18.04LTS JetPack: 4.5.1 CUDA: 10.2.89 cuDNN: 8.0.0.180 TensorRT: 7.1.3.0 Python: 3.6.9 torch: 1.8.0 torchvision: 0.9.0 torch2trt: 0.3.0 yolox: 0.1.0

thanks

g7xcy commented 2 years ago

I met this problem, too. I wanna know how to deal with it?

nocolour commented 2 years ago

YoloX and yolov5 python env setup are similar. Please refer below. Problem May solve.

This problem happening on yolov5 also.

https://cognitivexr.at/blog/2021/03/11/installing-pytorch-and-yolov5-on-an-nvidia-jetson-xavier-nx.html

If the Python process terminates with Illegal instruction (core dumped), it’s likely related to an issue with numpy 1.19.5 and OpenBLAS. Either run export OPENBLAS_CORETYPE=ARMV8, set it in your .bashrc file, or downgrade to numpy 1.19.4 by running pip3 install -U "numpy==1.19.4".

try run this 1st: export OPENBLAS_CORETYPE=ARMV8

if still problem then try downgrade numpy version.

tulbureandreit commented 2 years ago

@g7xcy @nocolour I have the same issue.

I have nmpy 1.19.4 installed and I ran export OPENBLAS.

When I run: sudo python3 tools/trt.py -n yolox-s yolo_s.pth I get a segmentation fault as a result without any additional info.

I have pytorch 1.8, torchvision 0.9, nvidia jetson xavier AGX with deepstream 6.0 and jetpack 4.6.

I try to run deepsort tracking with yolox on it.

tulbureandreit commented 2 years ago

@Rehide have you solved it ?

FateScript commented 2 years ago

What about trying code in #1118 ? We never try device like Jetson.

nocolour commented 2 years ago

I just reformat my Nvidia JetsonAGX. Just done all setup and requirement install. yolov5 and yolox tested working fine. below my setup note. Hope can help your.

Tested on Jetson Pack 4.6. (AGX Xavier)

  1. )Run script setup-xavier.sh (refer attachment setup-xavier.tx)

    • convert setup-xavier.txt to setup-xavier.sh
  2. )vim ~/.bashrc (Add below) export OPENBLAS_CORETYPE=ARMV8 export PATH=/usr/local/cuda/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

3.)REBOOT DEVICE !

4.)build latest Opencv-4.5.5

sudo apt-get update sudo apt-get install cmake sudo apt-get install gcc g++ sudo apt-get install python3-dev python3-numpy sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev sudo apt-get install libgtk2.0-dev sudo apt-get install libgtk-3-dev sudo apt-get install libpng-dev sudo apt-get install libjpeg-dev sudo apt-get install libopenexr-dev sudo apt-get install libtiff-dev sudo apt-get install libwebp-dev sudo apt-get install libxvidcore-dev libx264-dev sudo apt-get install libv4l-dev v4l-utils v4l2ucp qv4l2 libdc1394-22-dev libavresample-dev sudo apt-get install libblas-dev liblapack-dev libeigen3-dev

cd ~

wget -O opencv.zip https://github.com/opencv/opencv/archive/4.5.5.zip

wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.5.zip

unzip opencv.zip

unzip opencv_contrib.zip

cd ~/opencv-4.5.5

mkdir build

cd build

vim cmake-setup.sh (add below) (refer cmake-setup.txt)

!/bin/bash

OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-4.5.5/modules

cmake -D CMAKE_BUILD_TYPE=RELEASE \ -D CMAKE_INSTALL_PREFIX=/usr \ -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-4.5.5/modules \ -D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \ -D WITH_OPENCL=OFF \ -D WITH_CUDA=ON \ -D CUDA_ARCH_BIN=7.2 \ -D CUDA_ARCH_PTX="" \ -D WITH_CUDNN=ON \ -D WITH_CUBLAS=ON \ -D ENABLE_FAST_MATH=ON \ -D CUDA_FAST_MATH=ON \ -D OPENCV_DNN_CUDA=ON \ -D ENABLE_NEON=ON \ -D WITH_QT=OFF\ -D WITH_OPENMP=ON \ -D WITH_OPENGL=ON \ -D BUILD_TIFF=ON \ -D WITH_FFMPEG=ON \ -D WITH_GSTREAMER=ON \ -D WITH_TBB=ON \ -D BUILD_TBB=ON \ -D BUILD_TESTS=OFF \ -D WITH_EIGEN=ON \ -D WITH_V4L=ON \ -D WITH_LIBV4L=ON \ -D OPENCV_ENABLE_NONFREE=ON \ -D INSTALL_C_EXAMPLES=OFF \ -D INSTALL_PYTHON_EXAMPLES=OFF \ -D BUILD_opencv_python3=TRUE \ -D OPENCV_GENERATE_PKGCONFIG=ON \ -D BUILD_EXAMPLES=OFF .. make -j$(nproc) sudo make install

5.)run script cmake-setup.sh (run in build folder)

reboot setup-xavier.txt

setup-xavier.txt

yolov5.txt

nocolour commented 2 years ago

tested

nniv-cmyk commented 2 years ago

@nocolour YOLOX works fine, the problem is with converting to TensorRT. I am having this same problem on the Jetson nano 4GB JetPack4.6, Cuda10.2 CuDNN 8.2 TensorRT 8.0.1.6 Python3.6.9 OS: Ubuntu18.04LTS Torch1.9.0

tulbureandreit commented 2 years ago

@nocolour @loyclay I have successfully run YOLOX+deepsort on my AGX

I could only convert YOLOX_S to tensortRT and then use it for my deepstream pipeline. All the other smaller models were not working.