CPU Bottleneck on TX2?

[madebyollin]

EDIT 0: I think this may actually be an issue with not having run jetson_clock.sh before testing. Will update with results.

EDIT 1: After more testing, it definitely seems like not running jetson_clock.sh first was the issue. I get 14-20FPS on monocular now, depending on feature counts... Stereo is still slow, but that's probably expected.

---

Using the project presentation website and this thread on the NVIDIA forums, I've gotten this project running live on a Jetson TX2, with a lightly modified version of the original ROS Stereo node.

However, the performance of the GPU version in my install is generally slightly worse than the CPU version.

I'm curious if there are any suggestions for things to check or investigate while diagnosing this issue. Thanks!

---

More Explanation:

On a sample BAG file of 672x376 (ZED WVGA) stereo images broadcast at 30FPS, I can get around 5-6FPS with the original CPU-bound ORB_SLAM2, vs. around 4-5FPS for the GPU version. As a sanity check, the tx1 executable in /gpu gives ~8-9FPS monocular.

Here's a sample graph of what sudo ~/tegrastats gives for processor % utilization during the GPU version on the BAG file images: And the CPU and mono versions, for reference:

The GPU is being utilized in the GPU version, but it seems like it's spending most of the time waiting for data, which suggests that the CPU operations are bottlenecked somehow.

---

Things that shouldn't be the issue:

• The TX2 is running on Mode 0 (sudo nvpmodel -m 0).
• I've compiled both versions of OpenCV (ROS version and standalone) with CUDA, cuBLAS and fast math flags enabled:
  • OpenCV 3.2 (ROS, used when building the ROS node):

    -DCMAKE_BUILD_TYPE=Release \                 
    -DBUILD_PNG=OFF \
    -DBUILD_TIFF=OFF \
    -DBUILD_TBB=OFF \
    -DBUILD_JPEG=OFF \
    -DBUILD_JASPER=OFF \
    -DBUILD_ZLIB=OFF \
    -DBUILD_EXAMPLES=OFF \
    -DBUILD_opencv_java=OFF \
    -DBUILD_opencv_python2=ON \
    -DBUILD_opencv_python3=OFF \
    -DENABLE_PRECOMPILED_HEADERS=OFF \
    -DWITH_OPENCL=OFF \
    -DWITH_OPENMP=OFF \
    -DWITH_FFMPEG=ON \
    -DWITH_GSTREAMER=ON \
    -DWITH_GSTREAMER_0_10=OFF \
    -DWITH_GTK=ON \
    -DWITH_VTK=ON \
    -DWITH_TBB=ON \
    -DWITH_1394=OFF \
    -DWITH_OPENEXR=OFF \
    -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-8.0 \
    -DCUDA_ARCH_BIN=6.2 \
    -DCUDA_ARCH_PTX="" \
    -DINSTALL_C_EXAMPLES=OFF \
    -DINSTALL_TESTS=OFF \
    -DWITH_CUDA=ON \
    -DWITH_OPENCL=OFF \
    -DENABLE_FAST_MATH=1 \
    -DCUDA_FAST_MATH=1 \
    -DWITH_CUBLAS=1 \
    -DBUILD_DOCS=OFF \
    -DBUILD_PERF_TESTS=OFF \
    -DBUILD_TESTS=OFF \

  • OpenCV 3.3 (Standalone, used when building the project itself):

    
    -DCMAKE_BUILD_TYPE=Release \                 
    -DBUILD_PNG=OFF \                            
    -DBUILD_TIFF=OFF \                           
    -DBUILD_TBB=OFF \                            
    -DBUILD_JPEG=OFF \                           
    -DBUILD_JASPER=OFF \                         
    -DBUILD_ZLIB=OFF \                           
    -DBUILD_EXAMPLES=OFF \                       
    -DBUILD_opencv_java=OFF \                    
    -DBUILD_opencv_python2=ON \                  
    -DBUILD_opencv_python3=OFF \                 
    -DENABLE_PRECOMPILED_HEADERS=OFF \           
    -DWITH_OPENCL=OFF \                          
    -DWITH_OPENMP=OFF \                          
    -DWITH_FFMPEG=ON \                           
    -DWITH_GSTREAMER=ON \                        
    -DWITH_GSTREAMER_0_10=OFF \                  
    -DWITH_GTK=ON \                              
    -DWITH_VTK=OFF \                             
    -DWITH_TBB=ON \                              
    -DWITH_1394=OFF \                            
    -DWITH_OPENEXR=OFF \                         
    -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-8.0 \
    -DCUDA_ARCH_BIN=6.2 \                        
    -DCUDA_ARCH_PTX="" \                         
    -DINSTALL_C_EXAMPLES=OFF \                   
    -DINSTALL_TESTS=OFF \                        
    -DCMAKE_INSTALL_PREFIX=/usr/local \          
    -DWITH_CUDA=ON \                             
    -DWITH_OPENCL=OFF \                          
    -DENABLE_FAST_MATH=1 \                       
    -DCUDA_FAST_MATH=1 \                         
    -DWITH_CUBLAS=1 \                            
    -DBUILD_DOCS=OFF \                           
    -DBUILD_PERF_TESTS=OFF \                      
    -DBUILD_TESTS=OFF

[yunchih]

Hi, @madebyollin Thanks for the feedback! I don't have further comments regarding performance, but I do appreciate your writeup.