search

NVIDIA-ISAAC-ROS / isaac_ros_nvblox

NVIDIA-accelerated 3D scene reconstruction and Nav2 local costmap provider using nvblox
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
443 stars 81 forks source link

CMake (Colcon) can not find nvToolsExt #63

Closed SimonSchwaiger closed 1 year ago

SimonSchwaiger commented 1 year ago

Hello, I have been trying to follow the Tutorial For Realsense-based Reconstruction on a Jetson Xavier NX 16GB board running a freshly flashed Jetpack 5.1.1 install. As suggested by the tutorial, all components are set up in the dev Docker container on an external SSD and all dependencies are able to be resolved by rosdep.

However, when I try to build my ros 2 workspace using colcon, I get an error indicating that CMake can not find nvToolsExt. When running the Docker container and nvblox environment as shown in the tutorial, I get the following error: 

Starting >>> nvblox
--- stderr: nvblox                                                                            
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_nvToolsExt_LIBRARY (ADVANCED)
    linked by target "nvblox_lib" in directory /workspaces/isaac_ros-dev/src/isaac_ros_nvblox/nvblox/nvblox
    linked by target "nvblox_gpu_hash" in directory /workspaces/isaac_ros-dev/src/isaac_ros_nvblox/nvblox/nvblox

CMake Generate step failed.  Build files cannot be regenerated correctly.
---
Failed   <<< nvblox [8.88s, exited with code 1]
Aborted  <<< isaac_ros_nitros [33.5s]

I have verified that cuda-nvtx-11-4 is installed on the host and that the shared objects relating to nvToolsExt as well as nvToolsExtCuda.h are present. Furthemore, I did try to alter the run_dev.sh script to mount libnvToolsExt.so.1.0.0 and libnvToolsExt.so.1 to the container in addition to libnvToolsExt.so, which is mounted by default.

As a last resort, I have tried to manually set CUDA_nvToolsExt_LIBRARY statically in nvblox' CmakeLists.txt to point to the correct libraries. While the build progresses further than before, the linker produces the following error, saying that it can not link to nvToolsExt:

Starting >>> nvblox
[Processing: isaac_ros_nitros, nvblox]                                                            
--- stderr: isaac_ros_nitros                                                                          
/usr/bin/ld: cannot find -lnvToolsExt
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/isaac_ros_nitros_exe.dir/build.make:162: isaac_ros_nitros_exe] Error 1
make[1]: *** [CMakeFiles/Makefile2:185: CMakeFiles/isaac_ros_nitros_exe.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: cannot find -lnvToolsExt
/usr/bin/ld: cannot find -lnvToolsExt
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:359: libisaac_ros_nitros.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:159: CMakeFiles/isaac_ros_nitros.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
---
Failed   <<< isaac_ros_nitros [51.2s, exited with code 2]
Aborted  <<< nvblox [1min 48s]

Unfortunately I have arrived at a dead end when researching solutions.

Any help would be greatly appreciated. Best regards and thank you in advance.

-- Edit: Attached CMake logfile from nvblox

steams.log

jaiveersinghNV commented 1 year ago

Could you please double-check the following:

SimonSchwaiger commented 1 year ago

Thank you for the quick reply! I have verified that /usr/local/cuda-11.4/targets/aarch64-linux/include/nvToolsExt.h exists on both the host and the container. Unfortunately, deleting the build log and install directories and rebuilding does not fix the error.

I did some further debugging over the weekend to isolate, whether or not the issue stems from the Docker setup or something else. On the host, cuda-nvtx-11-4 seems to be installed correctly according to apt-cache, however, CMake fails to find nvToolsExt even outside of the Docker container. CMake finds other components, such as cudart for example, without any issue.

jaiveersinghNV commented 1 year ago

Just to be sure, does the corresponding .so file also exist? /usr/local/cuda-11.4/targets/aarch64-linux/lib/nvToolsExt.so

Could you try uninstalling and reinstalling cuda-nvtx-11-4 via apt?

SimonSchwaiger commented 1 year ago

I can confirm that the .so file did exist as well. Interestingly, I could not get the extensions to be detected and only a complete reflash of the Jetson's OS allowed me to solve the issue. The install procedure and utilised Jetpack version are the same as the previous install; only now nvtx is found without any problem.

Since I managed to get everything working, I will close the issue. Thank you again for your help and best regards.

BADAL244 commented 1 year ago

This also happen when you building a package that already build in some other workspace so if you already build a package in some other workspace just source it and build the remaining packages.

dakejahl commented 1 year ago

I am also running into this same issue, however it is only affecting the rviz2 plugin

--- stderr: nvblox_rviz_plugin                                                                                                             
/usr/bin/ld: cannot find -lnvToolsExt
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/nvblox_rviz_plugin.dir/build.make:308: libnvblox_rviz_plugin.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:138: CMakeFiles/nvblox_rviz_plugin.dir/all] Error 2
make: *** [Makefile:146: all] Error 2