nvToolsExt

[ikhann]

Hi!

I want to build an image for the aarch64 platform on my host machine. I added a building of isaac_ros_nitros in one of the layers in my user-defined Dockerfile:

RUN apt-get update && cd ${ROS_ROOT} \
    && source ${ROS_ROOT}/setup.bash \
    && rosdep install -y -r --ignore-src --from-paths src --rosdistro ${ROS_DISTRO} --skip-keys="fastrtps" \
    && colcon build --merge-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo --packages-up-to isaac_ros_apriltag isaac_ros_image_proc \
    && rm -Rf logs build \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

I have aarch64->humble->nav2->realsense->user pipeline of images before my dockerfile.

But during the compilation process, I get this error:

--- stderr: isaac_ros_nitros
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/types/nitros_type_base.cpp:18:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:118: CMakeFiles/isaac_ros_nitros.dir/src/types/nitros_type_base.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_format_agent.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_int64.hpp:21,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/types/nitros_int64.cpp:20:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:132: CMakeFiles/isaac_ros_nitros.dir/src/types/nitros_int64.cpp.o] Error 1
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:22,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_subscriber.hpp:24,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_subscriber.cpp:13:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:160: CMakeFiles/isaac_ros_nitros.dir/src/nitros_subscriber.cpp.o] Error 1
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:22,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher.hpp:24,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_group.hpp:35,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_publisher_subscriber_group.cpp:24:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:174: CMakeFiles/isaac_ros_nitros.dir/src/nitros_publisher_subscriber_group.cpp.o] Error 1
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:22,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher.hpp:24,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_publisher.cpp:13:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:146: CMakeFiles/isaac_ros_nitros.dir/src/nitros_publisher.cpp.o] Error 1
In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:22,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher.hpp:24,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_group.hpp:35,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_node.hpp:24,
                 from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_node.cpp:16:
/opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
   16 |   #include "nvToolsExt.h"
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:76: CMakeFiles/isaac_ros_nitros.dir/src/nitros_node.cpp.o] Error 1
/usr/bin/ld: cannot find -lnvToolsExt
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/isaac_ros_nitros_exe.dir/build.make:162: isaac_ros_nitros_exe] Error 1
make[1]: *** [CMakeFiles/Makefile2:165: CMakeFiles/isaac_ros_nitros_exe.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:139: CMakeFiles/isaac_ros_nitros.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
---
Failed   <<< isaac_ros_nitros [2min 14s, exited with code 2]

Could you please help how to resolve this issue?

Thanks in advance.

[Parosi]

Hi @ikhann,

I have the same problem, did you solve it?

Thanks.

[ikhann]

Hi @ikhann,

I have the same problem, did you solve it?

Thanks.

Hi! Unfortunately, no.

@jaiveersinghNV @hemalshahNV Could you help us, please?

[Parosi]

In my case I was building the package in the isaac ros docker on a Jetson device. I solved the problem by mapping the missing headers from the host to the docker. I don't know why these headers where missing in the docker image.

[ikhann]

In my case I was building the package in the isaac ros docker on a Jetson device. I solved the problem by mapping the missing headers from the host to the docker. I don't know why these headers where missing in the docker image.

Could you please add instructions for mapping the missing headers from the host to the docker? Thank you!

[Parosi]

In my case I was building the package in the isaac ros docker on a Jetson device. I solved the problem by mapping the missing headers from the host to the docker. I don't know why these headers where missing in the docker image.

Could you please add instructions for mapping the missing headers from the host to the docker? Thank you!

Yes follow these steps:

• Check if you have the following files in you host machine (xx.x is the version of cuda in your host system):

• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtCuda.h
• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtCudaRt.h
• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExt.h
• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtMeta.h
• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtSync.h
• • /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvtx3
• • /usr/local/cuda-xx.x/targets/aarch64-linux/lib/libnvToolsExt.so

• In my case the files exist in the host system but not inside the docker. So you can map them using the docker compose or the docker run command. For example I used the docker run command and I added the following lines (again xx.x is the version of cuda in you host system and yy.y is the version of cuda of the docker):

-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtCuda.h:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvToolsExtCuda.h \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtCudaRt.h:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvToolsExtCudaRt.h \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExt.h:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvToolsExt.h \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtMeta.h:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvToolsExtMeta.h \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvToolsExtSync.h:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvToolsExtSync.h \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/include/nvtx3:/usr/local/cuda-yy.y/targets/aarch64-linux/include/nvtx3 \
-v /usr/local/cuda-xx.x/targets/aarch64-linux/lib/libnvToolsExt.so:/usr/local/cuda-yy.y/targets/aarch64-linux/lib/libnvToolsExt.so \

Actually the error was only for the nvToolsExt library, that's why I added only the .so of that library. Probably you can remove the .h for the other libraries.

[hemalshahNV]

We do the same. We mount those files in run_dev.sh into your running container. We're looking to revamp our dev images so that we can make them more self-contained. In the meantime, you can take a look at work from Raffaello Bonghi here which might be helpful to you.

[batzor]

I was having same errors but had all the CUDA headers and libraries in the docker. To resolve, I had to add

find_package(CUDA  REQUIRED)
include_directories("${CUDA_INCLUDE_DIRS}")

to necessary CMakeLists (nearly all of them). But now when I colcon build, I am getting

In file included from /workspaces/isaac_ros-dev/src/isaac_ros_nitros/isaac_ros_nitros_type/isaac_ros_nitros_flat_scan_type/src/nitros_flat_scan.cpp:26:
/workspaces/isaac_ros-dev/src/isaac_ros_nitros/isaac_ros_nitros_type/isaac_ros_nitros_flat_scan_type/include/isaac_ros_nitros_flat_scan_type/nitros_flat_scan.hpp:24:10: fatal error: isaac_ros_pointcloud_interfaces/msg/flat_scan.hpp: No such file or directory
   24 | #include "isaac_ros_pointcloud_interfaces/msg/flat_scan.hpp"

Any idea how to resolve it? I am on Ubuntu 20.04 with dGPU CUDA11.8.

[hemalshahNV]

Have you tried deleting the 'build' and 'install' directories in your ROS workspace to try again? This file is new and the CMakeLists.txt files have been restructured a bit.

[batzor]

Have you tried deleting the 'build' and 'install' directories in your ROS workspace to try again? This file is new and the CMakeLists.txt files have been restructured a bit.

The issue still persists on rebuilding from scratch.

[batzor]

Updating isaac_ros_common made it work :sweat_smile: . Sorry for the hassle

[hemalshahNV]

No worries. Let me know if you still need the find_package lines you had added. They should not be necessary with the latest.