search

NVIDIA-ISAAC-ROS / isaac_ros_nitros

NVIDIA Isaac Transport for ROS package for hardware-acceleration friendly movement of messages
Other
95 stars 18 forks source link

nvToolsExt.h error during a building in Dockerfile #20

Closed ikhann closed 1 year ago

ikhann commented 1 year ago

Hi,

My System: Jetson Orin NX Ubuntu 20.04 Jetpack 5.1

I need to build Ros Isaac Nodes during a Dockerfile building on the Jetson. I added some layers in Dockerfile.user after "INSTALL ISAAC ROS" comment:

# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

# Setup non-root admin user
ARG USERNAME=admin
ARG USER_UID=1000
ARG USER_GID=1000

# Install prerequisites
RUN apt-get update && apt-get install -y \
        sudo \
        udev \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean

# Reuse triton-server user as 'admin' user if exists
RUN if [ $(getent group triton-server) ]; then \
        groupmod -o --gid ${USER_GID} -n ${USERNAME} triton-server ; \
        usermod -l ${USERNAME} -u ${USER_UID} -m -d /home/${USERNAME} triton-server ; \
        mkdir -p /home/${USERNAME} ; \
        sudo chown ${USERNAME}:${USERNAME} /home/${USERNAME} ; \
    fi

# Create the 'admin' user if not already exists
RUN if [ ! $(getent passwd ${USERNAME}) ]; then \
        groupadd --gid ${USER_GID} ${USERNAME} ; \
        useradd --uid ${USER_UID} --gid ${USER_GID} -m ${USERNAME} ; \
    fi

# Update 'admin' user
RUN echo ${USERNAME} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME} \
    && chmod 0440 /etc/sudoers.d/${USERNAME} \
    && adduser ${USERNAME} video && adduser ${USERNAME} plugdev && adduser ${USERNAME} sudo

# Copy scripts
RUN mkdir -p /usr/local/bin/scripts
COPY scripts/*entrypoint.sh /usr/local/bin/scripts/
RUN  chmod +x /usr/local/bin/scripts/*.sh

# Copy middleware profiles
RUN mkdir -p /usr/local/share/middleware_profiles
COPY middleware_profiles/*profile.xml /usr/local/share/middleware_profiles/

ENV USERNAME=${USERNAME}
ENV USER_GID=${USER_GID}
ENV USER_UID=${USER_UID}

# INSTALL ISAAC ROS

WORKDIR ${ROS_ROOT}/src
RUN git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_apriltag \
    && git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline \
    && git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

RUN apt-get update && cd ${ROS_ROOT} \
    && source ${ROS_ROOT}/setup.bash \
    && rosdep install -y -r --ignore-src --from-paths src --rosdistro ${ROS_DISTRO} --skip-keys="fastrtps" \
    && colcon build --merge-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo --packages-up-to isaac_ros_apriltag isaac_ros_image_proc \
    && rm -Rf logs build \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

But, I get the errors about missing the header file nvToolsExt.h:

nvidia@tegra-ubuntu:~/workspaces/isaac_dev/src/isaac_ros_common$ cd ~/workspaces/isaac_dev/src/isaac_ros_common &&   ./scripts/run_dev.sh
isaac_ros_dev not specified, assuming /home/nvidia/workspaces/isaac_dev/src/isaac_ros_common
Building aarch64.ros2_humble.user base as image: isaac_ros_dev-aarch64 using key aarch64.ros2_humble.user
Using base image name not specified, using ''
Using docker context dir not specified, using Dockerfile directory
Resolved the following Dockerfiles for target image: aarch64.ros2_humble.user
/home/nvidia/workspaces/isaac_dev/src/isaac_ros_common/scripts/../docker/Dockerfile.user
/home/nvidia/workspaces/isaac_dev/src/isaac_ros_common/scripts/../docker/Dockerfile.aarch64.ros2_humble
Building /home/nvidia/workspaces/isaac_dev/src/isaac_ros_common/scripts/../docker/Dockerfile.aarch64.ros2_humble as image: aarch64-ros2_humble-image with base: 
[+] Building 0.1s (5/5) FINISHED                                                                                                                    
 => [internal] load build definition from Dockerfile.aarch64.ros2_humble                                                                       0.0s
 => => transferring dockerfile: 575B                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [internal] load metadata for nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_c8ad2c6ecc68747eca2f1df62e25ae5e                                 0.0s
 => CACHED [1/1] FROM nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_c8ad2c6ecc68747eca2f1df62e25ae5e                                            0.0s
 => exporting to image                                                                                                                         0.0s
 => => exporting layers                                                                                                                        0.0s
 => => writing image sha256:a8b1b944a2887ae21de58136aac7c36068913c4751939f706e51d7a4427a8a7a                                                   0.0s
 => => naming to docker.io/library/aarch64-ros2_humble-image                                                                                   0.0s
Building /home/nvidia/workspaces/isaac_dev/src/isaac_ros_common/scripts/../docker/Dockerfile.user as image: isaac_ros_dev-aarch64 with base: aarch64-ros2_humble-image
[+] Building 143.6s (17/17) FINISHED                                                                                                                
 => [internal] load build definition from Dockerfile.user                                                                                      0.0s
 => => transferring dockerfile: 2.60kB                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/aarch64-ros2_humble-image:latest                                                            0.0s
 => [ 1/13] FROM docker.io/library/aarch64-ros2_humble-image                                                                                   0.0s
 => [internal] load build context                                                                                                              0.0s
 => => transferring context: 179B                                                                                                              0.0s
 => CACHED [ 2/13] RUN apt-get update && apt-get install -y         sudo         udev && rm -rf /var/lib/apt/lists/* && apt-get clean          0.0s
 => CACHED [ 3/13] RUN if [ $(getent group triton-server) ]; then         groupmod -o --gid 132 -n admin triton-server ;         usermod -l a  0.0s
 => CACHED [ 4/13] RUN if [ ! $(getent passwd admin) ]; then         groupadd --gid 132 admin ;         useradd --uid 1000 --gid 132 -m admin  0.0s
 => CACHED [ 5/13] RUN echo admin ALL=(root) NOPASSWD:ALL > /etc/sudoers.d/admin     && chmod 0440 /etc/sudoers.d/admin     && adduser admin   0.0s
 => CACHED [ 6/13] RUN mkdir -p /usr/local/bin/scripts                                                                                         0.0s
 => CACHED [ 7/13] COPY scripts/*entrypoint.sh /usr/local/bin/scripts/                                                                         0.0s
 => CACHED [ 8/13] RUN  chmod +x /usr/local/bin/scripts/*.sh                                                                                   0.0s
 => CACHED [ 9/13] RUN mkdir -p /usr/local/share/middleware_profiles                                                                           0.0s
 => CACHED [10/13] COPY middleware_profiles/*profile.xml /usr/local/share/middleware_profiles/                                                 0.0s
 => CACHED [11/13] WORKDIR /opt/ros/humble/src                                                                                                 0.0s
 => [12/13] RUN git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_apriltag     && git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_  96.8s
 => ERROR [13/13] RUN apt-get update && cd /opt/ros/humble     && source /opt/ros/humble/setup.bash     && rosdep install -y -r --ignore-src  46.7s
------                                                                                                                                              
 > [13/13] RUN apt-get update && cd /opt/ros/humble     && source /opt/ros/humble/setup.bash     && rosdep install -y -r --ignore-src --from-paths src --rosdistro humble --skip-keys="fastrtps"     && colcon build --merge-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo --packages-up-to isaac_ros_image_proc     && rm -Rf logs build     && rm -rf /var/lib/apt/lists/*     && apt-get clean:                                                 
#17 0.385 Get:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]                                                                       
#17 0.522 Get:2 http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu focal InRelease [23.8 kB]                                               
#17 0.679 Get:3 http://packages.ros.org/ros2-testing/ubuntu focal InRelease [4,685 B]
#17 0.705 Get:4 https://repo.download.nvidia.com/jetson/common r35.2 InRelease [2,555 B]
#17 0.768 Get:5 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
#17 0.856 Get:6 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease [108 kB]
#17 0.934 Get:7 http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu focal/main arm64 Packages [2,597 B]
#17 0.947 Get:8 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
#17 1.057 Get:9 http://packages.ros.org/ros2-testing/ubuntu focal/main arm64 Packages [1,169 kB]
#17 1.142 Get:10 https://apt.kitware.com/ubuntu focal InRelease [15.5 kB]
#17 1.154 Get:11 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages [1,234 kB]
#17 1.340 Get:13 https://repo.download.nvidia.com/jetson/common r35.2/main arm64 Packages [19.8 kB]
#17 1.480 Get:14 http://ports.ubuntu.com/ubuntu-ports focal/multiverse arm64 Packages [139 kB]
#17 1.540 Get:15 http://ports.ubuntu.com/ubuntu-ports focal/restricted arm64 Packages [1,317 B]
#17 1.541 Get:16 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 Packages [11.1 MB]
#17 2.044 Get:17 https://apt.kitware.com/ubuntu focal/main arm64 Packages [50.6 kB]
#17 2.478 Get:18 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 Packages [1,238 kB]
#17 2.535 Get:19 http://ports.ubuntu.com/ubuntu-ports focal-updates/restricted arm64 Packages [5,435 B]
#17 2.535 Get:20 http://ports.ubuntu.com/ubuntu-ports focal-updates/multiverse arm64 Packages [9,072 B]
#17 2.692 Get:12 https://packagecloud.io/github/git-lfs/ubuntu focal InRelease [24.4 kB]
#17 2.730 Get:21 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages [2,308 kB]
#17 3.751 Get:22 http://ports.ubuntu.com/ubuntu-ports focal-backports/universe arm64 Packages [27.8 kB]
#17 3.787 Get:23 http://ports.ubuntu.com/ubuntu-ports focal-backports/main arm64 Packages [54.8 kB]
#17 3.787 Get:24 http://ports.ubuntu.com/ubuntu-ports focal-security/multiverse arm64 Packages [3,260 B]
#17 3.788 Get:25 http://ports.ubuntu.com/ubuntu-ports focal-security/restricted arm64 Packages [5,200 B]
#17 3.788 Get:26 http://ports.ubuntu.com/ubuntu-ports focal-security/main arm64 Packages [1,919 kB]
#17 4.150 Get:27 http://ports.ubuntu.com/ubuntu-ports focal-security/universe arm64 Packages [937 kB]
#17 6.263 Fetched 20.9 MB in 6s (3,443 kB/s)
#17 6.263 Reading package lists...
#17 9.686 ERROR: the following packages/stacks could not have their rosdep keys resolved
#17 9.686 to system dependencies:
#17 9.686 isaac_ros_nitros_detection2_d_array_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_image_proc: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_tensor_list_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_gxf: Cannot locate rosdep definition for [isaac_ros_common]
#17 9.686 isaac_ros_nitros_disparity_image_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_image_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_compressed_image_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_april_tag_detection_array_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_pose_cov_stamped_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_std_msg_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_imu_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_flat_scan_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_occupancy_grid_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_interfaces: Cannot locate rosdep definition for [isaac_ros_common]
#17 9.686 isaac_ros_nitros: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_camera_info_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_point_cloud_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_stereo_image_proc: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_nitros_pose_array_type: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 isaac_ros_apriltag: Cannot locate rosdep definition for [isaac_ros_test]
#17 9.686 Continuing to install resolvable dependencies...
#17 9.686 #All required rosdeps installed successfully
#17 11.09 Starting >>> isaac_ros_gxf
#17 11.12 Starting >>> isaac_ros_nitros_interfaces
#17 14.05 Finished <<< isaac_ros_gxf [2.96s]
#17 18.03 Finished <<< isaac_ros_nitros_interfaces [6.90s]
#17 18.03 Starting >>> isaac_ros_nitros
#17 46.36 --- stderr: isaac_ros_nitros
#17 46.36 In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/types/nitros_type_base.cpp:18:
#17 46.36 /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
#17 46.36    16 |   #include "nvToolsExt.h"
#17 46.36       |            ^~~~~~~~~~~~~~
#17 46.36 compilation terminated.
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:118: CMakeFiles/isaac_ros_nitros.dir/src/types/nitros_type_base.cpp.o] Error 1
#17 46.36 make[2]: *** Waiting for unfinished jobs....
#17 46.36 In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_format_agent.hpp:21,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_empty.hpp:21,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/types/nitros_empty.cpp:20:
#17 46.36 /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
#17 46.36    16 |   #include "nvToolsExt.h"
#17 46.36       |            ^~~~~~~~~~~~~~
#17 46.36 compilation terminated.
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:132: CMakeFiles/isaac_ros_nitros.dir/src/types/nitros_empty.cpp.o] Error 1
#17 46.36 In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:25,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_subscriber.hpp:24,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_subscriber.cpp:13:
#17 46.36 /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
#17 46.36    16 |   #include "nvToolsExt.h"
#17 46.36       |            ^~~~~~~~~~~~~~
#17 46.36 compilation terminated.
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:160: CMakeFiles/isaac_ros_nitros.dir/src/nitros_subscriber.cpp.o] Error 1
#17 46.36 In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:25,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher.hpp:24,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_publisher.cpp:13:
#17 46.36 /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
#17 46.36    16 |   #include "nvToolsExt.h"
#17 46.36       |            ^~~~~~~~~~~~~~
#17 46.36 compilation terminated.
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:146: CMakeFiles/isaac_ros_nitros.dir/src/nitros_publisher.cpp.o] Error 1
#17 46.36 In file included from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/nitros_type_base.hpp:19,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_base.hpp:25,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher.hpp:24,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_publisher_subscriber_group.hpp:35,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/nitros_node.hpp:24,
#17 46.36                  from /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/src/nitros_node.cpp:16:
#17 46.36 /opt/ros/humble/src/isaac_ros_nitros/isaac_ros_nitros/include/isaac_ros_nitros/types/type_utility.hpp:16:12: fatal error: nvToolsExt.h: No such file or directory
#17 46.36    16 |   #include "nvToolsExt.h"
#17 46.36       |            ^~~~~~~~~~~~~~
#17 46.36 compilation terminated.
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros.dir/build.make:76: CMakeFiles/isaac_ros_nitros.dir/src/nitros_node.cpp.o] Error 1
#17 46.36 /usr/bin/ld: cannot find -lnvToolsExt
#17 46.36 collect2: error: ld returned 1 exit status
#17 46.36 make[2]: *** [CMakeFiles/isaac_ros_nitros_exe.dir/build.make:162: isaac_ros_nitros_exe] Error 1
#17 46.36 make[1]: *** [CMakeFiles/Makefile2:185: CMakeFiles/isaac_ros_nitros_exe.dir/all] Error 2
#17 46.36 make[1]: *** Waiting for unfinished jobs....
#17 46.36 make[1]: *** [CMakeFiles/Makefile2:159: CMakeFiles/isaac_ros_nitros.dir/all] Error 2
#17 46.36 make: *** [Makefile:146: all] Error 2
#17 46.36 ---
#17 46.36 Failed   <<< isaac_ros_nitros [28.3s, exited with code 2]
#17 46.41 
#17 46.41 Summary: 2 packages finished [35.7s]
#17 46.41   1 package failed: isaac_ros_nitros
#17 46.41   1 package had stderr output: isaac_ros_nitros
#17 46.41   3 packages not processed
------
executor failed running [/bin/bash -c apt-get update && cd ${ROS_ROOT}     && source ${ROS_ROOT}/setup.bash     && rosdep install -y -r --ignore-src --from-paths src --rosdistro ${ROS_DISTRO} --skip-keys="fastrtps"     && colcon build --merge-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo --packages-up-to isaac_ros_image_proc     && rm -Rf logs build     && rm -rf /var/lib/apt/lists/*     && apt-get clean]: exit code: 2
Failed to build base image: isaac_ros_dev-aarch64, aborting.

I saw that the Isaac ROS Dev container adds /usr/local/cuda-11.4/targets/aarch64-linux/lib to LD_LIBRARY_PATH, but it looks like the system doesn't see Nsight library directories.

Could you please help me to solve this issue? This has been my headache for a couple of weeks now. I will be very grateful to you.

jaiveersinghNV commented 1 year ago

Our container is built and run using the run_dev.sh script in Isaac ROS Common, which includes arguments to the docker run command here to mount the relevant header file into the container.

Since you're attempting to build the packages within an intermediate Dockerfile layer, the header file isn't available yet, and thus compilation fails.

One potential solution may be to explicitly copy over the relevant header files as part of your custom Dockerfile, while removing the corresponding bind mounts from run_dev.sh.

ikhann commented 1 year ago

Our container is built and run using the run_dev.sh script in Isaac ROS Common, which includes arguments to the docker run command here to mount the relevant header file into the container.

Since you're attempting to build the packages within an intermediate Dockerfile layer, the header file isn't available yet, and thus compilation fails.

One potential solution may be to explicitly copy over the relevant header files as part of your custom Dockerfile, while removing the corresponding bind mounts from run_dev.sh.

Thanks! It works. I added two lines in Dockerfile:

COPY ./rc_ws/libs/nvToolsExt.h /usr/local/cuda-11.4/targets/aarch64-linux/include/nvToolsExt.h
COPY ./rc_ws/libs/libnvToolsExt.so /usr/local/cuda-11.4/targets/aarch64-linux/lib/libnvToolsExt.so