search

NVIDIA-ISAAC-ROS / isaac_ros_image_pipeline

NVIDIA-accelerated ROS2 packages for camera image processing.
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
112 stars 19 forks source link

libcudart.so.10.2 & libcufft.so.10 not found, undefined reference to cuda dynamic libs #18

Closed dirksavage88 closed 1 year ago

dirksavage88 commented 1 year ago

I am on Jetpack 4.6.1, Jetson Nano, and using the issacros* release-ea2 branches for isaac_ros_common, isaac_ros_image_pipeline, & isaac_ros_apriltag. I have run "git lfs pull" in each of the repo folders.

Container successfully builds. I see libcudart and libcufft in /usr/local/cuda-10.2/lib64.

When I run colcon build within my container workspace, I get this error when building isaac_ros_image_proc:

**/usr/bin/ld: warning: libcudart.so.10.2, needed by /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: libcufft.so.10, needed by /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15, not found (try using -rpath or -rpath-link) /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpyToArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftExecC2R@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEGLStreamConsumerReleaseFrame@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEGLStreamProducerReturnFrame@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cufftMakePlanMany@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaRegisterFunction@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaUnregisterFatBinary@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy2DAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMallocManaged@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy2DToArrayAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaCreateTextureObject@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaDestroyTextureObject@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemset2D@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEventCreateWithFlags@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftCreate@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cufftSetStream@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGraphicsUnregisterResource@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaGetMipmappedArrayLevel@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGetErrorName@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaGetErrorString@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaCreateSurfaceObject@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to __cudaPopCallConfiguration@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaRegisterVar@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaGetDeviceProperties@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaRegisterFatBinary@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaArrayGetInfo@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemcpyToArrayAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaFreeHost@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaFree@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaStreamDestroy@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGetDevice@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaStreamWaitEvent@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEventDestroy@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEventSynchronize@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEGLStreamProducerDisconnect@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemset@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEGLStreamProducerPresentFrame@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cufftExecC2C@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEventCreate@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpyFromArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaDeviceGetAttribute@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaGetDeviceCount@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaStreamAttachMemAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEventQuery@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaStreamCreate@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEventElapsedTime@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEventRecord@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaFreeArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaDeviceSynchronize@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy2D@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaDestroySurfaceObject@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaGetDeviceFlags@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGraphicsEGLRegisterImage@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to __cudaRegisterFatBinaryEnd@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaEGLStreamConsumerAcquireFrame@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaHostUnregister@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaLaunchKernel@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEGLStreamConsumerConnect@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftGetSizeMany@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEGLStreamConsumerDisconnect@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaStreamAddCallback@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaHostGetDevicePointer@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMallocArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemset2DAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemcpy2DFromArrayAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cufftSetAutoAllocation@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemcpyArrayToArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy2DFromArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGraphicsResourceGetMappedEglFrame@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpyFromArrayAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemsetAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaHostAlloc@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMemcpy2DArrayToArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpyFromSymbol@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftDestroy@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaPointerGetAttributes@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMallocPitch@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaMalloc@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaEGLStreamProducerConnect@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaStreamSynchronize@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaHostRegister@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftSetWorkArea@libcufft.so.10' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaCreateChannelDesc@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to__cudaPushCallConfiguration@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpyAsync@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocudaGetLastError@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to cudaMemcpy2DToArray@libcudart.so.10.2' /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference tocufftExecR2C@libcufft.so.10' collect2: error: ld returned 1 exit status make[2]: [isaac_ros_image_proc] Error 1 make[1]: [CMakeFiles/isaac_ros_image_proc.dir/all] Error 2 make: *** [all] Error 2

Failed <<< isaac_ros_image_proc [2.82s, exited with code 2] Aborted <<< isaac_ros_stereo_image_proc [4.24s]**

Looks like vpi is not able to link to these cuda libs? isaac_ros_common builds correctly

hemalshahNV commented 1 year ago

Could you check that the CUDA 10.2 libs are available where LD_LIBRARY_PATH: Dockerfile.aarch64.base for EA2 

ENV LD_LIBRARY_PATH="/usr/local/cuda-10.2/targets/aarch64-linux/lib:${LD_LIBRARY_PATH}"

Could you try with EA3 instead which was the last ROS2 Foxy on JP 4.6.1 Isaac ROS release? We are not able to support any previous Early Access releases.

dirksavage88 commented 1 year ago

@hemalshahNV will EA3 work with the L4T r32.6.1 base image? I see that EA2 is the latest branch with that base image called out in the Docker.aarch64.base

EDIT: I see that we can use release ea-3 as the L4T r32.7.1 is in Jetpack 4.6.1. You can close this issue if you need to, I’ll move to release ea-3