search

NVIDIA-ISAAC-ROS / isaac_ros_argus_camera

ROS 2 packages based on NVIDIA libArgus library for NVIDIA-accelerated CSI camera support.
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
64 stars 12 forks source link

Dude, where's my CUDA? #5

Closed javadan closed 2 years ago

javadan commented 2 years ago

This is odd because this was working yesterday

I'm inside the isaac common docker, on Jetson NX with JetPack 4.6.1 and it's not finding cuda?

I've 'git lfs pull'ed the dirs.


admin@chicken:/workspaces/isaac_ros-dev$ cd src/
admin@chicken:/workspaces/isaac_ros-dev/src$ ls -l
total 8
drwxrwxr-x 6 admin admin 4096 Mar 12 18:12 isaac_ros_argus_camera
drwxrwxr-x 9 admin admin 4096 Mar 12 18:10 isaac_ros_common

admin@chicken:/workspaces/isaac_ros-dev$ colcon build --symlink-install
Starting >>> isaac_ros_test
Starting >>> isaac_ros_common
Starting >>> isaac_ros_nvengine_interfaces
Finished <<< isaac_ros_test [3.38s]                                                                                                   
Finished <<< isaac_ros_common [18.2s]                                                                                     
Starting >>> isaac_ros_argus_camera_mono
Starting >>> isaac_ros_argus_camera_stereo
Finished <<< isaac_ros_nvengine_interfaces [29.8s]                                                                                                                                   
Starting >>> isaac_ros_nvengine
--- stderr: isaac_ros_argus_camera_stereo                                                                                                                                   
/usr/bin/ld: warning: libcudart.so.10.2, needed by /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcufft.so.10, needed by /opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15, not found (try using -rpath or -rpath-link)
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyToArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftExecC2R@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamConsumerReleaseFrame@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamProducerReturnFrame@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftMakePlanMany@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaRegisterFunction@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaUnregisterFatBinary@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMallocManaged@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DToArrayAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaCreateTextureObject@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaDestroyTextureObject@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemset2D@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventCreateWithFlags@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftCreate@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftSetStream@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGraphicsUnregisterResource@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetMipmappedArrayLevel@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetErrorName@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetErrorString@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaCreateSurfaceObject@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaPopCallConfiguration@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaRegisterVar@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetDeviceProperties@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaRegisterFatBinary@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaArrayGetInfo@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyToArrayAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaFreeHost@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaFree@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamDestroy@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetDevice@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamWaitEvent@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventDestroy@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventSynchronize@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamProducerDisconnect@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemset@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamProducerPresentFrame@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftExecC2C@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventCreate@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyFromArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaDeviceGetAttribute@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetDeviceCount@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamAttachMemAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventQuery@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamCreate@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventElapsedTime@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEventRecord@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaFreeArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaDeviceSynchronize@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2D@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaDestroySurfaceObject@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetDeviceFlags@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGraphicsEGLRegisterImage@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaRegisterFatBinaryEnd@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamConsumerAcquireFrame@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaHostUnregister@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaLaunchKernel@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamConsumerConnect@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftGetSizeMany@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamConsumerDisconnect@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamAddCallback@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaHostGetDevicePointer@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMallocArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemset2DAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DFromArrayAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftSetAutoAllocation@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyArrayToArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DFromArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGraphicsResourceGetMappedEglFrame@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyFromArrayAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemsetAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaHostAlloc@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DArrayToArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyFromSymbol@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftDestroy@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaPointerGetAttributes@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMallocPitch@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMalloc@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaEGLStreamProducerConnect@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaStreamSynchronize@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaHostRegister@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftSetWorkArea@libcufft.so.10'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaCreateChannelDesc@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `__cudaPushCallConfiguration@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpyAsync@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaGetLastError@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cudaMemcpy2DToArray@libcudart.so.10.2'
/opt/nvidia/vpi1/lib64/libnvvpi.so.1.1.15: undefined reference to `cufftExecR2C@libcufft.so.10'
collect2: error: ld returned 1 exit status
make[2]: *** [isaac_ros_argus_camera_stereo] Error 1
make[1]: *** [CMakeFiles/isaac_ros_argus_camera_stereo.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< isaac_ros_argus_camera_stereo [39.0s, exited with code 2]
Aborted  <<< isaac_ros_argus_camera_mono [44.7s]                                                                                         
Aborted  <<< isaac_ros_nvengine [43.6s]                                       

Summary: 3 packages finished [1min 14s]
  1 package failed: isaac_ros_argus_camera_stereo
  2 packages aborted: isaac_ros_argus_camera_mono isaac_ros_nvengine
  2 packages had stderr output: isaac_ros_argus_camera_mono isaac_ros_argus_camera_stereo

( i tried sudo apt install nvidia-cuda but same error, afterwards. maybe LD_LIBRARY_PATH needs fixing? could someone suggest how I might fix it, if that is the issue?)

admin@chicken:/workspaces/isaac_ros-dev$ echo $LD_LIBRARY_PATH 
/opt/ros/foxy/install/opt/yaml_cpp_vendor/lib:/opt/ros/foxy/install/lib:/usr/lib/aarch64-linux-gnu/tegra-egl:/usr/local/cuda-10.2/targets/aarch64-linux/lib:/usr/lib/aarch64-linux-gnu/tegra:/opt/nvidia/vpi1/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib::/opt/tritonserver/lib

It doesn't make sense. i had the isaac ros argus camera working yesterday (with RPi 2.1 camera), and now CUDA disappeared today? That's enough internet for me today.

Thanks in advance.

javadan commented 2 years ago

Ok I managed to get it working again by adding the last line here, in scripts/run_dev.sh. But it doesn't seem like the proper solution. 

if [[ $PLATFORM == "aarch64" ]]; then
    DOCKER_ARGS+=("-v /usr/bin/tegrastats:/usr/bin/tegrastats")
    DOCKER_ARGS+=("-v /tmp/argus_socket:/tmp/argus_socket")
    DOCKER_ARGS+=("-v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra")
    DOCKER_ARGS+=("-v /usr/src/jetson_multimedia_api:/usr/src/jetson_multimedia_api")
    DOCKER_ARGS+=("-v /usr/local/cuda-10.2/targets/aarch64-linux/lib/:/usr/local/cuda-10.2/targets/aarch64-linux/lib/")
fi