opencv cannot use cuda

[JTShuai]

The build report from opencv shows it's built with cuda, but the cv2.cuda.getCudaEnabledDeviceCount() command gets an error.

Environment

• Device: Jetson TX2
• JetPack: 4.6 [L4T 32.6.1]
• Image: dustynv/ros:foxy-desktop-l4t-r35.4.1

Problem reproduce

• build container command:

  docker run --privileged -w /root --net=host --runtime nvidia --name jt_foxy --ipc=host -v /tmp/.X11-unix/:/tmp/.X11-unix/ -v /tmp/argus_socket:/tmp/argus_socket --cap-add SYS_PTRACE -e DISPLAY=$DISPLAY -it -v /dev:/dev --device-cgroup-rule='c *:* rmw'  -v /home/tx2/jt_ws:/host_data dustynv/ros:foxy-desktop-l4t-r35.4.1

• output from cv2.getBuildInformation():

  General configuration for OpenCV 4.5.0 =====================================
  Version control:               4.5.0
  
  Extra modules:
    Location (extra):            /opt/opencv_contrib/modules
    Version control (extra):     4.5.0
  
  Platform:
    Timestamp:                   2021-10-23T16:52:07Z
    Host:                        Linux 5.10.59-tegra aarch64
    CMake:                       3.16.3
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE
  
  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3
  
  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 9.3.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:
  
  OpenCV modules:
    To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cvv hdf java js julia matlab ovis python2 sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         YES
  
  GUI:
    GTK+:                        YES (ver 3.24.20)
      GThread :                  YES (ver 2.64.6)
      GtkGlExt:                  NO
    OpenGL support:              NO
    VTK support:                 NO
  
  Media I/O:
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.0.10)
    JPEG 2000:                   build (ver 2.3.1)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES
  
  Video I/O:
    DC1394:                      YES (2.2.5)
    FFMPEG:                      YES
      avcodec:                   YES (58.54.100)
      avformat:                  YES (58.29.100)
      avutil:                    YES (56.31.100)
      swscale:                   YES (5.5.100)
      avresample:                YES (4.0.0)
    GStreamer:                   YES (1.16.2)
    v4l/v4l2:                    YES (linux/videodev2.h)
  
  Parallel framework:            TBB (ver 2020.1 interface 11101)
  
  Trace:                         YES (with Intel ITT)
  
  Other third-party libraries:
    Lapack:                      YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
    Eigen:                       YES (ver 3.3.7)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)
  
  NVIDIA CUDA:                   YES (ver 11.4, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             72 87
    NVIDIA PTX archs:
  
  cuDNN:                         YES (ver 8.2.6)
  
  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.8.10)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.8.so (ver 3.8.10)
    numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.17.4)
    install path:                lib/python3.8/dist-packages/cv2/python-3.8
  
  Python (for build):            /usr/bin/python2.7
  
  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO
  
  Install to:                    /usr/local
  -----------------------------------------------------------------

• Error from cv2.cuda.getCudaEnabledDeviceCount():

  cv2.error: OpenCV(4.5.0) /opt/opencv/modules/core/src/cuda_info.cpp:62: error: (-217:Gpu API call) initialization error in function 'getCudaEnabledDeviceCount'

[dusty-nv]

@JTShuai can you confirm you can use CUDA in another independent container like l4t-jetpack or l4t-pytorch, and without all the extra docker run flags you added like --privileged / ect

[JTShuai]

@JTShuai can you confirm you can use CUDA in another independent container like l4t-jetpack or l4t-pytorch, and without all the extra docker run flags you added like --privileged / ect

Hi, I tried docker run --runtime nvidia -it --rm --network=host dustynv/l4t-pytorch:r35.4.1 and got the error:

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.

[JTShuai]

@dusty-nv I just noticed the comments:

Container images are compatible with other minor versions of JetPack/L4T:
    • L4T R32.7 containers can run on other versions of L4T R32.7 (JetPack 4.6+)
    • L4T R35.x containers can run on other versions of L4T R35.x (JetPack 5.1+)

So I tired docker run --runtime nvidia -it --rm --network=host dustynv/l4t-pytorch:r32.7.1, but got the same error.

[dusty-nv]

Hi @JTShuai - had you recently done an apt upgrade? With that adding seccomp filter rule for syscall error, it sounds like the same problem to this one:

https://forums.developer.nvidia.com/t/docker-containers-wont-run-after-recent-apt-get-upgrade/194369

[JTShuai]

Hi @JTShuai - had you recently done an apt upgrade? With that adding seccomp filter rule for syscall error, it sounds like the same problem to this one:

https://forums.developer.nvidia.com/t/docker-containers-wont-run-after-recent-apt-get-upgrade/194369

Thanks for your help! I tried the following commands you wrote in #108

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install nvidia-docker2=2.8.0-1

Now, I can enter container with the command docker run --runtime nvidia -it --rm --network=host dustynv/l4t-pytorch:r32.7.1, but I got a new error with pytorch:

root@tx2-4:/# python3
Python 3.6.9 (default, Mar 10 2023, 16:46:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 196, in <module>
    _load_global_deps()
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 149, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcurand.so.10: cannot open shared object file: No such file or directory
>>> 

[dusty-nv]

@JTShuai on jetpack 4, CUDA/cuDNN/TensorRT are mounted into the containers from the host device when --runtime nvidia is used. You should have that libcurand.so.10 under /usr/local/cuda/lib64. If you keep having problems with this, I might recommend reflashing your SD card given all the issues you have with docker. Then try again after a fresh re-install without doing the apt upgrade.

[JTShuai]

@JTShuai on jetpack 4, CUDA/cuDNN/TensorRT are mounted into the containers from the host device when --runtime nvidia is used. You should have that libcurand.so.10 under /usr/local/cuda/lib64. If you keep having problems with this, I might recommend reflashing your SD card given all the issues you have with docker. Then try again after a fresh re-install without doing the apt upgrade.

Hi, I manually downgraded the docker to docker.io=20.10.7-0ubuntu1~18.04.2 and containerd=1.5.2-0ubuntu1~18.04.3. And I checked that the libcurand.so.10 is under /usr/local/cuda/lib64.

Still having the same error, so I will try reflashing the SD card.

[JTShuai]

Problem solved after reflashing the TX2.