search

NVIDIA / nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs
Apache License 2.0
2.37k stars 254 forks source link

Expected host libraries not properly mapped into Docker container on Jetson Xavier R32 4.4 #135

Open eugeneswalker opened 1 year ago

eugeneswalker commented 1 year ago

1. Issue or feature description

On Jetson Xavier running R32 4.4, docker run --runtime nvidia ... is not properly mapping in expected libraries from host:

For example, on host:

$host> ldconfig -p | grep cudnn
...
    libcudnn.so.8 (libc6,AArch64) => /usr/lib/aarch64-linux-gnu/libcudnn.so.8
...

$> file $(realpath  /usr/lib/aarch64-linux-gnu/libcudnn.so.8)
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fc314899bf3fa5241319ff2c2d3886094e216bbb, stripped

$> du -sh $(realpath  /usr/lib/aarch64-linux-gnu/libcudnn.so.8)
180K    /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0

And then trying to get this in a Docker container:

$> docker run -it --rm --runtime=nvidia --gpus all ecpe4s/waggle-ml-aarch64:2022-10-31

root@c78e24183e02:/# env | grep NVIDIA
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=all

root@c78e24183e02:/# ldconfig -p | grep cudnn
root@c78e24183e02:/#

root@c78e24183e02:/# ls -l /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0
-rw-r--r-- 1 root root 0 Oct 19  2020 /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0

root@c78e24183e02:/# du -sh /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0
0   /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0

2. Steps to reproduce the issue

I am not sure how this could be reproduced. This was working at one point on our Xavier but is no longer working since some packages have been upgraded.

3. Information

I uncommented debug = in /etc/nvidia-container-runtime/config.toml, restarted docker, ran the docker command mentioned above but nothing was written to /var/log/nvidia-container-toolkit.log

Docker version 20.10.24, build 297e128

$> nvidia-container-cli -V
cli-version: 1.13.5
lib-version: 0.11.0+jetpack
build date: 2023-07-18T11:40+00:00
build revision: 66607bd046341f7aad7de80a9f022f122d1f2fce
build compiler: aarch64-linux-gnu-gcc-7 7.5.0
build platform: aarch64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
$> uname -a
Linux xavier2 4.9.140-tegra NVIDIA/nvidia-docker#1 SMP PREEMPT Tue Oct 27 21:02:46 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
Output of nvidia-container-cli -k -d /dev/tty info ``` -- WARNING, the following logs are for debugging purposes only -- I0814 14:27:07.635315 513982 nvc.c:281] initializing library context (version=0.11.0+jetpack, build=1b60893021cd00c87f201d11eb207215afa3ab11) I0814 14:27:07.635547 513982 nvc.c:255] using root / I0814 14:27:07.635584 513982 nvc.c:256] using ldcache /etc/ld.so.cache I0814 14:27:07.635600 513982 nvc.c:257] using unprivileged user 65534:65534 I0814 14:27:07.636222 513983 driver.c:134] starting driver service I0814 14:27:07.678466 513982 nvc_info.c:728] requesting driver information with '' I0814 14:27:07.679902 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4l2.so I0814 14:27:07.680093 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/desktop-shell.so I0814 14:27:07.680244 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/drm-backend.so I0814 14:27:07.680381 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/EGLWLInputEventExample I0814 14:27:07.680509 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/EGLWLMockNavigation I0814 14:27:07.680627 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/fullscreen-shell.so I0814 14:27:07.680751 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/gl-renderer.so I0814 14:27:07.680872 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/hmi-controller.so I0814 14:27:07.680988 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/ivi-controller.so I0814 14:27:07.681103 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/ivi-shell.so I0814 14:27:07.681270 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/LayerManagerControl I0814 14:27:07.681401 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmClient.so.2.2.0 I0814 14:27:07.681510 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmCommon.so.2.2.0 I0814 14:27:07.681667 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmControl.so.2.2.0 I0814 14:27:07.681781 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmInput.so.2.2.0 I0814 14:27:07.681936 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libweston-6.so.0 I0814 14:27:07.682064 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libweston-desktop-6.so.0 I0814 14:27:07.682178 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/simple-weston-client I0814 14:27:07.682285 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/spring-tool I0814 14:27:07.682408 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/wayland-backend.so I0814 14:27:07.682540 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston I0814 14:27:07.682655 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-calibrator I0814 14:27:07.682772 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-clickdot I0814 14:27:07.682940 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-cliptest I0814 14:27:07.683062 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-content-protection I0814 14:27:07.683176 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-debug I0814 14:27:07.683295 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-desktop-shell I0814 14:27:07.683428 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-dnd I0814 14:27:07.683554 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-eventdemo I0814 14:27:07.683701 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-flower I0814 14:27:07.683823 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-fullscreen I0814 14:27:07.683971 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-image I0814 14:27:07.684086 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-info I0814 14:27:07.684202 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-keyboard I0814 14:27:07.684347 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-launch I0814 14:27:07.684471 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-multi-resource I0814 14:27:07.684583 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-resizor I0814 14:27:07.684700 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-scaler I0814 14:27:07.684832 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-screenshooter I0814 14:27:07.684945 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-dmabuf-egldevice I0814 14:27:07.685074 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-egl I0814 14:27:07.685194 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-shm I0814 14:27:07.685311 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-touch I0814 14:27:07.685431 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-smoke I0814 14:27:07.685581 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-stacking I0814 14:27:07.685702 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-subsurfaces I0814 14:27:07.685826 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-terminal I0814 14:27:07.685947 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-transformed I0814 14:27:07.686065 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so I0814 14:27:07.686220 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcompositor.so I0814 14:27:07.686334 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvdrmvideosink.so I0814 14:27:07.686479 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so I0814 14:27:07.686609 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so I0814 14:27:07.686718 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so I0814 14:27:07.686893 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so I0814 14:27:07.687008 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvjpeg.so I0814 14:27:07.687122 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvtee.so I0814 14:27:07.687226 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so I0814 14:27:07.687328 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so I0814 14:27:07.687443 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so I0814 14:27:07.687577 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideocuda.so I0814 14:27:07.687706 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosink.so I0814 14:27:07.687814 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosinks.so I0814 14:27:07.687921 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so I0814 14:27:07.688040 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libgstnvegl-1.0.so.0 I0814 14:27:07.688142 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libgstnvexifmeta.so I0814 14:27:07.688238 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libgstnvivameta.so I0814 14:27:07.688330 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvsample_cudaprocess.so I0814 14:27:07.688471 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/ld.so.conf I0814 14:27:07.688580 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0 I0814 14:27:07.688715 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv1_CM_nvidia.so.1 I0814 14:27:07.688832 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv2_nvidia.so.2 I0814 14:27:07.688941 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/nvidia.json I0814 14:27:07.689086 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1 I0814 14:27:07.689209 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libdrm.so.2 I0814 14:27:07.689355 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libGLX_nvidia.so.0 I0814 14:27:07.689479 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so I0814 14:27:07.689586 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so I0814 14:27:07.689753 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so I0814 14:27:07.689875 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so I0814 14:27:07.689993 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvavp.so I0814 14:27:07.690107 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 I0814 14:27:07.690228 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0 I0814 14:27:07.690352 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so.1.0.0 I0814 14:27:07.690473 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 I0814 14:27:07.690585 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so I0814 14:27:07.690701 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so I0814 14:27:07.690870 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so I0814 14:27:07.690983 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so I0814 14:27:07.691093 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamv4l2.so I0814 14:27:07.691203 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcapture.so I0814 14:27:07.691316 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so I0814 14:27:07.691426 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so I0814 14:27:07.691533 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so I0814 14:27:07.691641 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so I0814 14:27:07.691787 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so I0814 14:27:07.691921 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so I0814 14:27:07.692083 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so.1.0.0 I0814 14:27:07.692191 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so I0814 14:27:07.692302 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so I0814 14:27:07.692438 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveventlib.so I0814 14:27:07.692548 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvexif.so I0814 14:27:07.692671 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so I0814 14:27:07.692804 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so I0814 14:27:07.692937 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so I0814 14:27:07.693061 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgbm.so I0814 14:27:07.693166 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_boot.so I0814 14:27:07.693272 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_camera.so I0814 14:27:07.693375 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_force.so I0814 14:27:07.693484 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_generic.so I0814 14:27:07.693586 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_gpucompute.so I0814 14:27:07.693686 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_graphics.so I0814 14:27:07.693788 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_il.so I0814 14:27:07.693919 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_spincircle.so I0814 14:27:07.694032 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_tbc.so I0814 14:27:07.694134 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_ui.so I0814 14:27:07.694241 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.32.4.4 I0814 14:27:07.694364 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so I0814 14:27:07.694508 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.4.4 I0814 14:27:07.694636 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glcore.so.32.4.4 I0814 14:27:07.694755 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glsi.so.32.4.4 I0814 14:27:07.694932 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glvkspirv.so.32.4.4 I0814 14:27:07.695097 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-ptxjitcompiler.so.32.4.4 I0814 14:27:07.695257 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-rmapi-tegra.so.32.4.4 I0814 14:27:07.695400 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-tls.so.32.4.4 I0814 14:27:07.695514 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so.1.0.0 I0814 14:27:07.695618 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so I0814 14:27:07.695720 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvisp_utils.so I0814 14:27:07.695825 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so I0814 14:27:07.695938 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvll.so I0814 14:27:07.696069 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so I0814 14:27:07.696192 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so I0814 14:27:07.696300 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so I0814 14:27:07.696409 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so I0814 14:27:07.696516 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so I0814 14:27:07.696621 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so I0814 14:27:07.696757 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so I0814 14:27:07.696880 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm.so I0814 14:27:07.696985 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so I0814 14:27:07.697099 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so I0814 14:27:07.697263 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvofsdk.so I0814 14:27:07.697384 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so I0814 14:27:07.697527 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvomx.so I0814 14:27:07.697649 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvosd.so I0814 14:27:07.697758 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvos.so I0814 14:27:07.697894 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so I0814 14:27:07.698008 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvphsd.so I0814 14:27:07.698122 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvphs.so I0814 14:27:07.698258 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvpva.so I0814 14:27:07.698378 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so I0814 14:27:07.698501 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so I0814 14:27:07.698635 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so I0814 14:27:07.698773 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so I0814 14:27:07.698926 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so I0814 14:27:07.699032 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so I0814 14:27:07.699135 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtracebuf.so I0814 14:27:07.699248 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so I0814 14:27:07.699368 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4l2.so I0814 14:27:07.699484 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4lconvert.so I0814 14:27:07.699593 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvvulkan-producer.so I0814 14:27:07.699705 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so I0814 14:27:07.699817 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors.hal-client.nvs.so I0814 14:27:07.699936 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors_hal.nvs.so I0814 14:27:07.700097 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors.l4t.no_fusion.nvs.so I0814 14:27:07.700218 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so I0814 14:27:07.700351 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvvidconv.so I0814 14:27:07.700504 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvvideocodec.so I0814 14:27:07.700618 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/libvulkan.so.1.2.132 I0814 14:27:07.700771 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json I0814 14:27:07.700879 513982 nvc_info.c:193] selecting /lib/firmware/tegra18x/nvhost_nvdec030_ns.fw I0814 14:27:07.700988 513982 nvc_info.c:193] selecting /lib/firmware/tegra19x/nvhost_nvdec040_ns.fw W0814 14:27:07.701189 513982 nvc_info.c:442] missing directory /lib/firmware/tegra21x W0814 14:27:07.701631 513982 nvc_info.c:498] missing device /dev/nvidiactl I0814 14:27:07.702194 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvinfer.so.7.1.3 I0814 14:27:07.702331 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3 I0814 14:27:07.702460 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvparsers.so.7.1.3 I0814 14:27:07.702593 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.7.1.3 I0814 14:27:07.702718 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libmyelin.so.1.1.0 I0814 14:27:07.702899 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInfer.h I0814 14:27:07.703025 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInferRuntime.h I0814 14:27:07.703147 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h I0814 14:27:07.703291 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInferVersion.h I0814 14:27:07.703414 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvUtils.h I0814 14:27:07.703515 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInferPlugin.h I0814 14:27:07.703644 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvInferPluginUtils.h I0814 14:27:07.703736 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvCaffeParser.h I0814 14:27:07.703834 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvUffParser.h I0814 14:27:07.703963 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvOnnxConfig.h I0814 14:27:07.704068 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/NvOnnxParser.h W0814 14:27:07.704303 513982 nvc_info.c:442] missing directory /usr/lib/python3.6/dist-packages/tensorrt I0814 14:27:07.704995 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0 I0814 14:27:07.705112 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.0.0 I0814 14:27:07.705312 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.0.0 I0814 14:27:07.705438 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.0.0 I0814 14:27:07.705575 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.0.0 I0814 14:27:07.705741 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.0.0 I0814 14:27:07.705870 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.0.0 I0814 14:27:07.705988 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_adv_infer_v8.h I0814 14:27:07.706089 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_adv_train_v8.h I0814 14:27:07.706188 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_backend_v8.h I0814 14:27:07.706287 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_cnn_infer_v8.h I0814 14:27:07.706384 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_cnn_train_v8.h I0814 14:27:07.706486 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_ops_infer_v8.h I0814 14:27:07.706612 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_ops_train_v8.h I0814 14:27:07.706712 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_v8.h I0814 14:27:07.706805 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_version_v8.h I0814 14:27:07.706949 513982 nvc_info.c:193] selecting /usr/include/aarch64-linux-gnu/cudnn_v8.h I0814 14:27:07.707067 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.0.0 I0814 14:27:07.707192 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer_static_v8.a I0814 14:27:07.707315 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.0.0 I0814 14:27:07.707463 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_adv_train_static_v8.a I0814 14:27:07.707637 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.0.0 I0814 14:27:07.707773 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer_static_v8.a I0814 14:27:07.707972 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.0.0 I0814 14:27:07.708101 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train_static_v8.a I0814 14:27:07.708276 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.0.0 I0814 14:27:07.708414 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer_static_v8.a I0814 14:27:07.708575 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.0.0 I0814 14:27:07.708714 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_ops_train_static_v8.a I0814 14:27:07.708854 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0 I0814 14:27:07.709048 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcudnn_static_v8.a I0814 14:27:07.709543 513982 nvc_info.c:414] missing library /usr/lib/libvisionworks_sfm.so.0.90.4 I0814 14:27:07.709633 513982 nvc_info.c:193] selecting /usr/lib/libvisionworks.so.1.6.0 I0814 14:27:07.709725 513982 nvc_info.c:414] missing library /usr/lib/libvisionworks_tracking.so.0.88.2 W0814 14:27:07.709767 513982 nvc_info.c:470] missing symlink /usr/lib/libvisionworks_sfm.so W0814 14:27:07.709795 513982 nvc_info.c:470] missing symlink /usr/lib/libvisionworks_sfm.so.0.90 W0814 14:27:07.709824 513982 nvc_info.c:470] missing symlink /usr/lib/libvisionworks.so W0814 14:27:07.709851 513982 nvc_info.c:470] missing symlink /usr/lib/libvisionworks_tracking.so W0814 14:27:07.709877 513982 nvc_info.c:470] missing symlink /usr/lib/libvisionworks_tracking.so.0.88 I0814 14:27:07.710133 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcublas.so.10.2.2.89 I0814 14:27:07.710249 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libcublasLt.so.10.2.2.89 I0814 14:27:07.710377 513982 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/libnvblas.so.10.2.2.89 I0814 14:27:07.710505 513982 nvc_info.c:193] selecting /usr/include/cublas_api.h I0814 14:27:07.710605 513982 nvc_info.c:193] selecting /usr/include/cublasLt.h I0814 14:27:07.710710 513982 nvc_info.c:193] selecting /usr/include/cublasXt.h I0814 14:27:07.711124 513982 nvc_info.c:193] selecting /usr/include/cublas.h I0814 14:27:07.711228 513982 nvc_info.c:193] selecting /usr/include/cublas_v2.h W0814 14:27:07.711330 513982 nvc_info.c:470] missing symlink /usr/loca/cuda I0814 14:27:07.711571 513982 nvc_info.c:785] requesting device information with opts: '' I0814 14:27:07.712233 513982 nvc_info.c:803] listing device (null) ((null) at (null)) NVRM version: (null) CUDA version: 10.2 Device Index: 0 Device Minor: 0 Model: Xavier Brand: (null) GPU UUID: (null) Bus Location: (null) Architecture: 7.2 I0814 14:27:07.712429 513982 nvc.c:314] shutting down library context I0814 14:27:07.712525 513983 driver.c:191] terminating driver service I0814 14:27:07.713937 513982 driver.c:231] driver service terminated successfully ```
Output of dpkg -l '*nvidia*' ``` Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-===================================-============-===================================================== un libgldispatch0-nvidia (no description available) ii libnvidia-container-tools 1.13.5-1 arm64 NVIDIA container runtime library (command-line tools) ii libnvidia-container0:arm64 0.11.0+jetpack arm64 NVIDIA container runtime library ii libnvidia-container1:arm64 1.13.5-1 arm64 NVIDIA container runtime library un nvidia-common (no description available) rc nvidia-container-csv-cuda 10.2.89-1 arm64 Jetpack CUDA CSV file rc nvidia-container-csv-cudnn 8.0.0.180-1+cuda10.2 arm64 Jetpack CUDNN CSV file rc nvidia-container-csv-tensorrt 7.1.3.0-1+cuda10.2 arm64 Jetpack TensorRT CSV file rc nvidia-container-csv-visionworks 1.6.0.501 arm64 Jetpack VisionWorks CSV file un nvidia-container-runtime (no description available) un nvidia-container-runtime-hook (no description available) ii nvidia-container-toolkit 1.13.5-1 arm64 NVIDIA Container toolkit ii nvidia-container-toolkit-base 1.13.5-1 arm64 NVIDIA Container Toolkit Base un nvidia-docker (no description available) ii nvidia-docker2 2.13.0-1 all nvidia-docker CLI wrapper ii nvidia-l4t-3d-core 32.4.4-20201027211332 arm64 NVIDIA GL EGL Package ii nvidia-l4t-apt-source 32.4.4-20201016123640 arm64 NVIDIA L4T apt source list debian package ii nvidia-l4t-bootloader 32.4.4-20201027211332 arm64 NVIDIA Bootloader Package ii nvidia-l4t-camera 32.4.4-20201027211332 arm64 NVIDIA Camera Package un nvidia-l4t-ccp-t186ref (no description available) ii nvidia-l4t-configs 32.4.4-20201016123640 arm64 NVIDIA configs debian package ii nvidia-l4t-core 32.4.4-20201016123640 arm64 NVIDIA Core Package ii nvidia-l4t-cuda 32.4.4-20201027211332 arm64 NVIDIA CUDA Package ii nvidia-l4t-firmware 32.4.4-20201027211332 arm64 NVIDIA Firmware Package ii nvidia-l4t-graphics-demos 32.4.4-20201027211332 arm64 NVIDIA graphics demo applications ii nvidia-l4t-gstreamer 32.4.4-20201016123640 arm64 NVIDIA GST Application files ii nvidia-l4t-init 32.4.4-20201027211332 arm64 NVIDIA Init debian package ii nvidia-l4t-initrd 32.4.4-20201016123640 arm64 NVIDIA initrd debian package ii nvidia-l4t-jetson-io 32.4.4-20201016123640 arm64 NVIDIA Jetson.IO debian package ii nvidia-l4t-kernel 4.9.140-tegra-32.4.4-20201027211332 arm64 NVIDIA Kernel Package ii nvidia-l4t-kernel-dtbs 4.9.140-tegra-32.4.4-20201027211332 arm64 NVIDIA Kernel DTB Package ii nvidia-l4t-kernel-headers 4.9.140-tegra-32.4.4-20201027211332 arm64 NVIDIA Linux Tegra Kernel Headers Package ii nvidia-l4t-multimedia 32.4.4-20201027211332 arm64 NVIDIA Multimedia Package ii nvidia-l4t-multimedia-utils 32.4.4-20201027211332 arm64 NVIDIA Multimedia Package ii nvidia-l4t-oem-config 32.4.4-20201016123640 arm64 NVIDIA OEM-Config Package ii nvidia-l4t-tools 32.4.4-20201027211332 arm64 NVIDIA Public Test Tools Package ii nvidia-l4t-wayland 32.4.4-20201027211332 arm64 NVIDIA Wayland Package ii nvidia-l4t-weston 32.4.4-20201027211332 arm64 NVIDIA Weston Package ii nvidia-l4t-x11 32.4.4-20201027211332 arm64 NVIDIA X11 Package ii nvidia-l4t-xusb-firmware 32.4.4-20201016123640 arm64 NVIDIA USB Firmware Package un nvidia-libopencl1-dev (no description available) un nvidia-prime (no description available) ```
elezar commented 11 months ago

@eugeneswalker the files you mention are not specified in the files:

meaning that these are considered runtime files and should be included in the containers that require them.

With regards to why the libraries are not in the ldcache, this may be a bug in updating the LDCache when the container is started, may aslo be due to a misconfiguration of the contianer itself. Are the libraries in the ldcache when the container is started without the NVIDIA Container Runtime.

Finally. With regards to:

I uncommented debug = in /etc/nvidia-container-runtime/config.toml, restarted docker, ran the docker command mentioned above but nothing was written to /var/log/nvidia-container-toolkit.log

The NVIDIA Container Toolkit no longer use the NVIDIA Container CLI on Tegra-based systems (and has not since v1.10.0). This means that the log of interest is the /var/log/nvidia-container-runtime.log and this should show information related to the detected files and devices.