search

OE4T / meta-tegra

BSP layer for NVIDIA Jetson platforms, based on L4T
MIT License
405 stars 223 forks source link

libnvdla_compiler.so: cannot open shared object file #1281

Closed dferrarigrowtech closed 1 year ago

dferrarigrowtech commented 1 year ago

I'm running demo-image-full image on jetson nano with yocto dunfell. I would like to run a nvidia container. When I import tensorrt through python in the container I get the following error:

ImportError: libnvdla_compiler.so: cannot open shared object file: No such file or directory

To reproduce

If I do the same thing with the jetson OS I don't get this messages. I think it could be related to the tegra-demo-distro or meta-tegra

Further details

Inside /etc/nvidia-container-runtime/host-files-for-container.d there is the tegra-libraries-core.csv that should map the /usr/lib/libnvdla_compiler.so file. But I'm not able to find that file inside the container

nvidia-container-toolkit.log ``` I0623 11:59:48.885184 5551 nvc.c:281] initializing library context (version=0.10.0+jetpack, build=61f57bcdf7aa6e73d9a348a7e36ec9fd94128fb2) I0623 11:59:48.885265 5551 nvc.c:255] using root / I0623 11:59:48.885281 5551 nvc.c:256] using ldcache /etc/ld.so.cache I0623 11:59:48.885293 5551 nvc.c:257] using unprivileged user 65534:65534 I0623 11:59:48.885691 5556 driver.c:134] starting driver service I0623 11:59:48.916517 5551 nvc_container.c:364] configuring container with 'compute utility jetpack-base-only supervised' I0623 11:59:48.917160 5551 nvc_container.c:384] setting pid to 5518 I0623 11:59:48.917180 5551 nvc_container.c:385] setting rootfs to /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged I0623 11:59:48.917193 5551 nvc_container.c:386] setting owner to 0:0 I0623 11:59:48.917205 5551 nvc_container.c:387] setting bins directory to /usr/bin I0623 11:59:48.917217 5551 nvc_container.c:388] setting libs directory to /usr/lib/aarch64-linux-gnu/ I0623 11:59:48.917228 5551 nvc_container.c:389] setting libs32 directory to /var/empty I0623 11:59:48.917239 5551 nvc_container.c:390] setting cudart directory to /usr/local/cuda I0623 11:59:48.917250 5551 nvc_container.c:391] setting ldconfig to @/sbin/ldconfig (host relative) I0623 11:59:48.917262 5551 nvc_container.c:392] setting mount namespace to /proc/5518/ns/mnt I0623 11:59:48.917273 5551 nvc_container.c:394] setting devices cgroup to /sys/fs/cgroup/devices/docker/976f84f621ff25ad0b2343a8b5b1e7be1384dce91ed0ff13abae12ea9835d8d4 I0623 11:59:48.917293 5551 nvc_info.c:728] requesting driver information with '' I0623 11:59:48.917964 5551 nvc_info.c:193] selecting /usr/lib/libnvapputil.so I0623 11:59:48.918034 5551 nvc_info.c:193] selecting /usr/lib/libnvargus.so I0623 11:59:48.918098 5551 nvc_info.c:193] selecting /usr/lib/libnvargus_socketclient.so I0623 11:59:48.918160 5551 nvc_info.c:193] selecting /usr/lib/libnvargus_socketserver.so I0623 11:59:48.918223 5551 nvc_info.c:193] selecting /usr/lib/libnvcam_imageencoder.so I0623 11:59:48.918284 5551 nvc_info.c:193] selecting /usr/lib/libnvcameratools.so I0623 11:59:48.918343 5551 nvc_info.c:193] selecting /usr/lib/libnvcamerautils.so I0623 11:59:48.918403 5551 nvc_info.c:193] selecting /usr/lib/libnvcamlog.so I0623 11:59:48.918463 5551 nvc_info.c:193] selecting /usr/lib/libnvcamv4l2.so I0623 11:59:48.918527 5551 nvc_info.c:193] selecting /usr/lib/libnveglstream_camconsumer.so I0623 11:59:48.918589 5551 nvc_info.c:193] selecting /usr/lib/libnveglstreamproducer.so I0623 11:59:48.918649 5551 nvc_info.c:193] selecting /usr/lib/libnvfnet.so I0623 11:59:48.918710 5551 nvc_info.c:193] selecting /usr/lib/libnvfnetstoredefog.so I0623 11:59:48.918771 5551 nvc_info.c:193] selecting /usr/lib/libnvfnetstorehdfx.so I0623 11:59:48.918832 5551 nvc_info.c:193] selecting /usr/lib/libnvodm_imager.so I0623 11:59:48.918892 5551 nvc_info.c:193] selecting /usr/lib/libnvscf.so I0623 11:59:48.918969 5551 nvc_info.c:193] selecting /usr/lib/libv4l/plugins/libv4l2_nvargus.so I0623 11:59:48.919084 5551 nvc_info.c:193] selecting /usr/lib/libGLX_nvidia.so.0 I0623 11:59:48.919152 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-glcore.so.32.7.3 I0623 11:59:48.919258 5551 nvc_info.c:193] selecting /lib/firmware/tegra21x_xusb_firmware I0623 11:59:48.919405 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvtee.so I0623 11:59:48.919521 5551 nvc_info.c:193] selecting /usr/lib/libnvinfer.so.8.2.1 I0623 11:59:48.919587 5551 nvc_info.c:193] selecting /usr/lib/libnvinfer_builder_resource.so.8.2.1 I0623 11:59:48.919651 5551 nvc_info.c:193] selecting /usr/lib/libnvparsers.so.8.2.1 W0623 11:59:48.919673 5551 nvc_info.c:470] missing symlink /usr/lib/libnvcaffe_parser.so W0623 11:59:48.919701 5551 nvc_info.c:470] missing symlink /usr/lib/libnvinfer.so W0623 11:59:48.919724 5551 nvc_info.c:470] missing symlink /usr/lib/libnvparsers.so I0623 11:59:48.919835 5551 nvc_info.c:193] selecting /usr/lib/libcuda.so.1.1 I0623 11:59:48.919900 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-fatbinaryloader.so.440.18 I0623 11:59:48.919976 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-ptxjitcompiler.so.440.18 I0623 11:59:48.920121 5551 nvc_info.c:193] selecting /usr/lib/libGLESv1_CM_nvidia.so.1 I0623 11:59:48.920184 5551 nvc_info.c:193] selecting /usr/lib/libGLESv2_nvidia.so.2 I0623 11:59:48.920376 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_boot.so I0623 11:59:48.920440 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_camera.so I0623 11:59:48.920499 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_force.so I0623 11:59:48.920559 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_generic.so I0623 11:59:48.920620 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_gpucompute.so I0623 11:59:48.920680 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_graphics.so I0623 11:59:48.920738 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_il.so I0623 11:59:48.920798 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_spincircle.so I0623 11:59:48.920857 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_tbc.so I0623 11:59:48.920916 5551 nvc_info.c:193] selecting /usr/lib/libnvgov_ui.so I0623 11:59:48.920974 5551 nvc_info.c:193] selecting /usr/lib/libnvphsd.so I0623 11:59:48.921092 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvdrmvideosink.so I0623 11:59:48.921218 5551 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json I0623 11:59:48.921286 5551 nvc_info.c:193] selecting /usr/lib/libnvvulkan-producer.so I0623 11:59:48.921523 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvvideosinks.so I0623 11:59:48.921753 5551 nvc_info.c:193] selecting /usr/lib/libv4l/plugins/libv4l2_nvvidconv_wrapper.so I0623 11:59:48.921953 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvvideo4linux2.so I0623 11:59:48.922152 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvvidconv.so I0623 11:59:48.922274 5551 nvc_info.c:193] selecting /usr/lib/libvisionworks.so.1.6.0 I0623 11:59:48.922401 5551 nvc_info.c:193] selecting /usr/lib/libnvbuf_fdmap.so.1.0.0 I0623 11:59:48.922465 5551 nvc_info.c:193] selecting /usr/lib/libnvbuf_utils.so.1.0.0 W0623 11:59:48.922487 5551 nvc_info.c:470] missing symlink /usr/lib/libnvbuf_utils.so I0623 11:59:48.922620 5551 nvc_info.c:193] selecting /usr/lib/libcudnn.so.8.2.1 I0623 11:59:48.922683 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_adv_infer.so.8.2.1 I0623 11:59:48.922749 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_adv_train.so.8.2.1 I0623 11:59:48.922810 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_cnn_infer.so.8.2.1 I0623 11:59:48.922871 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_cnn_train.so.8.2.1 I0623 11:59:48.922932 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_ops_infer.so.8.2.1 I0623 11:59:48.922996 5551 nvc_info.c:193] selecting /usr/lib/libcudnn_ops_train.so.8.2.1 W0623 11:59:48.923018 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn.so W0623 11:59:48.923041 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_adv_infer.so W0623 11:59:48.923064 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_adv_train.so W0623 11:59:48.923086 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_cnn_infer.so W0623 11:59:48.923108 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_cnn_train.so W0623 11:59:48.923129 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_ops_infer.so W0623 11:59:48.923151 5551 nvc_info.c:470] missing symlink /usr/lib/libcudnn_ops_train.so I0623 11:59:48.923371 5551 nvc_info.c:193] selecting /usr/lib/libdvbv5.so.0.0.0 I0623 11:59:48.923436 5551 nvc_info.c:193] selecting /usr/lib/libmediactl.so.0.0.0 I0623 11:59:48.923501 5551 nvc_info.c:193] selecting /usr/lib/libv4l/ov511-decomp I0623 11:59:48.923570 5551 nvc_info.c:193] selecting /usr/lib/libv4l/ov518-decomp I0623 11:59:48.923636 5551 nvc_info.c:193] selecting /usr/lib/libv4l/v4l1compat.so I0623 11:59:48.923702 5551 nvc_info.c:193] selecting /usr/lib/libv4l/v4l2convert.so I0623 11:59:48.923765 5551 nvc_info.c:193] selecting /usr/lib/libv4l1.so.0.0.0 I0623 11:59:48.923824 5551 nvc_info.c:193] selecting /usr/lib/libv4l2.so.0.0.0 I0623 11:59:48.923891 5551 nvc_info.c:193] selecting /usr/lib/libv4l2subdev.so.0.0.0 I0623 11:59:48.923953 5551 nvc_info.c:193] selecting /usr/lib/libv4lconvert.so.0.0.0 W0623 11:59:48.923974 5551 nvc_info.c:470] missing symlink /usr/lib/libdvbv5.so W0623 11:59:48.923995 5551 nvc_info.c:470] missing symlink /usr/lib/libmediactl.so W0623 11:59:48.924016 5551 nvc_info.c:470] missing symlink /usr/lib/libv4l1.so W0623 11:59:48.924037 5551 nvc_info.c:470] missing symlink /usr/lib/libv4l2.so W0623 11:59:48.924058 5551 nvc_info.c:470] missing symlink /usr/lib/libv4l2subdev.so W0623 11:59:48.924079 5551 nvc_info.c:470] missing symlink /usr/lib/libv4lconvert.so W0623 11:59:48.924100 5551 nvc_info.c:470] missing symlink /usr/lib/v4l1compat.so W0623 11:59:48.924116 5551 nvc_info.c:470] missing symlink /usr/lib/v4l2convert.so I0623 11:59:48.924241 5551 nvc_info.c:193] selecting /usr/lib/tegra/libdrm.so.2 I0623 11:59:48.924378 5551 nvc_info.c:193] selecting /usr/lib/libnvinfer_plugin.so.8.2.1 I0623 11:59:48.924445 5551 nvc_info.c:193] selecting /usr/lib/libnvonnxparser.so.8.2.1 W0623 11:59:48.924467 5551 nvc_info.c:470] missing symlink /usr/lib/libnvinfer_plugin.so W0623 11:59:48.924489 5551 nvc_info.c:470] missing symlink /usr/lib/libnvonnxparser.so I0623 11:59:48.924611 5551 nvc_info.c:193] selecting /etc/ld.so.conf.d/cuda-10-2.conf I0623 11:59:48.924769 5551 nvc_info.c:193] selecting /usr/lib/libtegrav4l2.so I0623 11:59:48.924849 5551 nvc_info.c:193] selecting /usr/lib/libv4l/plugins-wrapped/libv4l2_nvvidconv.so I0623 11:59:48.924926 5551 nvc_info.c:193] selecting /usr/lib/libv4l/plugins/libv4l2_nvvideocodec.so I0623 11:59:48.925135 5551 nvc_info.c:193] selecting /usr/lib/libnvomx.so I0623 11:59:48.925198 5551 nvc_info.c:193] selecting /usr/lib/libnvomxilclient.so I0623 11:59:48.925418 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvv4l2camerasrc.so I0623 11:59:48.925606 5551 nvc_info.c:193] selecting /usr/lib/libnvavp.so I0623 11:59:48.925671 5551 nvc_info.c:193] selecting /usr/lib/libnvbufsurface.so.1.0.0 I0623 11:59:48.925734 5551 nvc_info.c:193] selecting /usr/lib/libnvbufsurftransform.so.1.0.0 I0623 11:59:48.925798 5551 nvc_info.c:193] selecting /usr/lib/libnvdecode2eglimage.so I0623 11:59:48.925859 5551 nvc_info.c:193] selecting /usr/lib/libnvdsbufferpool.so.1.0.0 I0623 11:59:48.925919 5551 nvc_info.c:193] selecting /usr/lib/libnveventlib.so I0623 11:59:48.925980 5551 nvc_info.c:193] selecting /usr/lib/libnvexif.so I0623 11:59:48.926040 5551 nvc_info.c:193] selecting /usr/lib/libnvid_mapper.so.1.0.0 I0623 11:59:48.926100 5551 nvc_info.c:193] selecting /usr/lib/libnvjpeg.so I0623 11:59:48.926158 5551 nvc_info.c:193] selecting /usr/lib/libnvmedia.so I0623 11:59:48.926218 5551 nvc_info.c:193] selecting /usr/lib/libnvmm.so I0623 11:59:48.926279 5551 nvc_info.c:193] selecting /usr/lib/libnvmm_contentpipe.so I0623 11:59:48.926337 5551 nvc_info.c:193] selecting /usr/lib/libnvmm_parser.so I0623 11:59:48.926398 5551 nvc_info.c:193] selecting /usr/lib/libnvmm_utils.so I0623 11:59:48.926457 5551 nvc_info.c:193] selecting /usr/lib/libnvmmlite.so I0623 11:59:48.926518 5551 nvc_info.c:193] selecting /usr/lib/libnvmmlite_image.so I0623 11:59:48.926583 5551 nvc_info.c:193] selecting /usr/lib/libnvmmlite_utils.so I0623 11:59:48.926642 5551 nvc_info.c:193] selecting /usr/lib/libnvmmlite_video.so I0623 11:59:48.926702 5551 nvc_info.c:193] selecting /usr/lib/libnvofsdk.so I0623 11:59:48.926763 5551 nvc_info.c:193] selecting /usr/lib/libnvosd.so I0623 11:59:48.926821 5551 nvc_info.c:193] selecting /usr/lib/libnvparser.so I0623 11:59:48.926882 5551 nvc_info.c:193] selecting /usr/lib/libnvtestresults.so I0623 11:59:48.926941 5551 nvc_info.c:193] selecting /usr/lib/libnvtnr.so I0623 11:59:48.927002 5551 nvc_info.c:193] selecting /usr/lib/libnvtracebuf.so I0623 11:59:48.927061 5551 nvc_info.c:193] selecting /usr/lib/libnvtvmr.so W0623 11:59:48.927450 5551 nvc_info.c:498] missing device /dev/nvhost-nvdec1 W0623 11:59:48.927549 5551 nvc_info.c:498] missing device /dev/nvhost-ctrl-nvdla0 W0623 11:59:48.927594 5551 nvc_info.c:498] missing device /dev/nvhost-ctrl-nvdla1 W0623 11:59:48.927647 5551 nvc_info.c:498] missing device /dev/nvhost-nvdla0 W0623 11:59:48.927691 5551 nvc_info.c:498] missing device /dev/nvhost-nvdla1 W0623 11:59:48.927734 5551 nvc_info.c:498] missing device /dev/nvidiactl W0623 11:59:48.927902 5551 nvc_info.c:498] missing device /dev/nvhost-nvenc1 I0623 11:59:48.928169 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvjpeg.so I0623 11:59:48.928324 5551 nvc_info.c:193] selecting /usr/lib/libnvcolorutil.so I0623 11:59:48.928388 5551 nvc_info.c:193] selecting /usr/lib/libnvdc.so I0623 11:59:48.928448 5551 nvc_info.c:193] selecting /usr/lib/libnvddk_2d_v2.so I0623 11:59:48.928506 5551 nvc_info.c:193] selecting /usr/lib/libnvddk_vic.so I0623 11:59:48.928567 5551 nvc_info.c:193] selecting /usr/lib/libnvdla_compiler.so I0623 11:59:48.928629 5551 nvc_info.c:193] selecting /usr/lib/libnvdla_runtime.so I0623 11:59:48.928702 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-tls.so.32.7.3 I0623 11:59:48.928761 5551 nvc_info.c:193] selecting /usr/lib/libnvimp.so I0623 11:59:48.928821 5551 nvc_info.c:193] selecting /usr/lib/libnvll.so I0623 11:59:48.928881 5551 nvc_info.c:193] selecting /usr/lib/libnvos.so I0623 11:59:48.928938 5551 nvc_info.c:193] selecting /usr/lib/libnvphs.so I0623 11:59:48.928998 5551 nvc_info.c:193] selecting /usr/lib/libnvrm.so I0623 11:59:48.929057 5551 nvc_info.c:193] selecting /usr/lib/libnvrm_gpu.so I0623 11:59:48.929115 5551 nvc_info.c:193] selecting /usr/lib/libnvrm_graphics.so I0623 11:59:48.929178 5551 nvc_info.c:193] selecting /usr/lib/libsensors.hal-client.nvs.so I0623 11:59:48.929240 5551 nvc_info.c:193] selecting /usr/lib/libsensors.l4t.no_fusion.nvs.so I0623 11:59:48.929299 5551 nvc_info.c:193] selecting /usr/lib/libsensors_hal.nvs.so I0623 11:59:48.929457 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnveglglessink.so I0623 11:59:48.929549 5551 nvc_info.c:193] selecting /usr/lib/libgstegl-1.0.so.0.203.0 W0623 11:59:48.929575 5551 nvc_info.c:470] missing symlink /usr/lib/libgstegl-1.0.so I0623 11:59:48.929752 5551 nvc_info.c:193] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0 I0623 11:59:48.929826 5551 nvc_info.c:193] selecting /usr/lib/libEGL_nvidia.so.0 I0623 11:59:48.929888 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-eglcore.so.32.7.3 I0623 11:59:48.929950 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-glsi.so.32.7.3 I0623 11:59:48.930018 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-glvkspirv.so.32.7.3 I0623 11:59:48.930080 5551 nvc_info.c:193] selecting /usr/lib/libnvidia-rmapi-tegra.so.32.7.3 I0623 11:59:48.930158 5551 nvc_info.c:193] selecting /usr/share/glvnd/egl_vendor.d/10-nvidia.json I0623 11:59:48.930319 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so I0623 11:59:48.930482 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnveglstreamsrc.so I0623 11:59:48.930556 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvegltransform.so I0623 11:59:48.930625 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvivafilter.so I0623 11:59:48.930693 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvvideocuda.so I0623 11:59:48.930765 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstnvvideosink.so I0623 11:59:48.930825 5551 nvc_info.c:193] selecting /usr/lib/libgstnvegl-1.0.so.0 I0623 11:59:48.930885 5551 nvc_info.c:193] selecting /usr/lib/libgstnvexifmeta.so I0623 11:59:48.930946 5551 nvc_info.c:193] selecting /usr/lib/libgstnvivameta.so I0623 11:59:48.931007 5551 nvc_info.c:193] selecting /usr/lib/libnvsample_cudaprocess.so I0623 11:59:48.931164 5551 nvc_info.c:193] selecting /usr/lib/gstreamer-1.0/libgstomx.so I0623 11:59:48.931228 5551 nvc_info.c:785] requesting device information with opts: '' I0623 11:59:48.931634 5551 nvc_info.c:803] listing device (null) ((null) at (null)) I0623 11:59:48.931725 5551 nvc_mount.c:455] mount application profile I0623 11:59:48.931742 5551 nvc_mount.c:462] mount binaries I0623 11:59:48.931754 5551 nvc_mount.c:470] mount libraries I0623 11:59:48.931774 5551 nvc_mount.c:478] mount jetson libraries I0623 11:59:48.934158 5551 nvc_mount.c:486] mount jetson dirs I0623 11:59:48.934344 5551 nvc_mount.c:494] mount libraries32 I0623 11:59:48.934357 5551 nvc_mount.c:501] mount symlinks I0623 11:59:48.934369 5551 nvc_mount.c:505] create jetson symlinks I0623 11:59:48.934630 5551 nvc_mount.c:510] mount container library I0623 11:59:48.934642 5551 nvc_mount.c:527] mount IPCs I0623 11:59:48.934653 5551 nvc_mount.c:540] mount devices I0623 11:59:48.934782 5551 nvc_mount.c:112] mounting /dev/fb0 at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/fb0 I0623 11:59:48.934843 5551 nvc_mount.c:552] dev ptr: 0x559e80b9e0, tmp ptr: 0x559e7f44f0 I0623 11:59:48.934857 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.934908 5551 nvc_mount.c:351] whitelisting device node 29:0 I0623 11:59:48.934936 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.935055 5551 nvc_mount.c:112] mounting /dev/fb1 at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/fb1 I0623 11:59:48.935107 5551 nvc_mount.c:552] dev ptr: 0x559e80ba00, tmp ptr: 0x559e811680 I0623 11:59:48.935121 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.935158 5551 nvc_mount.c:351] whitelisting device node 29:1 I0623 11:59:48.935180 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.935297 5551 nvc_mount.c:112] mounting /dev/nvhost-as-gpu at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-as-gpu I0623 11:59:48.935344 5551 nvc_mount.c:552] dev ptr: 0x559e80ba20, tmp ptr: 0x559e8053c0 I0623 11:59:48.935358 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.935395 5551 nvc_mount.c:351] whitelisting device node 506:1 I0623 11:59:48.935417 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.935530 5551 nvc_mount.c:112] mounting /dev/nvhost-ctrl at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-ctrl I0623 11:59:48.935579 5551 nvc_mount.c:552] dev ptr: 0x559e80ba40, tmp ptr: 0x559e807540 I0623 11:59:48.935593 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.935629 5551 nvc_mount.c:351] whitelisting device node 242:0 I0623 11:59:48.935650 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.935764 5551 nvc_mount.c:112] mounting /dev/nvhost-ctrl-gpu at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-ctrl-gpu I0623 11:59:48.935810 5551 nvc_mount.c:552] dev ptr: 0x559e815940, tmp ptr: 0x559e803a30 I0623 11:59:48.935824 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.935859 5551 nvc_mount.c:351] whitelisting device node 506:2 I0623 11:59:48.935881 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.935993 5551 nvc_mount.c:112] mounting /dev/nvhost-dbg-gpu at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-dbg-gpu I0623 11:59:48.936043 5551 nvc_mount.c:552] dev ptr: 0x559e815960, tmp ptr: 0x559e80d630 I0623 11:59:48.936057 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.936094 5551 nvc_mount.c:351] whitelisting device node 506:3 I0623 11:59:48.936116 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.936227 5551 nvc_mount.c:112] mounting /dev/nvhost-gpu at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-gpu I0623 11:59:48.936274 5551 nvc_mount.c:552] dev ptr: 0x559e815980, tmp ptr: 0x559e80d850 I0623 11:59:48.936289 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.936324 5551 nvc_mount.c:351] whitelisting device node 506:0 I0623 11:59:48.936346 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.936460 5551 nvc_mount.c:112] mounting /dev/nvhost-nvdec at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-nvdec I0623 11:59:48.936510 5551 nvc_mount.c:552] dev ptr: 0x559e8159a0, tmp ptr: 0x559e811d70 I0623 11:59:48.936532 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.936568 5551 nvc_mount.c:351] whitelisting device node 242:9 I0623 11:59:48.936590 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.936703 5551 nvc_mount.c:112] mounting /dev/nvhost-prof-gpu at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-prof-gpu I0623 11:59:48.936753 5551 nvc_mount.c:552] dev ptr: 0x559e8159c0, tmp ptr: 0x559e811df0 I0623 11:59:48.936767 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.936803 5551 nvc_mount.c:351] whitelisting device node 506:4 I0623 11:59:48.936825 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.936939 5551 nvc_mount.c:112] mounting /dev/nvhost-vic at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-vic I0623 11:59:48.936989 5551 nvc_mount.c:552] dev ptr: 0x559e8159e0, tmp ptr: 0x559e8146f0 I0623 11:59:48.937003 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.937038 5551 nvc_mount.c:351] whitelisting device node 242:13 I0623 11:59:48.937059 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.937170 5551 nvc_mount.c:112] mounting /dev/nvmap at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvmap I0623 11:59:48.937219 5551 nvc_mount.c:552] dev ptr: 0x559e815a00, tmp ptr: 0x559e814770 I0623 11:59:48.937233 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.937268 5551 nvc_mount.c:351] whitelisting device node 10:61 I0623 11:59:48.937290 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.937399 5551 nvc_mount.c:112] mounting /dev/tegra_dc_0 at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/tegra_dc_0 I0623 11:59:48.937448 5551 nvc_mount.c:552] dev ptr: 0x559e815a20, tmp ptr: 0x559e816fc0 I0623 11:59:48.937463 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.937521 5551 nvc_mount.c:351] whitelisting device node 239:1 I0623 11:59:48.937545 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.937661 5551 nvc_mount.c:112] mounting /dev/tegra_dc_1 at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/tegra_dc_1 I0623 11:59:48.937713 5551 nvc_mount.c:552] dev ptr: 0x559e815a40, tmp ptr: 0x559e817040 I0623 11:59:48.937727 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.937762 5551 nvc_mount.c:351] whitelisting device node 239:2 I0623 11:59:48.937784 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.937897 5551 nvc_mount.c:112] mounting /dev/tegra_dc_ctrl at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/tegra_dc_ctrl I0623 11:59:48.937949 5551 nvc_mount.c:552] dev ptr: 0x559e815a60, tmp ptr: 0x559e80c8c0 I0623 11:59:48.937963 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.937999 5551 nvc_mount.c:351] whitelisting device node 239:0 I0623 11:59:48.938022 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.938134 5551 nvc_mount.c:112] mounting /dev/nvhost-msenc at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-msenc I0623 11:59:48.938185 5551 nvc_mount.c:552] dev ptr: 0x559e815a80, tmp ptr: 0x559e80c940 I0623 11:59:48.938199 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.938233 5551 nvc_mount.c:351] whitelisting device node 242:17 I0623 11:59:48.938255 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.938369 5551 nvc_mount.c:112] mounting /dev/nvhost-nvjpg at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged/dev/nvhost-nvjpg I0623 11:59:48.938418 5551 nvc_mount.c:552] dev ptr: 0x559e815aa0, tmp ptr: 0x559e814210 I0623 11:59:48.938432 5551 nvc_mount.c:554] setup cgroups I0623 11:59:48.938468 5551 nvc_mount.c:351] whitelisting device node 242:21 I0623 11:59:48.938490 5551 nvc_mount.c:358] device node whitelisted I0623 11:59:48.938506 5551 nvc_mount.c:569] returning to original mount_ns I0623 11:59:48.938532 5551 nvc_mount.c:571] returned there I0623 11:59:48.938563 5551 nvc_ldcache.c:344] executing /sbin/ldconfig from host at /var/lib/docker/overlay2/0ab1accc222eb0b878e608b97478a8ec369ce7ddc14561beb6d3eb8914c21ec4/merged I0623 11:59:49.094550 5551 nvc.c:314] shutting down library context I0623 11:59:49.094951 5556 driver.c:191] terminating driver service I0623 11:59:49.096964 5551 driver.c:231] driver service terminated successfully ```

The log shows that a lot of symlinks are missing. How can I fix this? Thank you!

dwalkes commented 1 year ago

Thanks @dferrarigrowtech

I can reproduce.

If I copy the content of tegra-libraries-core.csv into l4t.csv the libnvdla_compiler.so appears under /usr/lib/aarch64-linux-gnu/libnvdla_compiler.so. So for some reason it seems at least some the .csv files under /etc/nvidia-container-runtime/host-files-for-container.d are being ignored. May be related to https://github.com/OE4T/meta-tegra/pull/962#issuecomment-1100924427 but I thought this was specific to Jetpack 5. I haven't found the place where l4t.csv is referenced/hard-coded yet.

dwalkes commented 1 year ago

I haven't found the place where l4t.csv is referenced/hard-coded yet.

I think it's here

From docker inspect nvidia nvcr.io/nvidia/l4t-tensorrt:r8.2.1-runtime I see:

...
            "Env": [
...
                "NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=base-only",

And if I run:

docker run -it --rm --runtime nvidia -e NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS= nvcr.io/nvidia/l4t-tensorrt:r8.2.1-runtime

I'm able to run python3 and type import tensorrt without issue. So that's at least a workaround.

See related comment at https://github.com/NVIDIA/nvidia-container-toolkit/issues/274

I suspect the fix here is to do what we do with Jetpack 5 and try to put everything in l4t.csv. Curious if anyone else has thoughts about this.

dferrarigrowtech commented 1 year ago

Thanks for the quick reply and the workaround!

I just read this, according to nvidia:

for JetPack 4.x all of the .csv files should get used by --runtime nvidia (as long as the container is derived from l4t-base).

Source

madisongh commented 1 year ago

A partial backport of NVIDIA/nvidia-docker#1038 is what's needed here - just the part that consolidates all of the BSP-related passthrough mounts into l4t.csv, without removing the parts where we have separate CSV files for other pacakages. I wasn't aware that NVIDIA had published containers targeted at JetPack 4/L4T R32.x that used the new base-only passthrough mechanism, but support was added for them in the more recent releases. With JetPack 5/L4T R35.x, all containers NVIDIA publishes use the new mechanism.

ichergui commented 1 year ago

Hey @dferrarigrowtech Any update on this ?

dferrarigrowtech commented 1 year ago

Nope, I'm using the suggested workaround

ichergui commented 1 year ago

Can I close this ticket ?

dferrarigrowtech commented 1 year ago

The issue is not completely fixed, but at least there is a workaround available! You can close it

dferrarigrowtech commented 7 months ago

Sorry to bother you again but I just encountered a blocking issue with the aforementioned workaround.

Steps to reproduce:

The installation process fails. But if you run the same instructions without -e NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=, then pycuda will be installed without any issue. The problem is that I need to use both tensorrt (which works only with the env variable setted) and pycuda (which works only without the env variable)

Update I fixed by replacing the content of l4t.csv with:

l4t.csv ``` dev, /dev/fb0 dev, /dev/fb1 dev, /dev/nvhost-as-gpu dev, /dev/nvhost-ctrl dev, /dev/nvhost-ctrl-gpu dev, /dev/nvhost-dbg-gpu dev, /dev/nvhost-gpu dev, /dev/nvhost-nvdec dev, /dev/nvhost-nvdec1 dev, /dev/nvhost-prof-gpu dev, /dev/nvhost-vic dev, /dev/nvhost-ctrl-nvdla0 dev, /dev/nvhost-ctrl-nvdla1 dev, /dev/nvhost-nvdla0 dev, /dev/nvhost-nvdla1 dev, /dev/nvidiactl dev, /dev/nvmap dev, /dev/tegra_dc_0 dev, /dev/tegra_dc_1 dev, /dev/tegra_dc_ctrl dev, /dev/nvhost-msenc dev, /dev/nvhost-nvenc1 dev, /dev/nvhost-nvjpg dir, /lib/firmware/tegra21x sym, /usr/lib/libv4l2.so.0 lib, /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so lib, /usr/lib/gstreamer-1.0/libgstnveglglessink.so lib, /usr/lib/gstreamer-1.0/libgstnveglstreamsrc.so lib, /usr/lib/gstreamer-1.0/libgstnvegltransform.so lib, /usr/lib/gstreamer-1.0/libgstnvivafilter.so lib, /usr/lib/gstreamer-1.0/libgstnvv4l2camerasrc.so lib, /usr/lib/gstreamer-1.0/libgstnvvideo4linux2.so lib, /usr/lib/gstreamer-1.0/libgstnvvideocuda.so lib, /usr/lib/gstreamer-1.0/libgstnvvideosink.so lib, /usr/lib/gstreamer-1.0/libgstomx.so lib, /usr/lib/libgstnvegl-1.0.so.0 lib, /usr/lib/libgstnvexifmeta.so lib, /usr/lib/libgstnvivameta.so lib, /usr/lib/libnvsample_cudaprocess.so lib, /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0 lib, /usr/lib/libGLESv1_CM_nvidia.so.1 lib, /usr/lib/libGLESv2_nvidia.so.2 lib, /usr/lib/libcuda.so.1.1 lib, /usr/lib/libGLX_nvidia.so.0 lib, /usr/lib/libnvapputil.so lib, /usr/lib/libnvargus.so lib, /usr/lib/libnvargus_socketclient.so lib, /usr/lib/libnvargus_socketserver.so lib, /usr/lib/libnvavp.so lib, /usr/lib/libnvbuf_fdmap.so.1.0.0 lib, /usr/lib/libnvbufsurface.so.1.0.0 lib, /usr/lib/libnvbufsurftransform.so.1.0.0 lib, /usr/lib/libnvbuf_utils.so.1.0.0 lib, /usr/lib/libnvcameratools.so lib, /usr/lib/libnvcamerautils.so lib, /usr/lib/libnvcam_imageencoder.so lib, /usr/lib/libnvcamlog.so lib, /usr/lib/libnvcamv4l2.so lib, /usr/lib/libnvcolorutil.so lib, /usr/lib/libnvdc.so lib, /usr/lib/libnvddk_2d_v2.so lib, /usr/lib/libnvddk_vic.so lib, /usr/lib/libnvdecode2eglimage.so lib, /usr/lib/libnvdla_compiler.so lib, /usr/lib/libnvdla_runtime.so lib, /usr/lib/libnvdsbufferpool.so.1.0.0 lib, /usr/lib/libnveglstream_camconsumer.so lib, /usr/lib/libnveglstreamproducer.so lib, /usr/lib/libnveventlib.so lib, /usr/lib/libnvexif.so lib, /usr/lib/libnvfnet.so lib, /usr/lib/libnvfnetstoredefog.so lib, /usr/lib/libnvfnetstorehdfx.so lib, /usr/lib/libnvgov_boot.so lib, /usr/lib/libnvgov_camera.so lib, /usr/lib/libnvgov_force.so lib, /usr/lib/libnvgov_generic.so lib, /usr/lib/libnvgov_gpucompute.so lib, /usr/lib/libnvgov_graphics.so lib, /usr/lib/libnvgov_il.so lib, /usr/lib/libnvgov_spincircle.so lib, /usr/lib/libnvgov_tbc.so lib, /usr/lib/libnvgov_ui.so lib, /usr/lib/libnvidia-eglcore.so.32.7.4 lib, /usr/lib/libnvidia-fatbinaryloader.so.440.18 lib, /usr/lib/libnvidia-glcore.so.32.7.4 lib, /usr/lib/libnvidia-glsi.so.32.7.4 lib, /usr/lib/libnvidia-glvkspirv.so.32.7.4 lib, /usr/lib/libnvidia-ptxjitcompiler.so.440.18 lib, /usr/lib/libnvidia-rmapi-tegra.so.32.7.4 lib, /usr/lib/libnvidia-tls.so.32.7.4 lib, /usr/lib/libnvid_mapper.so.1.0.0 lib, /usr/lib/libnvimp.so lib, /usr/lib/libnvjpeg.so lib, /usr/lib/libnvll.so lib, /usr/lib/libnvmedia.so lib, /usr/lib/libnvmm_contentpipe.so lib, /usr/lib/libnvmmlite_image.so lib, /usr/lib/libnvmmlite.so lib, /usr/lib/libnvmmlite_utils.so lib, /usr/lib/libnvmmlite_video.so lib, /usr/lib/libnvmm_parser.so lib, /usr/lib/libnvmm.so lib, /usr/lib/libnvmm_utils.so lib, /usr/lib/libnvodm_imager.so lib, /usr/lib/libnvofsdk.so lib, /usr/lib/libnvomxilclient.so lib, /usr/lib/libnvomx.so lib, /usr/lib/libnvosd.so lib, /usr/lib/libnvos.so lib, /usr/lib/libnvparser.so lib, /usr/lib/libnvphsd.so lib, /usr/lib/libnvphs.so lib, /usr/lib/libnvrm_gpu.so lib, /usr/lib/libnvrm_graphics.so lib, /usr/lib/libnvrm.so lib, /usr/lib/libnvscf.so lib, /usr/lib/libnvtestresults.so lib, /usr/lib/libnvtnr.so lib, /usr/lib/libnvtracebuf.so lib, /usr/lib/libnvtvmr.so lib, /usr/lib/libsensors.hal-client.nvs.so lib, /usr/lib/libsensors_hal.nvs.so lib, /usr/lib/libsensors.l4t.no_fusion.nvs.so lib, /usr/lib/libtegrav4l2.so lib, /usr/lib/libv4l/plugins/libv4l2_nvargus.so lib, /usr/lib/libv4l/plugins-wrapped/libv4l2_nvvidconv.so lib, /usr/lib/libv4l/plugins/libv4l2_nvvideocodec.so sym, /usr/lib/libcuda.so lib, /usr/lib/libv4l/plugins/libv4l2_nvargus.so lib, /usr/lib/libv4l/plugins-wrapped/libv4l2_nvvidconv.so lib, /usr/lib/libv4l/plugins/libv4l2_nvvideocodec.so sym, /usr/lib/libcuda.so sym, /usr/lib/libnvbufsurface.so sym, /usr/lib/libnvbufsurftransform.so sym, /usr/lib/libnvbuf_utils.so sym, /usr/lib/libnvdsbufferpool.so sym, /usr/lib/libnvid_mapper.so ```