search

OE4T / meta-tegra

BSP layer for NVIDIA Jetson platforms, based on L4T
MIT License
419 stars 230 forks source link

docker: Error response from daemon: Unknown runtime specified nvidia. #654

Closed RomainGsd closed 3 years ago

RomainGsd commented 3 years ago

Hello, I am trying to run NVIDIA's l4t containers. I can pull an image and run it with docker run --gpus all --rm -it nvcr.io/nvidia/l4t-base:r32.5.0 bash

But if I add --runtime nvidia to docker options it result in this error : docker: Error response from daemon: Unknown runtime specified nvidia.

I am running on a Jetson Xavier NX devkit on dunfell branch and I added these layers to my local.conf : CORE_IMAGE_EXTRA_INSTALL_append = " nvidia-docker cuda-libraries cuda-samples tensorrt cudnn libvisionworks" CORE_IMAGE_EXTRA_INSTALL_append = " nvidia-container-runtime cudnn-container-csv tensorrt-container-csv libvisionworks-container-csv" DISTRO_FEATURES_append = " ldconfig virtualization"

First thought is that I missed something while local.conf configuration but I can't find anything related in READMEs, closed/open issues or the wiki

madisongh commented 3 years ago

Which dunfell branch? The dunfell (with no version suffix) branch doesn't support Xavier NX, so it must be one of the others. I ask because the L4T version of your built OS has to match the L4T version of the container you're trying to run. So is it dunfell-l4t-r32.5.0 ?

Any other layers in your build besides the ones mentioned on the wiki page?

The additions you made look OK (although the -container-csv packages should get pulled in automatically without explicitly listing them). Is the docker daemon starting OK? Can you provide its log output?

RomainGsd commented 3 years ago

I'm on dunfell-l4t-32.4.3, therefore I just pulled nvcr.io/nvidia/l4t-tensorflow:r32.4.3-tf2.2-py3 accordingly.

I also have PREFERRED_VERSION_python3-docker = "4.3.1" CORE_IMAGE_EXTRA_INSTALL_append = " python3-docker"

dockerd log:

INFO[2021-04-09T10:29:59.951181728Z] Starting up                                  
DEBU[2021-04-09T10:29:59.952753056Z] Listener created for HTTP on unix (/var/run/docker.sock) 
DEBU[2021-04-09T10:29:59.952992800Z] Containerd not running, starting daemon managed containerd 
INFO[2021-04-09T10:29:59.954354496Z] libcontainerd: started new containerd process  pid=6377
INFO[2021-04-09T10:29:59.954667168Z] parsed scheme: "unix"                         module=grpc
INFO[2021-04-09T10:29:59.954865216Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-04-09T10:29:59.955090592Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2021-04-09T10:29:59.955384768Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-04-09T10:30:00.005220352Z] starting containerd                           revision=fd103cb716352c7e19768e4fed057f71d68902a0.m version=v1.2.7-11-gfd103cb7.m
DEBU[2021-04-09T10:30:00.005641600Z] changing OOM score to -500                   
INFO[2021-04-09T10:30:00.006737568Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2021-04-09T10:30:00.007002112Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
WARN[2021-04-09T10:30:00.009392032Z] failed to load plugin io.containerd.snapshotter.v1.aufs  error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.9.140-sdtv-l4t-r32."
INFO[2021-04-09T10:30:00.017083328Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2021-04-09T10:30:00.030440896Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2021-04-09T10:30:00.043668512Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
WARN[2021-04-09T10:30:00.057027424Z] failed to load plugin io.containerd.snapshotter.v1.zfs  error="path /mnt/extended/docker_data_root/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesyste"
INFO[2021-04-09T10:30:00.079750400Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2021-04-09T10:30:00.091961248Z] could not use snapshotter zfs in metadata plugin  error="path /mnt/extended/docker_data_root/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to b"
WARN[2021-04-09T10:30:00.114886976Z] could not use snapshotter aufs in metadata plugin  error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.9.140-sdtv-l4t-r32.4+ga58"
INFO[2021-04-09T10:30:00.137559168Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2021-04-09T10:30:00.149783936Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2021-04-09T10:30:00.161370400Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.174949824Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.187742176Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.200834208Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.213822496Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.226496672Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.240003136Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2021-04-09T10:30:00.253207904Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2021-04-09T10:30:00.265513824Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2021-04-09T10:30:00.277448096Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2021-04-09T10:30:00.291233216Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
DEBU[2021-04-09T10:30:00.296967552Z] garbage collected                             d=35.355072ms
INFO[2021-04-09T10:30:00.302124544Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2021-04-09T10:30:00.325124480Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.336597984Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.348613344Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.360167744Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.371700448Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.384346048Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.396059040Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.407841344Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.419704800Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
WARN[2021-04-09T10:30:00.432094816Z] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt: read-only file system"
INFO[2021-04-09T10:30:00.445517664Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.457517600Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.469149536Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.480899392Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2021-04-09T10:30:00.493748160Z] serving...                                    address="/var/run/docker/containerd/containerd-debug.sock"
INFO[2021-04-09T10:30:00.507535552Z] serving...                                    address="/var/run/docker/containerd/containerd.sock"
INFO[2021-04-09T10:30:00.520802560Z] containerd successfully booted in 0.517335s  
DEBU[2021-04-09T10:30:00.537341248Z] Started daemon managed containerd            
DEBU[2021-04-09T10:30:00.538749280Z] Golang's threads limit set to 50490          
INFO[2021-04-09T10:30:00.546557952Z] parsed scheme: "unix"                         module=grpc
INFO[2021-04-09T10:30:00.555933152Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-04-09T10:30:00.567284192Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2021-04-09T10:30:00.583005952Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-04-09T10:30:00.595317440Z] parsed scheme: "unix"                         module=grpc
INFO[2021-04-09T10:30:00.603536384Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-04-09T10:30:00.614557664Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2021-04-09T10:30:00.630363616Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2021-04-09T10:30:00.642748960Z] Using default logging driver json-file       
DEBU[2021-04-09T10:30:00.648920960Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs] 
DEBU[2021-04-09T10:30:00.645295264Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2021-04-09T10:30:00.691771232Z] backingFs=<unknown>, projectQuotaSupported=false, indexOff=""  storage-driver=overlay2
INFO[2021-04-09T10:30:00.692056704Z] [graphdriver] using prior storage driver: overlay2 
DEBU[2021-04-09T10:30:00.694781376Z] Initialized graph driver overlay2            
WARN[2021-04-09T10:30:00.714578016Z] Your kernel does not support cgroup rt period 
WARN[2021-04-09T10:30:00.714760032Z] Your kernel does not support cgroup rt runtime 
WARN[2021-04-09T10:30:00.719313664Z] Your kernel does not support cgroup blkio weight 
WARN[2021-04-09T10:30:00.728019328Z] Your kernel does not support cgroup blkio weight_device 
DEBU[2021-04-09T10:30:00.737938208Z] Max Concurrent Downloads: 3                  
DEBU[2021-04-09T10:30:00.745596800Z] Max Concurrent Uploads: 5                    
INFO[2021-04-09T10:30:00.754239168Z] Loading containers: start.                   
DEBU[2021-04-09T10:30:00.754402464Z] processing event stream                       module=libcontainerd namespace=moby
DEBU[2021-04-09T10:30:00.762669888Z] Option Experimental: false                   
DEBU[2021-04-09T10:30:00.783704800Z] Option DefaultDriver: bridge                 
DEBU[2021-04-09T10:30:00.791850656Z] Option DefaultNetwork: bridge                
DEBU[2021-04-09T10:30:00.800167040Z] Network Control Plane MTU: 1500              
DEBU[2021-04-09T10:30:00.820665056Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION] 
DEBU[2021-04-09T10:30:00.823008960Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:00.832846880Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[2021-04-09T10:30:00.846693280Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:00.858020800Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING] 
DEBU[2021-04-09T10:30:00.866253120Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT] 
DEBU[2021-04-09T10:30:00.874817504Z] /usr/sbin/iptables, [--wait -t nat -F DOCKER] 
DEBU[2021-04-09T10:30:00.882575232Z] /usr/sbin/iptables, [--wait -t nat -X DOCKER] 
DEBU[2021-04-09T10:30:00.891081824Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER] 
DEBU[2021-04-09T10:30:00.899815456Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER] 
DEBU[2021-04-09T10:30:00.908273312Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:00.918536384Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:00.928482080Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:00.938801568Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:00.949044352Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION] 
DEBU[2021-04-09T10:30:00.957991232Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION] 
DEBU[2021-04-09T10:30:00.967154752Z] /usr/sbin/iptables, [--wait -t nat -n -L DOCKER] 
DEBU[2021-04-09T10:30:00.975965376Z] /usr/sbin/iptables, [--wait -t nat -N DOCKER] 
DEBU[2021-04-09T10:30:00.984330784Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER] 
DEBU[2021-04-09T10:30:00.993031744Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.003908288Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.014012992Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.024276544Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN] 
DEBU[2021-04-09T10:30:01.035338112Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN] 
DEBU[2021-04-09T10:30:01.045628032Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN] 
DEBU[2021-04-09T10:30:01.056502112Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN] 
DEBU[2021-04-09T10:30:01.072249760Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE] 
DEBU[2021-04-09T10:30:01.079290656Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.089198112Z] /usr/sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.099591040Z] /usr/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.109716128Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.121712384Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.133425216Z] /usr/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:01.146050464Z] /usr/sbin/iptables, [--wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:01.157629152Z] /usr/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8] 
DEBU[2021-04-09T10:30:01.171503904Z] /usr/sbin/iptables, [--wait -t nat -A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8] 
DEBU[2021-04-09T10:30:01.184746944Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.194961824Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.205165728Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.220070560Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.234459872Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.244797408Z] /usr/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.255248384Z] /usr/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.265691392Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.280247584Z] /usr/sbin/iptables, [--wait -t filter -I DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.295031072Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.306285568Z] /usr/sbin/iptables, [--wait -t filter -I DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.318159872Z] Network (f5248d1) restored                   
DEBU[2021-04-09T10:30:01.327159808Z] Allocating IPv4 pools for network bridge (f5248d1d9a794b4a8ae58fe0a42e68efe1fc4ea0ab52b2da860631ee74aabe68) 
DEBU[2021-04-09T10:30:01.338364928Z] RequestPool(LocalDefault, 172.17.0.0/16, , map[], false) 
DEBU[2021-04-09T10:30:01.347673024Z] RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway]) 
DEBU[2021-04-09T10:30:01.361494976Z] Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2 
DEBU[2021-04-09T10:30:01.401151648Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE] 
DEBU[2021-04-09T10:30:01.406121888Z] /usr/sbin/iptables, [--wait -t nat -D POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE] 
DEBU[2021-04-09T10:30:01.414092288Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.423358816Z] /usr/sbin/iptables, [--wait -t nat -D DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.433381824Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.445293440Z] /usr/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.455717408Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.467568416Z] /usr/sbin/iptables, [--wait -D FORWARD -i docker0 ! -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.478744960Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.489193920Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.499673408Z] /usr/sbin/iptables, [--wait -D FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.509266496Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.523614784Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.537806304Z] /usr/sbin/iptables, [--wait -D FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.551404896Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.565331648Z] /usr/sbin/iptables, [--wait -t filter -D DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.580318912Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.591811776Z] /usr/sbin/iptables, [--wait -t filter -D DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.617116512Z] releasing IPv4 pools from network bridge (f5248d1d9a794b4a8ae58fe0a42e68efe1fc4ea0ab52b2da860631ee74aabe68) 
DEBU[2021-04-09T10:30:01.617403040Z] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.1) 
DEBU[2021-04-09T10:30:01.624621568Z] Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.1 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected 
DEBU[2021-04-09T10:30:01.650327328Z] ReleasePool(LocalDefault/172.17.0.0/16)      
DEBU[2021-04-09T10:30:01.669487776Z] cleanupServiceDiscovery for network:f5248d1d9a794b4a8ae58fe0a42e68efe1fc4ea0ab52b2da860631ee74aabe68 
INFO[2021-04-09T10:30:01.688454528Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
DEBU[2021-04-09T10:30:01.688786880Z] Allocating IPv4 pools for network bridge (0587b7960775244c46f98216cad1ee59fda85dab8d37280e0b0e4d444d4e3a92) 
DEBU[2021-04-09T10:30:01.701520704Z] RequestPool(LocalDefault, 172.17.0.0/16, , map[], false) 
DEBU[2021-04-09T10:30:01.710831104Z] RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway]) 
DEBU[2021-04-09T10:30:01.724536640Z] Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2 
DEBU[2021-04-09T10:30:01.749619744Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE] 
DEBU[2021-04-09T10:30:01.764208192Z] /usr/sbin/iptables, [--wait -t nat -I POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE] 
DEBU[2021-04-09T10:30:01.776504704Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.786088160Z] /usr/sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN] 
DEBU[2021-04-09T10:30:01.796269408Z] /usr/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:01.806848320Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.818683648Z] /usr/sbin/iptables, [--wait -I FORWARD -i docker0 -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.829005344Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.841054816Z] /usr/sbin/iptables, [--wait -I FORWARD -i docker0 ! -o docker0 -j ACCEPT] 
DEBU[2021-04-09T10:30:01.851995264Z] /usr/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:01.863578144Z] /usr/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2021-04-09T10:30:01.875581440Z] /usr/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8] 
DEBU[2021-04-09T10:30:01.889255200Z] /usr/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8] 
DEBU[2021-04-09T10:30:01.902720640Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.912978624Z] /usr/sbin/iptables, [--wait -I FORWARD -o docker0 -j DOCKER] 
DEBU[2021-04-09T10:30:01.922671808Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.937447136Z] /usr/sbin/iptables, [--wait -I FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT] 
DEBU[2021-04-09T10:30:01.951292992Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.961633024Z] /usr/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.971473440Z] /usr/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION-STAGE-1] 
DEBU[2021-04-09T10:30:01.982038208Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:01.996363008Z] /usr/sbin/iptables, [--wait -t filter -I DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2] 
DEBU[2021-04-09T10:30:02.011389696Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:02.022870592Z] /usr/sbin/iptables, [--wait -t filter -I DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP] 
DEBU[2021-04-09T10:30:02.061259072Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-USER] 
DEBU[2021-04-09T10:30:02.063552064Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-USER -j RETURN] 
DEBU[2021-04-09T10:30:02.065882208Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-USER] 
DEBU[2021-04-09T10:30:02.068140224Z] /usr/sbin/iptables, [--wait -D FORWARD -j DOCKER-USER] 
DEBU[2021-04-09T10:30:02.073265248Z] /usr/sbin/iptables, [--wait -I FORWARD -j DOCKER-USER] 
INFO[2021-04-09T10:30:02.082359456Z] Loading containers: done.                    
INFO[2021-04-09T10:30:02.116551168Z] Docker daemon                                 commit=afacb8b7f0d8d4f9d2a8e8736e9c993e672b41f3 graphdriver(s)=overlay2 version=19.03.8-ce
INFO[2021-04-09T10:30:02.117076256Z] Daemon has completed initialization          
DEBU[2021-04-09T10:30:02.138646112Z] Registering routers                          
DEBU[2021-04-09T10:30:02.138830432Z] Registering GET, /containers/{name:.*}/checkpoints 
DEBU[2021-04-09T10:30:02.139272832Z] Registering POST, /containers/{name:.*}/checkpoints 
DEBU[2021-04-09T10:30:02.141956576Z] Registering DELETE, /containers/{name}/checkpoints/{checkpoint} 
DEBU[2021-04-09T10:30:02.151903680Z] Registering HEAD, /containers/{name:.*}/archive 
DEBU[2021-04-09T10:30:02.160362016Z] Registering GET, /containers/json            
DEBU[2021-04-09T10:30:02.168650080Z] Registering GET, /containers/{name:.*}/export 
DEBU[2021-04-09T10:30:02.177007808Z] Registering GET, /containers/{name:.*}/changes 
DEBU[2021-04-09T10:30:02.185266528Z] Registering GET, /containers/{name:.*}/json  
DEBU[2021-04-09T10:30:02.193456864Z] Registering GET, /containers/{name:.*}/top   
DEBU[2021-04-09T10:30:02.201664192Z] Registering GET, /containers/{name:.*}/logs  
DEBU[2021-04-09T10:30:02.210010912Z] Registering GET, /containers/{name:.*}/stats 
DEBU[2021-04-09T10:30:02.218056096Z] Registering GET, /containers/{name:.*}/attach/ws 
DEBU[2021-04-09T10:30:02.226807200Z] Registering GET, /exec/{id:.*}/json          
DEBU[2021-04-09T10:30:02.235056256Z] Registering GET, /containers/{name:.*}/archive 
DEBU[2021-04-09T10:30:02.243342080Z] Registering POST, /containers/create         
DEBU[2021-04-09T10:30:02.251690272Z] Registering POST, /containers/{name:.*}/kill 
DEBU[2021-04-09T10:30:02.260026592Z] Registering POST, /containers/{name:.*}/pause 
DEBU[2021-04-09T10:30:02.267972672Z] Registering POST, /containers/{name:.*}/unpause 
DEBU[2021-04-09T10:30:02.276676704Z] Registering POST, /containers/{name:.*}/restart 
DEBU[2021-04-09T10:30:02.284930080Z] Registering POST, /containers/{name:.*}/start 
DEBU[2021-04-09T10:30:02.293455552Z] Registering POST, /containers/{name:.*}/stop 
DEBU[2021-04-09T10:30:02.301677856Z] Registering POST, /containers/{name:.*}/wait 
DEBU[2021-04-09T10:30:02.310158592Z] Registering POST, /containers/{name:.*}/resize 
DEBU[2021-04-09T10:30:02.318082720Z] Registering POST, /containers/{name:.*}/attach 
DEBU[2021-04-09T10:30:02.326621792Z] Registering POST, /containers/{name:.*}/copy 
DEBU[2021-04-09T10:30:02.334592768Z] Registering POST, /containers/{name:.*}/exec 
DEBU[2021-04-09T10:30:02.343131136Z] Registering POST, /exec/{name:.*}/start      
DEBU[2021-04-09T10:30:02.351281696Z] Registering POST, /exec/{name:.*}/resize     
DEBU[2021-04-09T10:30:02.359681984Z] Registering POST, /containers/{name:.*}/rename 
DEBU[2021-04-09T10:30:02.367867904Z] Registering POST, /containers/{name:.*}/update 
DEBU[2021-04-09T10:30:02.376109152Z] Registering POST, /containers/prune          
DEBU[2021-04-09T10:30:02.384174848Z] Registering POST, /commit                    
DEBU[2021-04-09T10:30:02.392835840Z] Registering PUT, /containers/{name:.*}/archive 
DEBU[2021-04-09T10:30:02.401146944Z] Registering DELETE, /containers/{name:.*}    
DEBU[2021-04-09T10:30:02.409358048Z] Registering GET, /images/json                
DEBU[2021-04-09T10:30:02.417483232Z] Registering GET, /images/search              
DEBU[2021-04-09T10:30:02.425797408Z] Registering GET, /images/get                 
DEBU[2021-04-09T10:30:02.434112672Z] Registering GET, /images/{name:.*}/get       
DEBU[2021-04-09T10:30:02.442325344Z] Registering GET, /images/{name:.*}/history   
DEBU[2021-04-09T10:30:02.450613984Z] Registering GET, /images/{name:.*}/json      
DEBU[2021-04-09T10:30:02.458753440Z] Registering POST, /images/load               
DEBU[2021-04-09T10:30:02.467404384Z] Registering POST, /images/create             
DEBU[2021-04-09T10:30:02.475369312Z] Registering POST, /images/{name:.*}/push     
DEBU[2021-04-09T10:30:02.483910848Z] Registering POST, /images/{name:.*}/tag      
DEBU[2021-04-09T10:30:02.491822240Z] Registering POST, /images/prune              
DEBU[2021-04-09T10:30:02.500039072Z] Registering DELETE, /images/{name:.*}        
DEBU[2021-04-09T10:30:02.508353024Z] Registering OPTIONS, /{anyroute:.*}          
DEBU[2021-04-09T10:30:02.516742528Z] Registering GET, /_ping                      
DEBU[2021-04-09T10:30:02.524942400Z] Registering HEAD, /_ping                     
DEBU[2021-04-09T10:30:02.532908096Z] Registering GET, /events                     
DEBU[2021-04-09T10:30:02.541512640Z] Registering GET, /info                       
DEBU[2021-04-09T10:30:02.549468000Z] Registering GET, /version                    
DEBU[2021-04-09T10:30:02.557909120Z] Registering GET, /system/df                  
DEBU[2021-04-09T10:30:02.566100960Z] Registering POST, /auth                      
DEBU[2021-04-09T10:30:02.574269184Z] Registering GET, /volumes                    
DEBU[2021-04-09T10:30:02.582507424Z] Registering GET, /volumes/{name:.*}          
DEBU[2021-04-09T10:30:02.590824352Z] Registering POST, /volumes/create            
DEBU[2021-04-09T10:30:02.598736480Z] Registering POST, /volumes/prune             
DEBU[2021-04-09T10:30:02.607201472Z] Registering DELETE, /volumes/{name:.*}       
DEBU[2021-04-09T10:30:02.615272384Z] Registering POST, /build                     
DEBU[2021-04-09T10:30:02.623661632Z] Registering POST, /build/prune               
DEBU[2021-04-09T10:30:02.631697248Z] Registering POST, /build/cancel              
DEBU[2021-04-09T10:30:02.640106272Z] Registering POST, /session                   
DEBU[2021-04-09T10:30:02.648346720Z] Registering POST, /swarm/init                
DEBU[2021-04-09T10:30:02.656591072Z] Registering POST, /swarm/join                
DEBU[2021-04-09T10:30:02.664806496Z] Registering POST, /swarm/leave               
DEBU[2021-04-09T10:30:02.673002144Z] Registering GET, /swarm                      
DEBU[2021-04-09T10:30:02.681281696Z] Registering GET, /swarm/unlockkey            
DEBU[2021-04-09T10:30:02.689216832Z] Registering POST, /swarm/update              
DEBU[2021-04-09T10:30:02.697472640Z] Registering POST, /swarm/unlock              
DEBU[2021-04-09T10:30:02.705924224Z] Registering GET, /services                   
DEBU[2021-04-09T10:30:02.713872224Z] Registering GET, /services/{id}              
DEBU[2021-04-09T10:30:02.722381248Z] Registering POST, /services/create           
DEBU[2021-04-09T10:30:02.730672928Z] Registering POST, /services/{id}/update      
DEBU[2021-04-09T10:30:02.738872288Z] Registering DELETE, /services/{id}           
DEBU[2021-04-09T10:30:02.747047296Z] Registering GET, /services/{id}/logs         
DEBU[2021-04-09T10:30:02.755055584Z] Registering GET, /nodes                      
DEBU[2021-04-09T10:30:02.763498080Z] Registering GET, /nodes/{id}                 
DEBU[2021-04-09T10:30:02.771491904Z] Registering DELETE, /nodes/{id}              
DEBU[2021-04-09T10:30:02.779729696Z] Registering POST, /nodes/{id}/update         
DEBU[2021-04-09T10:30:02.788291168Z] Registering GET, /tasks                      
DEBU[2021-04-09T10:30:02.796352736Z] Registering GET, /tasks/{id}                 
DEBU[2021-04-09T10:30:02.804690656Z] Registering GET, /tasks/{id}/logs            
DEBU[2021-04-09T10:30:02.812611488Z] Registering GET, /secrets                    
DEBU[2021-04-09T10:30:02.821072096Z] Registering POST, /secrets/create            
DEBU[2021-04-09T10:30:02.829307200Z] Registering DELETE, /secrets/{id}            
DEBU[2021-04-09T10:30:02.837263392Z] Registering GET, /secrets/{id}               
DEBU[2021-04-09T10:30:02.845743520Z] Registering POST, /secrets/{id}/update       
DEBU[2021-04-09T10:30:02.854053856Z] Registering GET, /configs                    
DEBU[2021-04-09T10:30:02.861897184Z] Registering POST, /configs/create            
DEBU[2021-04-09T10:30:02.870381152Z] Registering DELETE, /configs/{id}            
DEBU[2021-04-09T10:30:02.878792672Z] Registering GET, /configs/{id}               
DEBU[2021-04-09T10:30:02.886619744Z] Registering POST, /configs/{id}/update       
DEBU[2021-04-09T10:30:02.895177248Z] Registering GET, /plugins                    
DEBU[2021-04-09T10:30:02.903074656Z] Registering GET, /plugins/{name:.*}/json     
DEBU[2021-04-09T10:30:02.911559040Z] Registering GET, /plugins/privileges         
DEBU[2021-04-09T10:30:02.919796896Z] Registering DELETE, /plugins/{name:.*}       
DEBU[2021-04-09T10:30:02.928064992Z] Registering POST, /plugins/{name:.*}/enable  
DEBU[2021-04-09T10:30:02.935987008Z] Registering POST, /plugins/{name:.*}/disable 
DEBU[2021-04-09T10:30:02.944507488Z] Registering POST, /plugins/pull              
DEBU[2021-04-09T10:30:02.952383808Z] Registering POST, /plugins/{name:.*}/push    
DEBU[2021-04-09T10:30:02.960738976Z] Registering POST, /plugins/{name:.*}/upgrade 
DEBU[2021-04-09T10:30:02.968938304Z] Registering POST, /plugins/{name:.*}/set     
DEBU[2021-04-09T10:30:02.977089664Z] Registering POST, /plugins/create            
DEBU[2021-04-09T10:30:02.985606624Z] Registering GET, /distribution/{name:.*}/json 
DEBU[2021-04-09T10:30:02.993802272Z] Registering POST, /grpc                      
DEBU[2021-04-09T10:30:03.001982848Z] Registering GET, /networks                   
DEBU[2021-04-09T10:30:03.010276736Z] Registering GET, /networks/                  
DEBU[2021-04-09T10:30:03.018481568Z] Registering GET, /networks/{id:.+}           
DEBU[2021-04-09T10:30:03.026818432Z] Registering POST, /networks/create           
DEBU[2021-04-09T10:30:03.034672672Z] Registering POST, /networks/{id:.*}/connect  
DEBU[2021-04-09T10:30:03.043198400Z] Registering POST, /networks/{id:.*}/disconnect 
DEBU[2021-04-09T10:30:03.051420736Z] Registering POST, /networks/prune            
DEBU[2021-04-09T10:30:03.059672064Z] Registering DELETE, /networks/{id:.*}        
INFO[2021-04-09T10:30:03.068520160Z] API listen on /var/run/docker.sock

docker log output in /var/log/message.log:

2021-04-09T09:05:57.655114+00:00 (none) root: dockerd has started.
2021-04-09T09:05:59.964011+00:00 (none) connmand[2689]: docker0 {create} index 7 type 1 <ETHER>
2021-04-09T09:05:59.964032+00:00 (none) connmand[2689]: docker0 {update} flags 4098 <DOWN>
2021-04-09T09:05:59.964045+00:00 (none) connmand[2689]: docker0 {newlink} index 7 address 02:42:3B:99:8A:65 mtu 1500
2021-04-09T09:05:59.964055+00:00 (none) connmand[2689]: docker0 {newlink} index 7 operstate 2 <DOWN>
2021-04-09T09:05:59.964361+00:00 (none) connmand[2689]: docker0 {add} address 172.17.0.1/16 label docker0 family 2
2021-04-09T09:06:00.014695+00:00 (none) connmand[2689]: docker0 {update} flags 4163 <UP,RUNNING>
2021-04-09T09:06:00.014721+00:00 (none) connmand[2689]: docker0 {newlink} index 7 address 02:42:3B:99:8A:65 mtu 1500
2021-04-09T09:06:00.014733+00:00 (none) connmand[2689]: docker0 {newlink} index 7 operstate 0 <UNKNOWN>
2021-04-09T09:06:00.015240+00:00 (none) connmand[2689]: docker0 {add} route 172.17.0.0 gw 0.0.0.0 scope 253 <LINK>
2021-04-09T09:06:00.017476+00:00 (none) kernel: [   21.481832] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
2021-04-09T09:06:00.901990+00:00 (none) connmand[2689]: docker0 {update} flags 4099 <UP>
2021-04-09T09:06:00.902021+00:00 (none) connmand[2689]: docker0 {newlink} index 7 address 02:42:3B:99:8A:65 mtu 1500
2021-04-09T09:06:00.902033+00:00 (none) connmand[2689]: docker0 {newlink} index 7 operstate 2 <DOWN>
madisongh commented 3 years ago

All that looks OK. You mentioned the l4t-base container in your OP, though, and now you're talking about the tensorflow container. Did you get the base container to work? Are you getting the same error as before?

I usually test with the base container and the DeepStream container, using the demo-image-full image built out of our demo distro, to verify that the container support is working. If I get a chance in the next couple of days, I'll try the tensorflow container.

RomainGsd commented 3 years ago

Yes I'm testing with these two containers alternatively sorry. Base container also works, but the problem comes from docker who don't want to run if I add --runtime nvidia to the command line.

DeepStream runs too, I'll take a look at the demo-image-full image for my tests then.

arpit6232 commented 3 years ago

The following seemed to work for me on the following parameters Board: NVIDIA Jetson Nano (Developer Kit) Branch : dunfell-l4t-r32.4.3 Commit Hash : d5348114b0a9acd42a473aa4542a19f27f3a2a2c Yocto Setup : . ./setup-env --machine jetson-nano-qspi-sd Build : bitbake demo-image-full NVIDIA Deepstream SDK Installed as part of the binaries under /nvidia/downloads Command Run : docker run --gpus all -it --runtime nvidia nvcr.io/nvidia/l4t-base"r32.5.0

Results on the Jetson Nano IMG_20210409_085627__01

ichergui commented 3 years ago

Thanks @arpit6232 for sharing this

RomainGsd commented 3 years ago

Is it a layer that is installing the nvidia runtime ? I guess it's nvidia-container-runtime but I can't debug why it isn't on my system

Edit: I also got a warning during build from tegra-bootfiles_32.4.3.bb update-boot-partition.sh: file could not be found

madisongh commented 3 years ago

Is it a layer that is installing the nvidia runtime ? I guess it's nvidia-container-runtime but I can't debug why it isn't on my system

It should get pulled in automatically if you install nvidia-docker in your image.

I also got a warning during build from tegra-bootfiles_32.4.3.bb update-boot-partition.sh: file could not be found

No idea where that's coming from. There's no reference to any update-boot-partition.sh script in any of our recipes. Are you including other layers in your build setup?

RomainGsd commented 3 years ago

Are you including other layers in your build setup?

Yes I'm building a system that was doing other things at first and now I try to integrate meta-tegra

I can see that for normal usage nvidia-container-runtime adds /etc/docker/daemon.json with this inside :

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I don't have this file on my system so I created it but it is deleted when I reboot, I might look into that

dwalkes commented 3 years ago

I don't have this file on my system

It should be added by nvida-docker I believe, see this line, so it seems something is going wrong there.

madisongh commented 3 years ago

Yes I'm building a system that was doing other things at first and now I try to integrate meta-tegra

[...]

I don't have this file on my system so I created it but it is deleted when I reboot, I might look into that

Double-check the .manifest file for the image you built (which should end up in$BUILDDIR/tmp/deploy/images/${MACHINE}, with the image) to ensure that nvidia-docker, nvidia-container-runtime and anything else you tried to add to your image actually got included. If they aren't there, then perhaps something in your layers isn't using CORE_IMAGE_EXTRA_INSTALL, or is overwriting the additions you've made.

RomainGsd commented 3 years ago

Double-check the .manifest file for the image you built to ensure that nvidia-docker, nvidia-container-runtime and anything else you tried to add to your image actually got included 

cat sdtv-image-iot-dev-jetson-xavier-nx-devkit.manifest | grep nvidia
libnvidia-container-tools aarch64 0.9.0-r1
libnvidia-egl-wayland1 aarch64 1.1.5-r0
nvidia-container-runtime aarch64 3.1.0-r0
nvidia-container-toolkit aarch64 1.0.5-r0
nvidia-docker aarch64 2.2.2-r0

That's weird, I have it. If I manually create a daemon.json in /etc/docker with the "runtimes" as above and reload dockerd it works and docker info shows nvidia runtime. But it gets deleted when I reboot my device. Problem might come from somewhere else, I will investigate upon that

Edit: Solved! The system is read-only, so we were mounting part of the RAM as tmpfs in order to allow docker write. It's fixed now... Still have other issues though (python tensorrt module empty)