Docker Issue / Documentation

[bryanhughes]

On a newly flashed Jetson Orin AGX 64GB Developer Kit by the SDK Manager, docker fails to run because iptables is configured for /usr/sbin/iptables-nft resulting in the following error:

$ sudo docker run hello-world
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

Check the status of docker daemon

$ sudo systemctl status docker
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-04-15 12:07:27 PDT; 1h 52min ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
   Main PID: 2688 (code=exited, status=1/FAILURE)
        CPU: 219ms

Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 15 12:07:27 mimzy-jetson systemd[1]: Stopped Docker Application Container Engine.
Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Start request repeated too quickly.
Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 15 12:07:27 mimzy-jetson systemd[1]: Failed to start Docker Application Container Engine.

To fix, switch to iptables-legacy

$ sudo update-alternatives --config iptables
There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).

  Selection    Path                       Priority   Status
------------------------------------------------------------
* 0            /usr/sbin/iptables-nft      20        auto mode
  1            /usr/sbin/iptables-legacy   10        manual mode
  2            /usr/sbin/iptables-nft      20        manual mode

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode

Restart docker

$ sudo systemctl restart docker

Docker is now happy:

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-04-15 14:02:37 PDT; 4s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 15010 (dockerd)
      Tasks: 16
     Memory: 38.3M
        CPU: 413ms
     CGroup: /system.slice/docker.service
             └─15010 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.106155008-07:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf:>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.156396528-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.156732499-07:00" level=info msg="Loading containers: start."
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.238743844-07:00" level=warning msg="Could not load necessary modules for IPSEC rules: protocol not supported"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.351532492-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon op>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.407781438-07:00" level=info msg="Loading containers: done."
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.434780554-07:00" level=info msg="Docker daemon" commit=60b9add containerd-snapshotter=false storage-driver=overla>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.435385169-07:00" level=info msg="Daemon has completed initialization"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.482331996-07:00" level=info msg="API listen on /run/docker.sock"
Apr 15 14:02:37 mimzy-jetson systemd[1]: Started Docker Application Container Engine.

Run hello world

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
478afc919002: Pull complete 
Digest: sha256:03b30c6a3c320ff172b52bd68eddffde6ded08ce47e650fe52de861c5e9df46d
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[dusty-nv]

Thanks @bryanhughes, this is an upstream issue with docker/nvidia-container-runtime - did you do apt-get upgrade on your system? Obviously that is something you should be able to do, but I would recommend against it (or pinning the relevant packages or using workarounds like yours) until this issue gets resolved by the docker runtime maintainers.

[dusty-nv]

If you got it working with the iptables-legacy workaround, then I wouldn't worry about reflashing as long as its still working for you!

[bryanhughes]

All fixed with Jetpack 6 GA