search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
757 stars 85 forks source link

Docker Engine does not start on Ubuntu 22.04 due to iptables issues #1437

Open kellervater opened 2 years ago

kellervater commented 2 years ago

Got a server with Ubuntu 22.04 and tried to execute these steps (like described in official docs):

#first remove everything
apt-get remove docker docker-engine docker.io containerd runc -y
apt-get purge docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
apt clean 
apt autoremove -y
rm -rf /var/lib/docker
rm -rf /var/lib/containerd
rm -rf /var/run/docker

apt-get update
apt-get install ca-certificates curl gnupg lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

I always end up having this output:

Setting up docker-ce (5:20.10.19~3-0~ubuntu-jammy) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2022-10-14 08:24:26 UTC; 5ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 23891 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 23891 (code=exited, status=1/FAILURE)
        CPU: 139ms
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1

I tried all Docker versions down to 5:20.10.13~3-0~ubuntu-jammy.

When checking the logs with dockerd -D I found 2 interesting lines:

...
WARN[2022-10-14T08:24:30.801222966Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.`, error: exit status 1
...
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
 (exit status 4))

So I also reinstalled iptables (v1.8.7, there's no other version to try when looking in apt-cache madison ).

Nothing helped so far. I also tried installing it via Ansible roles (geerlingguy) in the first place. this works perfectly fine on my Ubuntu 20.04 servers.

AND one addition: It once worked on Ubuntu 22.04 when installing docker during the installation of the OS itself. I recently reinstalled the OS without docker and did the installation as described above, which led to this issue.

Does someone have a clue what's going on or what am I doing wrong?

mijofa commented 2 years ago

I seem to have also run into this issue, it was working fine for weeks, but I rebooted early this week and now it's all stopped working.

Looks like maybe a change to the kernel/iptables versions had broken iptables entirely, which docker requires. I don't think it's a problem with docker itself

fabianbees commented 2 years ago

I have also experienced this issue, but could resolve it by downgrading iptables (with running the following comands as root):

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
mijofa commented 2 years ago

Oh I forgot I'd checked in on this one. The issue I ran into was actually this one with Ubuntu upstream: https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1958260

I don't know if you are having the same problem, but just in case, I was able to get it working again with::

sudo umount --lazy /usr/lib/modules
# Make sure this only reinstalls things, no newly installed packages.
# You should be safe to purge any packages it wants to newly install before rerunning it.
sudo apt install --reinstall $(sudo dpkg-query -S /lib/modules | sed 's/,//g;s/:.*$//')  
sudo reboot

I have not yet confirmed that it won't come back again, nor have I determined why/how it became a problem in the first place since I'm not running anything "cloud" on this system.

eagleusb commented 1 year ago

I have also experienced this issue, but could resolve it by downgrading iptables (with running the following comands as root):

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

I confirm that the latest default version of iptables is against nftables and using the legacy ones solve it. 😸

kellervater commented 1 year ago

Since I was under time-pressure, I went back to Ubunut 20.04 to workaround this. But if you guys were able to workaround this by using legacy modules, I think we can let this issue go stale or even close it.

Rositza commented 1 year ago

Hi, I found the issue while searching for decision. Actually I have the same problem on Ubuntu 20.04.3 LTS.

davidmiguel02 commented 1 year ago

Help me. Thank you

Idosegevv commented 1 year ago

Worked for me ass well on Petalinux dist of PYNQ-RFSoC

spirillen commented 7 months ago

Well this is still an issue that have to be fixed, as iptables have been deprecated for year for nftables. So why are the commands not updated to match the real world realities...

DEBU[2024-04-09T13:26:36.186488670Z] Network Control Plane MTU: 1500              
DEBU[2024-04-09T13:26:36.188829616Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION] 
DEBU[2024-04-09T13:26:36.189759746Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2024-04-09T13:26:36.190840751Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[2024-04-09T13:26:36.191795232Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2024-04-09T13:26:36.192827253Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING] 
DEBU[2024-04-09T13:26:36.193805364Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT] 
DEBU[2024-04-09T13:26:36.194672033Z] /usr/sbin/iptables, [--wait -t nat -F DOCKER] 
DEBU[2024-04-09T13:26:36.195517505Z] /usr/sbin/iptables, [--wait -t nat -X DOCKER] 
DEBU[2024-04-09T13:26:36.262160016Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER] 
DEBU[2024-04-09T13:26:36.265093372Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER] 
DEBU[2024-04-09T13:26:36.267894264Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-1] 
DEBU[2024-04-09T13:26:36.322261864Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-1] 
DEBU[2024-04-09T13:26:36.325635441Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-2] 
DEBU[2024-04-09T13:26:36.390282204Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.393465502Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[2024-04-09T13:26:36.396226517Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[2024-04-09T13:26:36.399023334Z] /usr/sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[2024-04-09T13:26:36.401828580Z] /usr/sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[2024-04-09T13:26:36.404467507Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[2024-04-09T13:26:36.407222513Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER]
DEBU[2024-04-09T13:26:36.409686321Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-1]
DEBU[2024-04-09T13:26:36.412109893Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-1]
DEBU[2024-04-09T13:26:36.414436184Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.416687171Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.418934837Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2024-04-09T13:26:36.421334686Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2024-04-09T13:26:36.423607874Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2024-04-09T13:26:36.425957943Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2024-04-09T13:26:36.435780805Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.438306631Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.440625841Z] /usr/sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.442995562Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.445555108Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j DROP]
DEBU[2024-04-09T13:26:36.447780342Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.450094555Z] /usr/sbin/iptables, [--wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT]
WARN[2024-04-09T13:26:36.486139751Z] could not create bridge network for id ca495b73f0d5180761e35cfab4e58571671bf1bb1eb1c66e681b7fbac426d96f bridge name docker0 while booting up from persistent state: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
(exit status 4))
DEBU[2024-04-09T13:26:36.486184603Z] Network (ca495b7) restored
INFO[2024-04-09T13:26:36.492367578Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
DEBU[2024-04-09T13:26:36.492435409Z] Allocating IPv4 pools for network bridge (a6381b2be7e366a9597643115f5bf823a7c28ee2f6dd5fa0642d1981841ab132)
DEBU[2024-04-09T13:26:36.492454280Z] RequestPool(LocalDefault, 172.17.0.0/16, , _, false)
DEBU[2024-04-09T13:26:36.492498645Z] RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway])
DEBU[2024-04-09T13:26:36.492537499Z] Request address PoolID:172.17.0.0/16 Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:172.17.0.1
DEBU[2024-04-09T13:26:36.492919071Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.495827334Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.498628169Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.501635211Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j DROP]
DEBU[2024-04-09T13:26:36.504260105Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.506858071Z] /usr/sbin/iptables, [--wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.554482717Z] releasing IPv4 pools from network bridge (a6381b2be7e366a9597643115f5bf823a7c28ee2f6dd5fa0642d1981841ab132)
DEBU[2024-04-09T13:26:36.554532727Z] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.1)
DEBU[2024-04-09T13:26:36.554573789Z] Released address Address:172.17.0.1 Sequence:Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0
DEBU[2024-04-09T13:26:36.554591415Z] ReleasePool(LocalDefault/172.17.0.0/16)
DEBU[2024-04-09T13:26:36.554620258Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2024-04-09T13:26:36.554640143Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2024-04-09T13:26:36.556095581Z] Unix socket /var/run/docker/libnetwork/e28645b2e610.sock was closed. The external key listener will stop.
INFO[2024-04-09T13:26:36.556801370Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
DEBU[2024-04-09T13:26:36.556828184Z] Cleaning up old mountid : start.
DEBU[2024-04-09T13:26:36.557417089Z] Cleaning up old mountid : done.
failed to start daemon: Error initializing network controller: error creating default "bridge" network: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
(exit status 4))

Keep using deprecated software is just so bad and calls out for errors and insecurity

nandaadi241 commented 5 months ago

thanks bro its help me