Docker networking fails after iptables service is restarted

[makeamark1]

Docker networking fails permanently on all new containers after restarting the iptables service. Steps to reproduce on CentOS 7.1 (some output elided for brevity):

$ sudo yum -y install docker iptables-services
$ sudo systemctl start docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms
$ sudo systemctl restart iptables
$ sudo docker run --rm centos bash -c "ping www.docker.com"
ping: unknown host www.docker.com

A workaround to restore networking to containers is to restart the Docker daemon:

$ sudo systemctl restart docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms

What should be fixed in Docker so that reloading iptables rules won't break its networking? This is a major issue because it breaks networking for all containers on the host.

[thaJeztah]

The advanced networking may contain some information that can help troubleshooting.

Also, could you add the output of;

• uname -a
• docker version
• docker -D info

[phemmer]

This is a known issue/limitation. I'm hesitant to call it an issue as I don't think it's docker's problem. Basically when you run systemctl restart iptables, you're not really restarting iptables as iptables is always on (it's built into the kernel). Instead you're clearing & loading saved rule sets. Docker adds rules to the iptables chains, so when you clear the chains, the rules docker created are gone. There is nothing docker can do about this. You just can't wipe the iptables chains.

[LK4D4]

@phemmer There is proposal about firewalld support. WDYT about it?

[phemmer]

Great if you're using firewalld :-P Unfortunately basic iptables is what a lot of people use (myself included).

From the 40,000 foot level, I see only 2 ways to solve this problem:

1. make docker depend on the iptables service. If iptables is shut down, so is docker. Not great as I'm sure people will not expect docker to go down when iptables does.
   1. Could also have a separate service which depends on the iptables service, so that when it comes up, it pings the docker daemon to tell it to reinstate its rules.
      Maybe as part of the new docker network driver design, the network manager would be a separate process, and it could be stopped/started instead. That way containers would remain running, but the network service wouldn't.
2. monitor iptables and reinstate the rules if they get deleted. Doesn't look like there's any sort of accounting interface you can use to watch changes, so would have to be polling (and that's a lot of polling if you have a lot of rules). Plus there's still going to be brief gaps where the rules are missing.
3. Yeah I said 2, but 3: not docker's problem. Just like wiping iptables breaks docker, so would wiping the filesystem, and Docker doesn't prevent that. Just because there's a handy tool for doing so doesn't mean the existence of that tool is something docker should handle. Just throw a warning somewhere in the documentation (if there isn't one already).

[LK4D4]

I think using firewalld is not very bad option if you're fan of restarting iptables service :)

[gdm85]

I made this utility to cover a similar need: being able to restore sets of iptables rules that affect multiple containers.

[clnperez]

I've seen this on another project I worked on (kimchi: https://github.com/kimchi-project/kimchi/wiki) and we concluded that it's not Kimchi's problem if you wiped the rules it creates on startup. Something in the docs about saving and restoring is the best way to go, IMO.

[virtualfunction]

While iptables shouldn't be docker's responsibility if it gets restarted, it would be good if there was a mechanism (maybe there is one hiding, and just needs to be documented to make it clear to users) to save / restore rules docker has created. I'm not really a iptables / sysadmin expert but I guess one could do something with iptables-save and filtering out the DOCKER chain as a basic starting point. (Just noticed in the referenced issue there are instructions, but that saves the whole iptables state - some just want to clear everything except the docker rules)

[chulkilee]

What about adding a command or mechanism that invoke that network setup phase (updating iptables) without restarting docker daemon?

[michaljrk]

Another workaround (?), not elegant at all, but quick. Put service docker restart in the /etc/init.d/iptables-persistent, in start|restart|reload|force-reload) section.

@virtualfunction - mainly agree on that.

But I would not say While iptables shouldn't be docker's responsibility

Exactly, if you relay on some other service in OS, it would be nice to assume that user still may use this mechanism for some other purposes (like making host level iptables rules for whatever reason). Then it will be nice to help to compromise this kind of misbehaviour.

+1

[bish0polis]

sed -i~ '
/^IPTABLES_SAVE_ON_/s/=.*/="yes"/
' /etc/sysconfig/iptables-config

Something I'm not getting, here?

I mean, it's gonna be way harder if you're using chef/etc to scorch and recreate the tables config from scratch each time, but there's a workaround for that (which I was doing at the shop, which led me here) anyway. For most people in the target audience, the tuning above should be all ya need.

[colinmollenhour]

Definitely having a command like "docker network reload-firewall" would be nice so that if you run into a situation you can fix the firewall without having to restart all of your containers..

[taladar]

iptables-save is a messy non-solution. Any semi-complex set of rules needs to be written in a more structured way, with comments. You also do not want to save Docker's temporary rules since the saved rules file is usually used on system startup (especially when it results from an unclean shutdown without a save and the saved rules file is a little older), when half of them are obsolete because the containers they belonged to no longer exist.

Docker simply needs a command to recreate its rules without restarting docker itself. It does not have to determine automatically when that command has to be called, it is easy enough to add it to any script flushing and recreating firewall rules. Honestly, I can't see how anyone can run Docker on a production server with any sort of availability requirement without a way to use iptables for other things.

[gdm85]

Meanwhile, there's been some progress in go-libiptc, you might want to give a look for a possible lower-level integration. Has anybody tried my docker-fw with most recent Docker?

[qwIvan]

I want to recreate iptables rules without restart dockerd

[taladar]

The fact that this ticket is still open is one of the things I regularly point to when people ask why I consider Docker far from production ready (the others include the lack of proper garbage collection).

Quite frankly the whole Docker container hype is a bit of a bad joke while tickets like this one are completely ignored.

[strootman]

@taladar All of GH is running on Kubernetes! IMO, it's well past hype.

[taladar]

It works for anything large enough that you can just take a host where Docker breaks out of your load balancer and fix it while the rest keeps on running but if you actually need your hosts to work reliably without a few hours of unplanned downtime every once in a while Docker still isn't there yet.

[cpuguy83]

It would be inappropriate for Docker to re-insert rules automatically after they have been removed from iptables.

Providing an API to re-init iptables rules is something to look at, but it's not obvious (to me) what this API would look like. It could be supported on "SIGUSR1", but then that's really starting to overload that function.

[bish0polis]

Brian Goff wrote:

It would be inappropriate for Docker to re-insert rules automatically after they have been removed from iptables.

It's doing so on startup, for objective definitions of 'remove'. It's just not doing it otherwise.

Ideally, if Docker adds rules, it should remove them on shutdown/unload. Then we can save the rules on iptables unload, and restore them on restart, and both iptables and docker could play nice.

Enter automation, which often defines a static set of rules, and we spot a common situation where docker wasn't tested. That's all.

[taladar]

You can not save and restore Docker iptables rules externally unless you have only static, long-running Docker containers with exposed ports.

What Docker really just needs is the same thing most other, similar software does in this case. Call a configurable script on each event (startup, container start, container stop, shutdown) and pass all the relevant data to the script as parameters.

[paklids]

I have a slightly different use case and have been frustrated there isn't a workaround. I'm pushing out very specific iptables rules using ansible and those remove the docker created rules. So I wrote some shell scripts (leveraging netfilter-persistent) that make a backup of the running rules, allow me to push my ansible rules, then restore only the docker specific ones. Its a bit of a hack but gets me closer to my goal. If these might be useful for you then you can pull them down at:

https://github.com/paklids/docker-iptables-restore

[ahmadalli]

kube-proxy is doing the same but also reattach itself to iptables after reload

[hudsantos]

This helped me a lot with this issue: https://unrouted.io/2017/08/15/docker-firewall/ Baiscally doing things with iptables-restore -n to avoid flushing existing rules.. Docker rules untouched.

[taladar]

As a workaround for Docker's inability to play nice with the rest of the system we are now just isolating Docker in its own named network namespace using

https://github.com/Jamesits/systemd-named-netns

That way Docker has its own iptables rules and its own IP and we can finally modify the regular iptables rules freely without requiring constant Docker restarts that take ages.

[nixikanius]

@taladar could you please give more explanation about your workaround? Will be very appreciated it.

[adrmaas]

It seems that originally code checked for the DOCKER chain and adds it if necessary each time a new rule is added but a suggestion to the PR changed this to only check on startup: https://github.com/moby/moby/pull/7003#issuecomment-49115936

I'd be curious to know if the original code would have had the same defect.

[benjamin-kirkbride]

Maybe we should just run Docker inside of Docker, so that we don't have to keep the ephemeral nature of these rules from causing problems? \s

[anarcat]

we are also having this problem in GitLab CI runners here: when the firewall system (ferm) reloads because of outside conditions, it flushes the firewall rules and breaks Docker. it would be nice if service docker reload would also reload the firewall rules (it seems we need service docker restart now) so that we could have a PropagateReloadsTo relationship between the two services.

as things stand now, we had to add this horrible override in /etc/systemd/system/ferm.service.d/override.conf:

[Service]
ExecReload=/etc/init.d/ferm reload
ExecReload=service docker restart

i understand the docker people might not feel responsible for this, but maybe this small tweak could be sufficient to solve this problem?

alternatively, we're also considering using podman as a container runner as well. we suspect it might be better suited for short-lived jobs like this anyways, and will likely not suffer from this problem.

[crabdancing]

it's been six years. How hard is it to add a docker command to re-apply docker's firewall rules? :/

[brandnewx]

This is a major flaw of Docker that the team refuses to fix. There must be a way for Docker to reload the containers' iptables rules.

[Vlad1mir-D]

libvirtd handles SIGHUP for flushed firewall rules: https://libvirt.org/firewall.html Why can't Docker handle this?

[q0xz]

Estoy tan frustrado que es mi primera vez que comento un proyecto público. Por favor, agreguen un comando que vuelva a cargar las reglas de iptables sin reiniciar docker. Gracias.

[dm17]

Has anyone tried these projects as a work around? https://github.com/shinebayar-g/ufw-docker-automated https://github.com/chaifeng/ufw-docker

[rcstanciu]

I've been having a similar issue with GCP CentOS VMs, where after a while, like 30 days uptime, the iptables rules would get flushed, basically making the containers useless. As I don't have extensive knowledge on sysops, I did not pinpoint the exact issue that was causing this behaviour. After reading this issue, I thought about an automatic package update, that would trigger the reset of iptables.

[jerrac]

Has anyone using firewalld tried setting their default zone to the docker zone, and then applying their public rules to that zone?

The docs say that anything in the DOCKER-USER chain will get applied before any other rules. Which made me think that the solution to this issue is to put all our rules into DOCKER-USER.

My thought is that the docker zone might do that for firewalld.

Anyway, as soon as I have time, I'll be experimenting to see if that approach works. I'm just kinda hoping someone else might have already done so. :)

Merry Christmas, btw.

[andreasruxanda]

I'm surprised that nobody has mentioned it, but there is actually a built-in, working solution to this problem: the Docker "live restore" option (which has been around since 2016!): https://docs.docker.com/config/containers/live-restore/

After configuring that option, you need to restart docker once to load it, and after that, the containers will continue to run during all subsequent restarts (and not only will the restart process be very fast, but it will also restore the firewall rules if they don't exist).

I'm using it in multiple Ansible & Docker environments, and it works like a charm.

[hudsantos]

After configuring that option, you need to restart docker once to load it

It's preferable to only reload (not restart) like systemctl reload docker OR send a SIGHUP (kill -1) signal to the dockerd process.

But what's important is that you said it restores firewall rules.. That's very nice. Didn't knew about that.

[andreasruxanda]

I do agree. Still, you can look at this as being a pseudo-reload, because it does not affect the running containers (or the traffic going to them) in any way.

[ccastillop]

For now I've just restarted host server. I'm having this:

# docker -D info
Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: true

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-1045-aws
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB
 Name: ip-172-31-8-215
 ID: abf70869-8073-4b84-b054-d53b15f79463
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
  File Descriptors: 32
  Goroutines: 44
  System Time: 2023-10-01T11:57:37.825778075-05:00
  EventsListeners: 1
 Username: xxxxxx
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false