Ideas to make it work with firewalld

[coolcodemy]

I do have problem with firewalld where everytime i do firewall-cmd --reload, it clears everything. At 1st i'm having problem with docker itself not retaining the config after reload, but it can be fixed by adding <chain ipv="ipv4" table="filter" chain="DOCKER-USER"/> to /etc/firewalld/direct.xml. So I tried adding <chain ipv="ipv4" table="filter" chain="DOCKER-INGRESS"/> to the same file, it did retain certain iptables config, but not the DIRD config. Tried firewall-cmd --runtime-to-permanent where it will write current iptables that is currently running to the xml config file, but after reload, it still doesn't work.

So I tried collecting variables like $NID, $NODE_ID, $NODE_IP and $INGRESS_SUBNET and testing added everything manually after firewall reload, but it doesn't work. It will only work after I kill the running container for swarm to launch a new one. So I thought it is probably Docker, not DIRD. But after stopping DIRD and only using Docker and swarm, everything works, except the real visitor ip address.

Im using DIRD with nginx container, and whenever i reload my firewall, the site that im trying to browse will load indefinitely. Hope you could shed some light on how to make it work. Thanks.

[coolcodemy]

I did try check ip route, ip rule and iptables by using nsenter, however, everything is still there as being set by DIRD. so, not sure what is the problem.

[struanb]

Hi @coolcodemy. Thanks for trying DIND.

I can well understand it is hard to make DIND play nice with firewalld or other iptables rule managers! Not only that, but DIND has to load after dockerd since dockerd itself also manages iptables rules.

DIND currently detects when dockerd exits, or is not running, or has not yet joined the swarm: in these cases, it sleeps one second and aborts, to allow dockerd time to start up (and if necessary join the swarm), assuming that systemd (or another supervisor) will restart it promptly.

However DIND cannot detect when firewall rules are manipulated by a third program. I would recommend that whenever you restart firewalld, you arrange to clear all existing firewall rules before firewalld rewrites them, and restart dockerd as well, so that firewalld, dockerd, and then DIND, can reinstate the firewall rules they are each responsible for in the correct order. (And, if you can't do this automatically, arrange to do this manually).

This is the approach we've had to take at NewsNow, where we don't run firewalld, but do run a lengthy custom script to set up our firewall rules.

I hope this helps.

[coolcodemy]

Thank you for answering @struanb. However, restarting Docker or killing the container is not an option. I may need to find new way to tackle this. Reading DIRD script, I could understand well what it does, however I'm not proficient in iptables.

I did try printing all iptables rules before reload (the working ones) and after reload (which loading indefinitely) and I could find the difference. After reload, It is missing these lines.

-A DOCKER-INGRESS -p tcp -m tcp --dport 443 -j ACCEPT   
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 443 -j ACCEPT  
-A DOCKER-INGRESS -p tcp -m tcp --dport 80 -j ACCEPT    
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 80 -j ACCEPT   
-A DOCKER-INGRESS -j RETURN

But after rerun iptables with those lines, it still not routing to the container. Perhaps you know something which I don't.

[struanb]

Hi @coolcodemy. The iptables lines you've listed look like those installed by dockerd for a service listening on ports 80 and 443. I would not recommend trying to reproduce firewall rules dockerd is responsible for (or DIND for that matter).

Instead, the approach I'd recommend is to try and arrange that your firewalld firewall rules are written in such a way that they can be added and removed independently from dockerd's rules - in other words trying to separate the concerns.

I'm not familiar with firewalld, but to achieve this I would aim to arrange that each iptables chain you need to modify, has a custom subchain hooked up initially, perhaps by a separate script to firewalld; then have firewalld populate the custom subchains only.

For example:

• for the filter table INPUT chain, run iptables -N RAW_INPUT; iptables -I INPUT -j RAW_INPUT then rewrite any INPUT rules you have to be RAW_INPUT rules
• for the filter table FORWARD chain, dockerd already creates the DOCKER-USER chain; rewrite any FORWARD chain rules you have to be added to the DOCKER-USER table, as dockerd won't mess with these

At the same time, I would also arrange for firewalld never to touch the top-level chains (INPUT, OUTPUT, FORWARD, and so on) but only my custom subchains, so that when firewalld is restarted (or told to update firewall rules) it does not interfere with the top-level chains, or with anything dockerd installs.

If firewalld cannot be so configured, then I would think it necessary to find an alternative to firewalld that 'plays nice' with firewall rules created by other systems.

I do hope this helps.

[struanb]

P.S. As to your specific question, about the DOCKER-INGRESS lines: as mentioned, dockerd would create these when it sets up a service listening on ports 80 and 443 to which it needs to load balance incoming requests, but these are not the only rules it sets up. The ones you listed are set up in the filter table; but others are set up in the nat table; both in the host's default namespace.

And others are set up in the ingress network namespace. Try running: nsenter --net=/var/run/docker/netns/ingress_sbox iptables-save to see these.

If firewalld messes with any of these rules on a reload, then it is going to break a running docker service. And if it cannot be configured to avoid doing this, then in my opinion it is may not be the right tool for the job of configuring firewall rules that are needed to coexist with those configured by dockerd.

[coolcodemy]

@struanb thank you very much. Hope i could come back with an answer for this dilemma.

[struanb]

@coolcodemy You're welcome. As I don't think this represents a problem with DIND, I'll close this issue now.