Docker bypasses ufw firewall rules

[binaryfire]

• [x] This is a bug report
• [ ] This is a feature request
• [x] I searched existing issues before opening this one

Expected behavior

Hi all!

ufw in ubuntu should be treated as the "master" when it comes to low level firewall rules (like firewalld in rhel). However docker bypasses ufw completely and does it's own thing with iptables. It was only by chance (luckily!) we discovered this. Example:

ufw deny 8080 (blocks all external access to port 8080) docker run jboss/keycloak

Expected behaviour: the Keycloak container should be available at port 8080 on localhost/127.0.0.1, but not from the outside world.

Actual behavior

UFW reports port 8080 as blocked but the keycloak docker container is still accessible externally on port 8080.

There is a workaround (https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/) however I think techrepublic are correct when then describe it as a "security flaw", and it's a pretty serious one. Most people using ubuntu user ufw. I imagine a large number of them are unaware their UFW rules are being bypassed and all their containers are exposed.

Is this something that can be addressed in the next update? That article was published in Jan 2018.

[preciz]

When I used docker run -p [ipv6address]:port:port and enabled ufw the port got blocked properly even on that ipv6 address. Until this issue is solved I assume it's a more obscure option to bind the port to an IPv6 address.

[lllibailll]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

It worked for me too.

[jhgehrs]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

thank you, worked for me too

[huggge]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

thank you, worked for me too

Note that the $INTERFACE variable must be replaced with the name of the primary host interface used by Docker (such as eth0 or eno1).

How do I find out what primary host interface Docker is using?

[art-dambrine]

Is there any official/best practice/whatever approach to get pass this problem?

Another option, which is what I usually do, is bind exposed container ports to 127.0.0.1 (by default they're bound to any IP address - 0.0.0.0), which makes them inaccessible outside the local system. I then run a reverse proxy and route traffic to the containers in that way.

Worked like a charm in my case thank you ! Didn't think of it the first time I was facing this problem 👍 (I might have to go deeper for more complex cases but it suits my needs for now using a nginx reverse proxy)

[Gaulomatic]

This issue will be addressed shortly after the Vulcans land in Bozeman, Montana.

[peterthehermit]

This is ridiculous, get yer house in order docker.

[ninobaldo]

After a long time I found the must simple way do to this. Just follow https://github.com/chaifeng/ufw-docker#solving-ufw-and-docker-issues

[Cryotize]

Its been nearly 2 years now, any update on this?

[Gaulomatic]

Docker gives a crap about this issue (among others). I switched over to Podman for everything I can. This might not be the a solution in all cases, but in production is most certainly is. Podman works quite well with firewalld, which is more flexible anyway. It took me a couple of days to work around the differences between Docker and Podman, but I don't look back.

Container orchestration is not quite there yet, but it is worth taking a look at. For me it seems like Docker is on the Nokia type track.

[mr-karan]

Another company who got seriously impacted by this: https://news.ycombinator.com/item?id=27613661

[Pimmelton]

Thanks, @mr-karan . This is a great example of how even non-novice Docker users can easily be bitten by this problem because it blatantly violates the principle of least surprise. Silently bypassing security measures set up by a primary system security tool on an officially supported (https://docs.docker.com/engine/install/) distribution is not what people expect. This almost meets the definition of backdoor: "A backdoor is a means to access a computer system or encrypted data that bypasses the system's customary security mechanisms.". Maybe this is an accidental backdoor created in the name of convenience rather than a deliberate one intended for malicious purposes, but the results are the same.

Unbelievable.

[matthewblott]

Excellent solution provided here. It's the instructions (reasonably straightforward) to create an alternative ufw-docker to use instead of ufw for creating rules. It has over a thousand stars and I'm surprised it's not been flagged already in this thread but I'm using it now and it seems okay so far.

[g4z]

it's been mentioned multiple times in this thread. for example, kaysond commented on 10 Sep 2019 above. personally, i cannot get it to block the incoming traffic to container ports over tun0 :(

[rmrf-sl4sh]

How is something like this even possible????

[oonqt]

The fact that his has gone completely ignored for years is extremely pathetic. Another reason for me not being a fan of Docker.

[J4gQBqqR]

I come across the news today and it surprises me. I never thought that I opened my server's ports to the public internet because Docker circumvent UFW!

It gives me a sudden rise of blood pressure and keeps it high for the rest of the day until I shut my ports off behind a physical firewall.

This is really BAD.

I would not blame UFW for not handling docker but the opposite. If I installed Chrome on my machine and Chrome opened my firewall ports, I will hold Chrome for responsibility. The same hold for Docker.

According to my reading, this issue has been reported in the 2014. The team member closed that issue by asking everyone to not use UFW but to use IPTABLES instead (if I am reading it correctly).

It is pathetic. 7 years, nothing done. Not even a significant warning in the documentation. I am now seriously starting to consider other containerization tools.

[d-513]

It has been 3 years. Even more from older closed issues. Why has NOTHING been done to fix this? Why is this issue completely ignored from docker maintainers? There are workarounds, but come on, this is not okay

[matthewblott]

FWIW I've switched to using rootless docker and the problem has gone away.

[yeahman45]

I use rootless too..though it has limitations (ex. Host mode does not work)

On Wed, Oct 27, 2021, 2:06 PM Matthew Blott @.***> wrote:

FWIW I've switched to using rootless docker and the problem has gone away.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/690#issuecomment-952755976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEJTZP3G3KHLXBUWEAG3GJDUI7MKVANCNFSM4HTV7X6Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[d-513]

Honestly: I was complaining like everybody else here. In the meantime I switched to firewalld which works excellent on Ubuntu on Debian:

• can be used quite easily once you get your head around zones (and runtime config vs permanent config -- which is a great feature in itself)
• works flawless with docker and podman

Do yourself a favour: ditch UFW. Either go iptables (hardcore), or enjoy firewalld (comfy).

Doesn’t solve the problem. Docker shouldn’t bypass any firewall, be it UFW, firewalld or iptables. It’s not a UFW problem - it’s docker’s problem. If Chrome opened ports in my firewall, that wouldn’t be my firewall’s bug but Chrome’s

I’m sure firewalld is great but for the thousands of users that think their UFW is protecting them this is a very major security flaw

[BouDev]

Damned this one is nasty ! It is that I run an automated nmap scan on my infra every week or else I've would have missed this one.

[TumTumTom]

This problem has been going on for years. I'm also still waiting for a fix from Docker. I also use this workaround, as I use UFW by default in Ubuntu: https://github.com/chaifeng/ufw-docker

[BouDev]

This problem has been going on for years. I'm also still waiting for a fix from Docker. I also use this workaround, as I use UFW by default in Ubuntu: https://github.com/chaifeng/ufw-docker

Yeah looked at it. Lot of hassle to keep using an "uncomplicated" firewall :D. In my case I just switched back to Iptables. But still, apart from this nice feature. It at least would be nice to get a "REDALERT!!!" somewhere along the line from UFW or in a log file.

[picode7]

Got hacked because I trusted the ufw. Now all data is gone and got published. At least I have backups.

[BlueskyFR]

This is becoming a joke as the years are passing 😅

[danielfree19]

Why is it still a thing? Why docker writes to iptables automatically and we dont have a setting if it is a desired option?

[amirshnll]

btw, I'm think all that's needed is to insert a jump rule from DOCKER-USER to ufw-user-forward

e.g.

sudo iptables -A DOCKER-USER -j ufw-user-forward

work it

[grandeto]

Here's is a relatively simple way to make ufw generated iptables rules take precedence over the docker ones

sudo nano /etc/ufw/after.rules inside *filter section append the following 2 lines just before the # End required lines, like this:

:DOCKER-USER - [0:0]
:ufw-user-input - [0:0]
# End required lines

then insert the following rule -I DOCKER-USER -j ufw-user-input -m comment --comment "forward to ufw" just before COMMIT, like this:

# Forward DOCKER-USER chain to ufw-user-input chain
-I DOCKER-USER -j ufw-user-input -m comment --comment "forward to ufw"

# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT

reboot and validate the DOCKER-USER chain now has the ufw forward rule at position 1:

sudo iptables -t filter -L DOCKER-USER --line-numbers -n -v

the output should look like this:

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1      104 40743 ufw-user-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* forward to ufw */
2       48 32821 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

test if your use-cases match against the ufw defined rules

EDIT:

Setup DOCKER-USER chain to be rebuild on ufw stop (to prevent ufw start failure after ufw disable/enable)

sudo nano /etc/ufw/before.init

    stop)
        # typically required
        iptables -F DOCKER-USER || true
        iptables -I DOCKER-USER -j RETURN || true
        ;;

sudo chmod +x /etc/ufw/before.init

sudo ufw disable && sudo ufw enable

[zhen-huan-hu]

test if your use-cases match against the ufw defined rules

Using ufw-user-input could potential cause problems as explained here: https://github.com/chaifeng/ufw-docker

[3dprogramin]

I setup docker compose for mongodb on CentOS 7 with ufw as firewall, was surprised to see the mongodb accessible from the outside, without adding any ufw rule for it.

The easy (and safe if you ask me) solution I'm using is specifying the IP address on which to bind as well.

27017:27017 to 127.0.0.1:27017:27017

However you put it, it's 2023 and this still has big security implications. At least on docker installation on an OS with ufw installed and enabled, there should be big warning in red to let the users know about this.

[swfsql]

Can one make root-mode docker "simply" behave like root-less docker in regards to the networking rules?
(I had to switch to root-mode, and got surprised by this iptables behavior..)

[Darkzarich]

Just learned about this issue. I installed Crowdsec on my server and all banned IPs still can request to my services. What a joke 😞

[Darkzarich]

To anyone facing the same issue while setting up fail2ban or Crowdsec do the following

Crowdsec:

sudo iptables -I FORWARD -m set --match-set crowdsec-blacklists src -j DROP

For fail2ban something similar to this

sudo iptables -I FORWARD 1 -p tcp -j f2b-HTTPS

In fact, fail2ban has its own "targets" in iptables, you can find them all with

sudo iptables -L | grep f2b

It will place an entry in iptables before Docker rule that will reject requests before they reach Docker rule (that allows them all)

[madiele]

I only now discovered the issue because I found a MINER inside one of my docker containers that exploited ports of a database service I though were under ufw firewall that was meant to be accessed only via personal VPN.

Why is this a thing, I'm dumbfound...

[zhen-huan-hu]

Yeah, it has been for many years. The lack of official resolution to this security issue is just unbelievable. I guess we have to wait until a high-profile user got hacked because of this.

On Sat, Mar 11, 2023 at 6:58 AM Mattia Di Eleuterio < @.***> wrote:

I only now discovered the issue because I found a MINER inside one of my docker containers that exploited ports of a database service I though were under ufw firewall that was meant to be accessed only via personal VPN.

Why is this a thing, I'm dumbfound...

— Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/690#issuecomment-1464929058, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGX3EOTLNX6JV7DWDBNZULW3SHI3ANCNFSM4HTV7X6Q . You are receiving this because you commented.Message ID: @.***>

[UplandsDynamic]

Yeah, it has been for many years. The lack of official resolution to this security issue is just unbelievable. I guess we have to wait until a high-profile user got hacked because of this.

Being that the Docker maintainers appear so unresponsive to this issue, I'd suggest it might be worth bringing it to the attention of the infosec research community. I recently pinged Steve Gibson, of the well known 'Security Now' podcast. If others do likewise, he may be inclined to take a look and perhaps cover it on the show, to at least shine some publicity on this and alert his audience to the potential risks. His Twitter handle is @SGgrc.

[J4gQBqqR]

I switched to Arch recently. One of the most important reason for doing so is to configure my firewall from ground up without preconfigured ufw in place. Thank you docker for always having me check, double check and triple check my firewall settings.

And I dare not put my server under DMZ thanks to you. I have no confidence in you messing up my firewalls. I am not capable enough to use iptables and even under firewalld I always felt insecure because how you handle ufw. Could you please gain yourselves some credit by cleaning up this mess?

[burntoc]

Part of the problem is there aren't enough tech media screaming about this. I can't remember what I was working on that put this on my radar, but that was a couple of years ago and I still only hear about this via subbing to this issue.

Honestly if we could get this identified as a PUP for this behavior, they'd fix it pretty darn quick I bet.

On Mon, Mar 13, 2023 at 6:45 PM J4gQBqqR @.***> wrote:

I switched to Arch recently. One of the most important reason for doing so is to configure my firewall from ground up without preconfigured ufw in place. Thank you docker for always having me check, double check and triple check my firewall settings.

And I dare not put my server under DMZ thanks to you. I have no confidence in you messing up my firewalls. I am not capable enough to use iptables and even under firewalld I always felt insecure because how you handle ufw. Could you please gain yourselves some credit by cleaning up this mess?

— Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/690#issuecomment-1467216499, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZWF2KGJI54APNISPRUC73W37ER5ANCNFSM4HTV7X6Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[MathewJohn1414]

You can configure Docker to use the ufw firewall by creating a new file /etc/docker/daemon.json if it doesn't exist, and add the following content to it:

{
  "iptables": false,
  "ip6tables": false
}

Then, restart the Docker service using the following command:

sudo systemctl restart docker

This will tell Docker to disable its own iptables rules and use the system's ufw firewall instead.

[g4z]

can you point to any documentation or other source for this information?

[UplandsDynamic]

You can configure Docker to use the ufw firewall by creating a new file /etc/docker/daemon.json

@g4z: Here is the official documentation, FWIW:

Prevent Docker from manipulating iptables
It is possible to set the iptables key to false in the Docker engine’s configuration file at /etc/docker/daemon.json, but this option is not appropriate for most users. It is not possible to completely prevent Docker from creating iptables rules, and creating them after-the-fact is extremely involved and beyond the scope of these instructions. Setting iptables to false will more than likely break container networking for the Docker engine.

Source: https://docs.docker.com/network/iptables/#prevent-docker-from-manipulating-iptables

The entire situation is a steaming hot mess.

This repo is legacy, so doubtful if anyone's actually bothering to check here; the README refers users to the Moby upstream repo, to report issues, here:

https://github.com/moby/moby

Security issues should be reported direct to the team, at: security@docker.com

[akerouanton]

I created a discussion about what's going on with custom iptables rules (whether created through ufw or manually) and what we can do to improve that. It's available here:

• https://github.com/moby/moby/discussions/45524

[chrisgilbert]

I just discovered this issue on Rocky Linux 8, using portainer to setup docker compose . I've been using docker for years professionally, but only either on AWS, my mac, and behind other external firewalls. I discovered it when setting up a new dedicated server for Nextcloud and other home stuff.

It seems that the iptables rules break any setup in firewalld, so without manual changes to DOCKER-USER chain you are unable to block docker ports on the internet.

What an atrocious and badly thought through 'feature'. If I'm a professional DevOps Engineer and I didn't realise this after using docker for 10 years or so, I don't know what hope a casual user has.

[matthewblott]

@chrisgilbert as I say above, I'd advise running Docker as rootless. It's fairly straightforward to setup, works with Portainer and gives you peace of mind! Unfortunately some things like Caprover install Docker so it's not always an option but if your setup is pretty vanilla you should be okay.

[arfedulov]

I've just got my database stolen. The database port was supposed to be closed by ufw but it wasn't. Lukilly, the database was not very important.

For those who use docker compose. The following is the solution that seems to work for me. In docker-compose.yml when mapping host ports to container ports specify also ip address part IPADDR:HOSTPORT:CONTAINERPORT. docs . E.g.

"ports":
    - "127.0.0.1:8001:8001"

This will hide host port from the outside world.

[yrro]

I've just got my database stolen.

Good learning opportunity. 😉

• Don't run docker as root unless you need it.
• Don't enable iptables support in the Docker daemon unless you need it.
• Don't publish a port without binding to 127.0.0.1.

There's a more general point too: if you are running a service and don't want to make it available to other networks, check!

[EssentialRecruit]

This is 2023 about to crossover to 2024 and am still experiencing this issue This is a serious security flaw or bug as the case maybe.

Please @docker do something

[dmuensterer]

I've just got my database stolen.

Good learning opportunity. 😉

• Don't run docker as root unless you need it.
• Don't enable iptables support in the Docker daemon unless you need it.
• Don't publish a port without binding to 127.0.0.1.

There's a more general point too: if you are running a service and don't want to make it available to other networks, check!

While I 100% agree with your points made, this is still a confession of failure by docker and waiting for disasters to happen. There is no reasoning as to why docker has to behave like it does. Docker is used by millions and them not being able to patch such a simple design flaw in almost 5 years is just sad.

[arigit]

I ran into this problem with FirewallD as well, to my surprise all containers that had Ports configured to them were directly accessible from the LAN even if I have a zone with supposedly strict rules controlling what inbound connections are allowed to the server.

My workaround to this problem is, for the containers that had ports and I wanted to control access to from the LAN, I switched to using host networking (network_mode: host in docker compose). This gives control back to firewalld. I assume the same workaround applies to ufw.