Open flobernd opened 3 years ago
I can confirm that. We've found out this is happening after our server was blocked by Hetzner for sending packets with invalid source IP.
We have the same issue most likely, running the container in a LXC.
In my case problem appear on Ubuntu 20.04 (on 18.04 no problems) and only with one output interface: tun (VPN).
tcpdump problematic output: 09:00:15.224652 IP 192.168.1.93.52672 > 10.7.x.x.80: Flags [S], seq 3396909146, win 64240, options [mss 1460,sackOK,TS val 1825245854 ecr 0,nop,wscale 7], length 0 (source IP from eth0 interface instead of tun)
On other computer tcpdump output (working scenario): 09:04:35.061067 IP 10.50.x.x.60280 > 10.7.x.x.80: Flags [S], seq 628596480, win 64240, options [mss 1460,sackOK,TS val 1018754195 ecr 0,nop,wscale 7], length 0
Same here from traffic coming from a VPN tunnel inside docker and exiting on the local LAN using Ubuntu LTS 18.04 and latest docker.
Same, just noticed bridged IP addresses in my firewall this morning.
I can confirm that. We've found out this is happening after our server was blocked by Hetzner for sending packets with invalid source IP.
Same here.
This is a huge issue and is able to be remotely exploited as a DoS in some cases. We had a very short incoming ddos attack causing the machine to send out these packets which ended up locking our machine for a couple of hours.
A remotely triggerable DoS like this should really be fixed.
Sadly, this issue exists for over 7 months now and I saw similar reports from other people which were created even years ago.
No response from any "official" source yet. That's kinda disappointing.
We are also having the same issue running on multiple different types of hosts.
I can confirm it's happening with overlay
network too
Confirmed. bridge network leaks with container's inner IPs outside. Trying to bypass it by additional SNAT iptables rule, but withous success.
Confirmed. bridge network leaks with container's inner IPs outside. Trying to bypass it by additional SNAT iptables rule, but withous success.
Untested, but a iptables rule that a friend of mine gave that should fix this:
iptables -I FORWARD -m conntrack --ctstate INVALID -j DROP
that should fix this
this rule just drops packets with problems, but not fix origin of them.
I'm seeing this happening with Docker running on Ubuntu 20.04 LTS as well... In my case, it's the Ubiquiti "UNMS" / "UISP" controller system (which uses a bunch of docker containers).
It has several bridge
networks defined, and I know that at least one of them leaks IPs to the physical network based on the packets logged in some firewall rules...
docker network ls
NETWORK ID NAME DRIVER SCOPE
fdfe7232dd05 bridge bridge local
29632a79e61c host host local
bf6196de91d9 none null local
53872b1005fc unms_internal bridge local
ff4a7fb8e469 unms_public bridge local
I am seeing mostly traffic from 172.18.251.5, source port 443, destination port random, different IPs across my internal network (point to point wireless gear that UNMS/UISP is managing).
docker network inspect unms_public
[
{
"Name": "unms_public",
"Id": "ff4a7fb8e4690d4218b0e5c74acb40c839b71d5cea92518952abe4beaa293ccf",
"Created": "2021-08-16T19:02:49.179292875-04:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.251.1/25",
"Gateway": "172.18.251.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"149251efd6d881b7f89dfcc81b69da5e9f297f5f2d6d157bccb65f5f67b3476c": {
"Name": "unms",
"EndpointID": "8f1ab646e832bfe4258a9056e31729c9b8b2b0b7960fedeb070c142f54a4ca28",
"MacAddress": "02:42:ac:12:fb:03",
"IPv4Address": "172.18.251.3/25",
"IPv6Address": ""
},
"15af5d8659cf4f1e8010933449e77c04462895a344196de37d88b4d789f23d54": {
"Name": "ucrm",
"EndpointID": "e1de09780b56cd16e279e5de5f0d93f4ff8c44615cea77e007ee1fc5e9382ee4",
"MacAddress": "02:42:ac:12:fb:06",
"IPv4Address": "172.18.251.6/25",
"IPv6Address": ""
},
"43074d6bae500c67b3eaae3899d8f568b75619e6d2fcd82573e99009540827eb": {
"Name": "unms-netflow",
"EndpointID": "6ebb76d9954450baeef4256d73619b578b4b24cbbcc304e88ccb7c2271dbc189",
"MacAddress": "02:42:ac:12:fb:04",
"IPv4Address": "172.18.251.4/25",
"IPv6Address": ""
},
"c341c3f5bc8736338bbe837eb6221046aaf9e92494eaf08e281b56fc032b92e4": {
"Name": "unms-nginx",
"EndpointID": "7588f6708b4e8818cbf6e3d6ab86c35647fc6ef44fa78bedb15ee5d00f480e97",
"MacAddress": "02:42:ac:12:fb:05",
"IPv4Address": "172.18.251.5/25",
"IPv6Address": ""
},
"f29ad48117e1304a051771931fb11a918998976053ac6dbbe9e411a421be3bec": {
"Name": "unms-fluentd",
"EndpointID": "1bcda7f1d1b745ff846600f46823ffb88a09095e79a4dc852a8972db5e527d76",
"MacAddress": "02:42:ac:12:fb:02",
"IPv4Address": "172.18.251.2/25",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Server: Docker Engine - Community
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:53 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Can confirm that bridged IP addresses show up in the log of my firewall...
Details of the host running the docker containers:
$ docker version
Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:03:17 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:23 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ cat /etc/debian_version
11.4
This is a legacy project, if you have a look at the readme. Odd, that they keep this legacy project open. The bug should be reported to the moby project?
@karniemi Well, I reported that issue nearly 2 days ago. The notice in the readme is new (12 days ago).
I re-created the issue in the new repository, but judging from 4000 open issues, they seem to have just as nice community management as in this repo. So maybe in 10 years we will get an official answer 🙃
@flobernd ... you meant 2 years, not 2 days :-). And yes, I did notice, but I wanted to point out why there's propably no progress. And further, I'm seeing the same problem, that's why I'm concerned about it.
sudo ufw deny out from any to 172.16.0.0/12
to stop the bleeding so that Hetzner will stop blocking my server IP.It's "funny" that it's still not fixed after 3 years.
This problem is still painful...
Hi all, we have same problem with Docker on Linux both Centos7 and Ubuntu 22.04 with docker 25.0.x and 26.0.x with containerd 1.6.28 tons of FIN packet from container to remote host are sent without masquerading source ip. This causa tons of packet drop on our firewall and connection that isn't closed well. Below an example of firewall log
Jun 12 14:22:43 FIREWALL-PKTLOG: 1dca0e40 INET match PASS 13390 OUT 52 TCP 10.159.0.5/46366->172.22.26.28/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 1dca0e40 INET match PASS 13390 OUT 52 TCP 10.159.0.5/46036->172.22.26.28/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 06055e16 INET match PASS 13390 OUT 52 TCP 10.159.10.5/48444->172.22.26.26/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 1dca0e40 INET match PASS 13390 OUT 52 TCP 10.159.0.5/37394->172.22.26.26/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 1dca0e40 INET match PASS 13390 OUT 52 TCP 10.159.0.5/49457->172.22.26.27/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 06055e16 INET match PASS 13390 OUT 52 TCP 10.159.10.5/56342->172.22.26.26/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 06055e16 INET match PASS 13390 OUT 52 TCP 10.159.10.5/45235->172.22.26.28/443 FA
Jun 12 14:22:43 FIREWALL-PKTLOG: 06055e16 INET match PASS 13390 OUT 52 TCP 10.159.10.5/33241->172.22.26.28/443 FA
Jun 12 14:22:44 FIREWALL-PKTLOG: ddbfe6a8 INET match PASS 13390 IN 40 TCP 10.151.30.4/42554->172.22.26.103/6551 R
our rule that drop this kind of traffic has 300 hit per second
Expected behavior
Docker containers using only the
bridge
network should not send any packets with internal IP addresses to the outside.Actual behavior
Docker containers using the
bridge
network sometimes send packets from the internal (172.17.0.X
) IP to the network interface without masquerading them.Steps to reproduce the behavior
Run a docker container of your choice (in my case
portainer/portainer-ce
) using the defaultbridge
network. Inspect outgoing traffic usingtcpdump
(e.g. on the router device).Related:
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.)
In my example the docker container is running on a Debian VM which is running on a VMware ESXi host.
First noticed the leaked IP addresses in the "client overview" of my networking hardware (Ubiquity UniFi). This list shows the currently assigned IP address for each connected client. For all VMs running a docker container with
bridge
network this IP from time to time is changed to172.17.0.X
for some seconds until it switches back to the correct value.