search

docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
33.98k stars 5.23k forks source link

iptables config is missing a rule for accessing container from inside the host #9131

Open LouWii opened 2 years ago

LouWii commented 2 years ago

Description

I've setup iptables with strict DROP rules. It looks like Docker is missing iptables configuration for local container access. I can access the container from outside the server, but not from inside.

Steps to reproduce the issue:

  1. Install docker and docker-compose
  2. Setup iptables to drop all OUTPUT packets, but keep some of the required ones to keep your server running. Here's my basic script (not all rules are needed, be careful to not lock you out). 
    
#!/bin/sh

Reset rules

sudo iptables -t filter -F sudo iptables -t filter -X

Log all packets

sudo iptables -t filter -A INPUT -m conntrack --ctstate NEW -j LOG --log-prefix='[iptables_input] '

sudo iptables -t filter -A OUTPUT -m conntrack --ctstate NEW -j LOG --log-prefix='[iptables_output] '

Block all traffic

sudo iptables -t filter -P INPUT DROP sudo iptables -t filter -P FORWARD DROP sudo iptables -t filter -P OUTPUT DROP

Allow loopback connections

sudo iptables -t filter -A INPUT -i lo -j ACCEPT sudo iptables -t filter -A OUTPUT -o lo -j ACCEPT

Allow established and related connections

sudo iptables -t filter -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT sudo iptables -t filter -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

ICMP (Ping)

sudo iptables -t filter -A INPUT -p icmp -j ACCEPT sudo iptables -t filter -A OUTPUT -p icmp -j ACCEPT

Allow incoming SSH (connect to the server)

sudo iptables -t filter -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT sudo iptables -t filter -A OUTPUT -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

Allow outgoing SSH (connect from the server)

sudo iptables -t filter -A OUTPUT -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT sudo iptables -t filter -A INPUT -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

Allow HTTP and HTTPS

sudo iptables -t filter -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT sudo iptables -t filter -A OUTPUT -p tcp -m multiport --sports 80,443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Git

sudo iptables -t filter -A OUTPUT -p tcp --dport 9418 -j ACCEPT sudo iptables -t filter -A INPUT -p tcp --dport 9418 -j ACCEPT

2. Run `sudo docker restart` to force Docker to re-init all its `iptables` rules.
3. Create a docker-compose file like so:

version: "2.1" services: apach: image: httpd container_name: apach volumes:

Describe the results you received: Curl fails with a timeout.

Describe the results you expected: Curl returns the page as you would expect.

However, when I try to access that container from outside the server with server_public_ip:8113, that works perfectly fine.

Running sudo iptables -A OUTPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT fixes the problem.

But I think that if Docker sets up the iptable rule to make the container accessible from outside, it should also do the same so that we can access the container from the inside.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

docker-compose version 1.29.2, build 5becea4c

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.13.0-27-generic
 Operating System: Ubuntu 21.10
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.32GiB
 Name: ns508203
 ID: 24OY:6YEY:JXWC:K3K6:4VLJ:EJSP:C2ZB:MMQH:G46B:CKU4:WYNU:2J54
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

LouWii commented 2 years ago

I've kind of nailed it down to this rule to fix the problem without compromising security. This allows a docker interface to make new connections to itself. I've got the ip from ifconfig.

iptables -A OUTPUT -p tcp -s 172.19.0.0/24 --destination 172.19.0.0/24 -m conntrack --ctstate NEW -j ACCEPT

I still believe that's something Docker should automatically create in iptables. If Docker automatically creates a rule to make a Docker container accessible from outside, it should also create a rule to make Docker accessible from the inside...? Because having to manually add a rule for each container is a bit of a pain.

or

Maybe the problem is actually Docker exposing my containers to the outside where I specifically have rules in iptables to not expose anything beside what I whitelist. Having to add my own rules to make my containers work make sense, as I am the one who setup iptables to be strict, and Docker is circumventing that.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

paslavskyit commented 2 years ago

hey, I have a similar issue, please stay in touch about this one on how you've fixed it

LouWii commented 2 years ago

I currently just run the iptables command to manually add the rule to make everything work properly. Looks like nobody really cares here.

gabrielfpvh commented 2 years ago

I found the same behaviour.

Output of docker info:

Containers: 41
 Running: 28
 Paused: 0
 Stopped: 13
Images: 28
Server Version: 18.09.3
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
 NodeID: qhugret8km7yrxicdytjgbdt2
 Is Manager: true
 ClusterID: m4j8dyapep27sj7fecic0ha6f
 Managers: 1
 Nodes: 1
 Default Address Pool: 10.0.0.0/8
 SubnetSize: 24
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: ####HIDDEN####
 Manager Addresses:
  ####HIDDEN####:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-193-generic
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.616GiB
Name: ####HIDDEN####
ID: Q2GA:57YK:FPKI:JGSH:7TS6:Y2CW:A4PE:VC22:YK7O:E43H:JEG4:E3R6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

I guess it may be related to a failure on executing the iptables rule. Does docker trace any log to check that?

stale[bot] commented 2 years ago

This issue has been automatically marked as not stale anymore due to the recent activity.

gabrielfpvh commented 2 years ago

Other issue with similar feedback is proposing that it is a moby issue: https://github.com/docker/compose/issues/9833#issuecomment-1244566793

It might be true due to the amount of open issues that include iptables: https://github.com/moby/moby/issues?q=is%3Aissue+is%3Aopen+iptables

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

oliv3r commented 1 year ago

I seem to see the same issue, but then 'the other way around' or rather, contacting the gateway/host from within the container fails with -P INPUT DROP as default policy. So instead, I'll have to use -P INPUT ACCEPT; -A INPUT -i <internet iface> -j DROP which is 'good enough' but cumbersome

stale[bot] commented 1 year ago

This issue has been automatically marked as not stale anymore due to the recent activity.

Kitt3120 commented 5 months ago

I have the same issue.

My server has a public domain that ends up at an nginx reverse proxy. That proxy forwards to a docker container that has a REST API running.

I can curl that REST API from any machine on the internet and from the host itself. But if I fire up another container, like a debian:stable-slim, I can't curl it. It times out. As soon as I disable iptables or nftables (tested with both, both not working), it works.

To fix this for now, I will just put both of those containers into a docker network and curl against the container name instead. But that's not how my application should work :/ It now bypasses my reverse proxy restrictions.

Edit: A friend of mine was able to replicate this on his Proxmox. He tried to curl his Vaultwarden server through his public domain and nginx reverse proxy from inside his Nextcloud container. It works if he disables nftables on his Proxmox host. If nftables is enabled, it times out.

Same problem described here in 2019: https://community.traefik.io/t/curl-from-container-a-to-b-blocked/1040/14

I made a little visualization. The diamond shaped entities are Docker containers.

Docker missing iptables_nftables rule