swarm mode publish port behavior and dockerd --ip option

[muradm]

Description

Hard to protect Docker Swarm Mode node publish, port always binds to 0.0.0.0. According to docker service create and Use swarm mode routing mesh there are two options for publishing service --publish <PUBLISHED-PORT>:<TARGET-PORT>. According to dockerd options there is an option: --ip=0.0.0.0 Default IP when binding container ports Changing --ip option for dockerd has no impact on --publish option of docker service create command. Although, there is no explicit indication that swarm mode publish port should obey --ip option of dockerd, implicitly it would be expected.

Steps to reproduce the issue:

1. configure dockerd to start with --ip=127.0.0.1 option specified
2. create docker swarm
3. create docker service with --publish 80:80 option specified

Describe the results you received: Regardless of --ip option in dockerd, published service always gets bound to 0.0.0.0:

tcp6       0      0 :::80                   :::*                    LISTEN      6110/dockerd    

Describe the results you expected: Since --ip=127.0.0.1 option specified in dockerd it is expected that service is bound to 127.0.0.1 like:

tcp       0      0 127.0.0.1:80                   0.0.0.0:*                    LISTEN      6110/dockerd    

Additional information you deem important (e.g. issue happens only occasionally): There are also a number of other issues (like #26696) reported for the similar case where users want to have service --published port bind to specific ip/interface, I didn't find similar point of view as explained above.

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 35
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 34
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge null host
Swarm: active
 NodeID: 50b50557xu7l6fexck2mni7tx
 Is Manager: true
 ClusterID: 3bedltspub354myauu2qpy4o9
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 127.0.0.1
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.678 GiB
Name: lab1
ID: 63ZL:DTO7:J5ZM:EQRI:W6CT:XAMD:OKH7:Q6QB:SX3S:V6MJ:MGBC:BEDT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: muradm
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.): Ubuntu 16.04

[ehazlett]

/cc @mrjana

[vlasbaard]

Description

I'm having the same issue on:

• version 1.12.1
• not using swarm, just dockerd + compose
• running with --iptables=false
• Ubuntu 14.04

Specifying the ip with -p ip:port:port -does- work.

Steps to reproduce the issue:

1. Configure docker with --ip=(some ip, not 0.0.0.0)
2. Check with ps -elf that it is actually running with the specified --ip
3. Start a container with -p someport:someport
4. Check netstat -nlp46 and see docker-proxy is bound to 0.0.0.0 instead of specified IP
5. Start a container with -p someip:someport:someport
6. Check netstat -nlp46 and see docker-proxy is bound to someip

Describe the results you received:

Docker-proxy is bound to 0.0.0.0

Describe the results you expected:

Docker-proxy to be bound to the specified ip

Additional information you deem important (e.g. issue happens only occasionally):

None.

Output of docker version:

(Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 23
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.16.0-53-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 15.57 GiB
Name: misfits
ID: WAXU:RZDY:MLZD:MK4F:XHVT:DHGW:HIWF:ZAIU:I6OR:2GC2:3XUN:IF4L
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 dockerhub.fuga.com
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical machine, dual-homed network (that's why I need specific binding).

[thaJeztah]

The --ip option is only used for the default ("bridge") network, but I see the documentation does not mention that;

(daemon started with --ip 127.0.0.1);

$ docker network create foo
2d81dac46dc39e65fa48fe443456d3b1ab58b20a8c1d11e91c9c952136f33bc5

$ docker run -d -p 80:80 --name default-net nginx:alpine
b23a006fe4cb0844ebfaa8e6a98b0b2f1ce82b70319df14ce61adbe96eed6442

$ docker run -d -p 81:80 --network foo --name foo-net nginx:alpine
7d261225eaa4cd85aff9299031803bb8ebb8185219f75a819189566a1dfe2d90

$ docker ps --format 'table {{ .Names }}\t{{ .Ports}}'
NAMES               PORTS
foo-net             443/tcp, 0.0.0.0:81->80/tcp
default-net         127.0.0.1:80->80/tcp, 443/tcp

@muradm @vlasbaard can you explain the use case; if you don't want the container to be publicly accessible, why are you publishing its ports?

[vlasbaard]

Since the machine is dual-homed, I only want it (actually, any containers on that host) to be available to one of networks.

Of course, I could firewall it shut, but a) firewalling the other network interface will leave me with other (non-docker-related) issues, b) being able to bind-by-default to one specific ip would be cleaner and c) the other services running on that box are already configured with address-specific binding, so "otherwise it works".

Would be great to have an option to set (or make default) the address of the proxy!

[thaJeztah]

Binding to a specific port/interface is an open feature request in https://github.com/docker/docker/issues/26696. We've been discussing that, but it will need a proper design (getting the UX right for swarm mode can be challenging, as one may not know the IP-addresses in advance).

[muradm]

@thaJeztah, I already went through conversations I could find on this subject while trying to solve my problem. Concerns illustrated there are mostly depend on the issues like "ip address is not known in advance, since docker service launched in swarm mode will end up on multiple docker servers", some discussions go deep into service with "how do we know which ip address to get to which service" etc. I think that questions raised there should not be applicable to docker swarm setup, since if one decides to go with docker swarm service, has to accept that service will run on multiple hosts with different ip addresses. I.e. trying to attach service / service instance to specific IP address somewhat contradicting with docker swarm service concept, and probably wont be solved on per service / service instance basis.

However, here I join @vlasbaard, we are trying to solve infrastructure related problem in multi-homed environment. At installation time it is being decided once which interface / ip address to use regardless of number and type of services running in setup. Once docker binds to 0.0.0.0 there is nothing can be done to control it.

For instance, at installation time I could tell dockerd start with --ip , spin-up all necessary services, and then put haproxy near it to bridge / load-balance services accessible between and .

Of course, there are always solutions to overcome it. But they feel like workaround in my opinion.

[dfyx]

This would be another solution for the problem I described in #28469

I have a web server with two public IP addresses: one is used for non-docker traffic (e.g. SSH access to the host) and one for a docker node (e.g. SSH to a dockerized GitLab instance). Right now I have to do some trickery with non-standard ports and socat but that just doesn't feel right.

Setting a public IP address per node (instead of per service) would be enough for my use case.

[dfyx]

Another idea how to solve this per service would be to define multiple ingress networks which each can be restricted to ip addresses per node. The default would be the existing network that listens on all addresses on all nodes but for special cases one could add custom networks that default to no publich addresses at all and only start working when you add some public addresses. Something like the following:

# Let's assume that all nodes have the whole matching 10.0.x.0/24 subnet as their public addresses
node1 $ docker network create --driver ingress my-restricted-ingress
node1 $ docker network ip my-restricted-ingress add 10.0.1.1/32
node2 $ docker network ip my-restricted-ingress add 10.0.1.2/32

# Containers that connect to a non-standard ingress network automatically get disconnected from the default one
node1 $ docker service create --name restricted --network my-restricted-ingress --network some-other-network -p 80:80 my-image

[patran]

Definitely provide an option to have docker networks (ingress, etc) to be formed on a specific set of network interfaces or even just 1 specific network interface.