search

hashicorp / docker-consul

Official Docker images for Consul.
Mozilla Public License 2.0
397 stars 238 forks source link

Cannot bind to port 53 for DNS with host network mode #15

Closed marcuslinke closed 8 years ago

marcuslinke commented 8 years ago

The documentation mentions the possibility to map the DNS port:

docker run -d --net=host -p 53:8600/tcp -p 53:8600/udp consul

But when using --net=host the port mappings are not used by docker. So what is the recommended way to use the default DNS port? I've tried with

-e 'CONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true, "ports": {"dns": 53}}'

but this leads to 

==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Error starting dns server: dns udp setup failed: listen udp 10.32.24.16:53: bind: permission denied
marcuslinke commented 8 years ago

Binding of port 53 fails because consul is started as user consul here:

https://github.com/hashicorp/docker-consul/blob/master/0.6/docker-entrypoint.sh#L78

pauzed commented 8 years ago

Run the container in bridge mode if you want to forward port 53 -> container, or bind your container dns port to 8600, and use ipmasq/iptables to forward port 53 from the host, to the container.

scyellleader commented 8 years ago

The following text appears in the documentation for the docker image. This would imply that running in bridge mode is not Hashicorp's best practice.

Consul should always be run with --net=host in Docker because Consul's consensus and gossip protocols are sensitive to delays and packet loss, so the extra layers involved with other networking types are usually undesirable and unnecessary.

I also have been trying to reproduce what the documentation says and cannot seem to get the port mappings to work on host network. I have tried multiple configurations and multiple consul bindings settings. Any assistance would be nice. I have also opened a ticket with docker, Request #14732, but have not had a response in 10 days.

I agree that pauzed ipmasq/iptables rule will probably work but I'm not sure if that extra step is needed. From the documentation it seems like it should work with the normal docker run commands.

ryansch commented 8 years ago

It may soon be possible to do this without iptables through ambient capability support: https://github.com/docker/docker/issues/8460#issuecomment-222489432

Until then you can either play with setcap or iptables.

slackpad commented 8 years ago

Sorry if I led folks astray with the documentation. We originally had some dnsmasq magic baked in and replaced it during the review process with the port mapping, but that may have been before I changed to a non-root user. I'll try to repro what the docs say and update them if they are indeed incorrect.

marcuslinke commented 8 years ago

@slackpad What about an environment variable that controls whether consul is started as root user?

gittycat commented 8 years ago

@ryansch Yes it looks like docker is looking at allowing in the future it but it would be limited to recent kernels. We however need a solution now.

@slackpad I have also spent quite a bit of time on this. It currently can't work as documented (network=host, port fowarding). But it's not just a documentation issue. We need to run consul with host networking and by far the easiest way to achieve this is to run Consul as 'root'. As it stands now, we must either use bridge networking (which you don't recommend) or play around with the host's iptables (which I much rather avoid).

slackpad commented 8 years ago

@gittycat and @marcuslinke I don't think we'd bake an option for running Consul as root into the official image - that's really not a good way to work around this because people shouldn't be running Consul as root. I'd lean towards deploying something like dnsmasq alongside Consul - https://www.consul.io/docs/guides/forwarding.html. That's what we have always recommended to get port 53, and should work fine with Dockerized Consul's port 8600.

gittycat commented 8 years ago

@slackpad You're probably aware already but running a container as root doesn't give root access to the host and other containers. The container is highly isolated. See https://docs.docker.com/engine/security/security/#linux-kernel-capabilities A container running as root has only access to the following whitelisted features: https://github.com/docker/docker/blob/master/oci/defaults_linux.go#L64-L79. The container can be further restricted by removing features from the list. My understanding is that the CAP_NET_BIND_SERVICE allows a container the elevated privilege to listen to port 53.

ryansch commented 8 years ago

@gittycat Beware: https://github.com/docker/docker/issues/8460

gittycat commented 8 years ago

@ryansch I'm not totally clear about the issue you list. It looks like it would be a problem when you start a Consul container with -u someuser --cap-add NET_BIND_SERVICE to allow it to listen to port 53. That doesn't work apparently. I'm instead suggesting that Consul should run as root, and maybe further restricts permissions if needed but keeping NET_BIND_SERVICE.

ryansch commented 8 years ago

@gittycat You're right. Been playing with user namespaces where root != root. Sorry for the confusion.

joonas-fi commented 8 years ago

I've been struggling with port 53 + --net=host, --cap-add=NET_BIND_SERVICE and even desperately --privileged and also tried port forwards.

Now I however understand, that this is because Consul runs inside the container as non-root, which I have to think is a wtf, since apps inside containers are meant to run as root, if I understand the point of containerization correctly (=> container IS the security boundary).

I've now been stung many times with the "zomg root is bad, mmkay" herd mentality in other projects as well. For example the official PHP Docker image with Apache insists on running as www-data which got me back to the chgrp, chmod, messing with user accounts, changing home directory misery/nonsense that I thought this new container movement was supposed to get rid of.

edit: Also, I'd rather not like messing around with iptables, since my current setup is so clean (server only needs Docker running and no other configuration), plus I'd rather not deploy another piece of infrastructure (dnsmasq) just because docs tell me to use --net=host but that apparently rules out port 53 because apparently root is always bad, even inside a container, which I don't understand in this new world.

ryansch commented 8 years ago

It's generally considered a best practice to run any daemon with reduced privileges. Docker engine security is still a moving target (but they're making good progress). They have an article here: https://docs.docker.com/engine/security/security/

Note at the bottom in 'Conclusions' that they say it's preferable to run with non-privileged users.

This is not a problem that docker has specifically; this is the linux security model at work. It has always been a good idea to run a larger daemon with a larger attack surface under a dedicated non-root user and forward ports with either iptables or something like socat. I'm looking forward to the ambient cap support because it will make this entire problem easier to solve with docker.

joonas-fi commented 8 years ago

You're right, Docker guys seem to advise for running as non-root.

Okay, maybe my argument is based on:

But obviously, I'm no expert so maybe my frustration might be misguided.

ryansch commented 8 years ago

I'll address your points:

In our production environment all but one of the containers runs as a non-root user. For developers new to docker this is often overlooked. Note that most of the official (library) images run with non-root users. Note the library best practices includes a gosu example to drop privileges: https://github.com/docker-library/official-images#consistency

In fact it does. If you break out of a container then you have the same privileges on the host as you do in the container. If you're root in the container, now you're root on the host. Furthermore any bug that allows you to write to the filesystem when combined with volume bind mounts can allow a compromised container to write executables to the host system.

I'm honestly not sure what you're driving at here. I'd love some clarification. I haven't found that I needed a root user for most of the things I'm doing with docker.

You have a point here. The learning curve to get your head around capabilities/seccomp/security stuff while also learning about how to rearchitect applications to work in containers is a bit much. I still bump my head on this.

At the end of the day, I would rather a bug in consul that leads to remote code execution be stymied by a non-privileged user. If you truly need root in a container then I would look into user namespacing which makes container root != host root.

stphngrtz commented 8 years ago

... or bind your container dns port to 8600, and use ipmasq/iptables to forward port 53 from the host, to the container.

Can anyone give me a hint on this? This is what I'm trying to do:

$ docker run -d --name=consul --net=host consul agent -server -bind=192.168.99.100 -client=0.0.0.0 -bootstrap -ui

$ dig @192.168.99.100 consul.service.consul
-

$ dig @192.168.99.100 -p 8600 consul.service.consul
;; ANSWER SECTION:
consul.service.consul.  0   IN  A   192.168.99.100

$ docker run -it --rm --dns=192.168.99.100 --dns=8.8.8.8 ubuntu sh -c "apt-get update && apt-get install -y dnsutils && dig consul.service.consul"
-

$ docker run -it --rm --dns=192.168.99.100 --dns=8.8.8.8 ubuntu sh -c "apt-get update && apt-get install -y dnsutils && dig -p 8600 consul.service.consul"
;; ANSWER SECTION:
consul.service.consul.  0   IN  A   192.168.99.100

Edit: I'm running docker on OSX.

ryansch commented 8 years ago

@stphngrtz I need some additional information:

stphngrtz commented 8 years ago

I'm thinking of something like this:

docker run -it --rm --dns=192.168.99.100 --dns=8.8.8.8 ubuntu sh -c "apt-get update \
    && apt-get install -y dnsutils \
    && iptables ... \
    && dig consul.service.consul"

I found this link where someone is doing exactly what I am looking for, i guess. And this link which I haven't read yet, but looks like an read-worthy short explanation of iptables.

stphngrtz commented 8 years ago

Got it!

docker run -it --rm --dns=192.168.99.100 --dns=8.8.8.8 --cap-add=NET_ADMIN ubuntu sh -c "apt-get update \
    && apt-get install -y iptables \
    && iptables -t nat -A OUTPUT -p tcp -d 192.168.99.100 --dport 53 -j DNAT --to-destination 192.168.99.100:8600 \
    && iptables -t nat -A OUTPUT -p udp -d 192.168.99.100 --dport 53 -j DNAT --to-destination 192.168.99.100:8600 \
    && apt-get install -y dnsutils \
    && dig consul.service.consul \
    && apt-get install -y inetutils-ping \
    && ping -c 5 consul.service.consul"
ryansch commented 8 years ago

Awesome! I'm glad you got it figured out. I had weekend LAGGGG. ;-)

gittycat commented 8 years ago

@stphngrtz Your shell script only modifies the iptables in your ubuntu container, not the host. For port 53 to be forwarded to 8600 for all containers we need to edit the host's iptable.

What I take home from the above thread is that modifying the iptables on the host to redirect port 53 to Consul's port 8600 is the least worst option at this point. This is not really a Consul image limitation. It's not clear to me that even running Consul as root would solve the problem since "container root" isn't the same as "host root" and you need to be host root to listen to port 53.

The documentation on the Official Consul Docker image really needs to be updated though. Many developers will waste a lot of time trying port forwarding instructions that don't work.

venkataramanr commented 8 years ago

@slackpad Is there a way to do this now apart from using dnsmasq or iptables ? I still see the docks have the --host=net and -p 53:8600/udp, is this appropriate ?

slackpad commented 8 years ago

There's some good iptables info brewing here which may be useful - https://github.com/hashicorp/consul/pull/2006.

slackpad commented 8 years ago

And a possible fix is in PR here for newer versions of Docker - https://github.com/hashicorp/docker-consul/pull/24.

nhi-vanye commented 8 years ago

I find it kind of depressing that 2months after this was opened that nobody can be bothered to update the documentation.

slackpad commented 8 years ago

https://github.com/docker-library/docs/pull/691

4iAmAve commented 7 years ago

Hi, could anyone help with getting the name resolving issue fixed for MacOS (latest). Can't seem to get consul.service.consul resolved.

Regards

BlinkyStitt commented 6 years ago

Thanks @slackpad ! I was getting this error on Fedora 26 when following the steps in the docs if I started the agent with -dns-port 53:

==> WARNING: Bootstrap mode enabled! Do not enable unless necessary                                                     
==> Starting Consul agent...                                                                                            
==> Error starting agent: agent: timeout starting DNS servers

Adding CONSUL_ALLOW_PRIVILEGED_PORTS didn't change anything. Then I added net.ipv4.ip_unprivileged_port_start=0 to sysctls and then it started!

Interestingly I'm using IPv6 but net.ipv6.ip_unprivileged_port_start=0 doesn't exist and doesn't seem necessary.

Ganesh-DevOps-Eng commented 6 years ago

how can i configure DNS 53 port on 0.0.0.0 ip iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 53 -j DNAT --to-destination 0.0.0.0:53

we have run this command, not show listen 53 on 0.0.0.0 ip

tcp 0 0 127.0.0.1:53 0.0.0.0: LISTEN 7046/named tcp 0 0 192.168.122.1:53 0.0.0.0: LISTEN 1472/dnsmasq tcp 0 0 127.0.0.1:953 0.0.0.0: LISTEN 7046/named tcp 0 0 0.0.0.0:49153 0.0.0.0: LISTEN 1382/samba tcp6 0 0 :::49153 :::* LISTEN 1382/samba #########we have also try listen-on port 53 { any; }; or listen-on port 53 { 127.0.0.1; any; }; or { 0.0.0.; any; } listen-on-v6 port 53 { ::1; any; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { localhost; any; };

how can i configure 0.0.0.0:53 please help me