search

cmulk / wireguard-docker

Wireguard setup in Docker meant for a simple personal VPN
345 stars 92 forks source link

Docker service DNS resolution breaks #19

Closed ghostserverd closed 4 years ago

ghostserverd commented 4 years ago

Hey there. Firstly, thanks for building this container. It's been very informative as I build out VPN connected services.

I have an issue to submit regarding DNS resolution for other docker services in the same network.

Some Background

As you likely know, when multiple containers are running in the same docker network, they are accessible via <container_name>:<port> as opposed to having to connect to each container by its IP address. This is accomplished by docker's internal DNS server which is accessible from within each container at address 127.0.0.11.

I believe that because wg-quick rewrites /etc/resolv.conf, this wireguard container breaks docker DNS resolution from within the container, and from within any container that uses this wireguard container as a network service.

I have an example docker-compose that shows this in action:

version: "3"
services:
  wireguard:
    image: cmulk/wireguard-docker:buster
    container_name: wireguard
    volumes:
      - /config/wireguard:/etc/wireguard
    restart: unless-stopped
    privileged: true
    sysctls:
      - "net.ipv6.conf.all.disable_ipv6=0"
      - "net.ipv6.conf.default.forwarding=1"
      - "net.ipv6.conf.all.forwarding=1"
      - "net.ipv4.ip_forward=1"
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    networks:
      default:
        aliases:
          - watismyip_vpn
    ports:
      - 4444:4444
    volumes:
      - /server/config/wireguard:/etc/wireguard
      - /lib/modules:/lib/modules

  watismyip_vpn:
    image: garrettsparks/watismyip
    container_name: watismyip_vpn
    restart: unless-stopped
    network_mode: "service:wireguard"
    depends_on:
      - wireguard
    environment:
      - PORT=4444

  watismyip_clear:
    image: garrettsparks/watismyip
    container_name: watismyip_clear
    restart: unless-stopped
    environment:
      - PORT=5555
    ports:
      - 5555:5555

  watismyip_clear_2:
    image: garrettsparks/watismyip
    container_name: watismyip_clear_2
    restart: unless-stopped
    environment:
      - PORT=6666
    ports:
      - 6666:6666

garrettsparks/watismyip is just a small container I found which queries some external services for the current public IP address (with a fallback to the private IP if the container cannot contact any of the external services) of the container.

There are three services along with the wireguard service. watismyip_vpn uses the wireguard container as its network. Then watismyip_clear and watismyip_clear_2 are just normal containers not using the wireguard network.

The goal is that watismyip_vpn can call watismyip_clear:5555, watismyip_clear can call watismyip_vpn:4444 and watismyip_clear can call watismyip_clear_2:6666.

The last two scenarios work as expected. The two clear containers can talk to each other just fine using their service names

/ # curl watismyip_clear_2:6666
<non_vpn_public_ip>

and the clear containers can talk to the watismyip_vpn container via its service name because of the network alias set up in the wireguard container.

/ # curl watismyip_vpn:4444
<vpn_ip_address>

However, when I attempt to call watismyip_clear:5555 from within the watismyip_vpn container, I get a DNS resolution failure

/ # curl watismyip_clear:5555
curl: (6) Could not resolve host: watismyip_clear

I believe that the root cause of this is that /etc/resolv.conf is re-written by wg-quick. For example, in the non-vpn containers, this is /etc/resolv.conf

/ # cat /etc/resolv.conf
search localdomain
nameserver 127.0.0.11
options ndots:0

vs in the watismyip_vpn container

/ # cat /etc/resolv.conf
# Generated by resolvconf
nameserver <vpn_dns_server_address>

which makes perfect sense because the wireguard configuration specifies a different DNS address to force DNS to the VPN's DNS resolver which avoids leaking your DNS queries.

The Question

My kludgy workaround is to mount docker.sock, do a docker network inspect on the docker network, and then write /etc/hosts with each of the service's docker network IP address, but I'd prefer to find a way to do this without having to mount docker.sock, and with this workaround, if the service IPs ever change, the wireguard container would have to be restarted to pick up the new IP address.

I believe this should be possible with a dnsmasq config that's something like

strict-order
server=/watismyip_clear/127.0.0.11
server=127.0.0.11@eth0

and updating /etc/resolv.conf to be

# Generated by resolvconf
nameserver 127.0.0.1
nameserver <vpn_dns_server_address>

but that didn't work, and I'm not a dnsmasq expert unfortunately.

Do you know of a way to retain support for docker DNS resolution while also using the VPN's DNS resolution for public DNS queries with this wireguard container?

Sorry for the wall of info and thanks again!

cmulk commented 4 years ago

Thanks for the kind words and your detailed explanation! This is also a good learning for me as I have not used the network_mode: service before and I like what you are doing here.

What you need is a conditional forwarder, which requires a local dnsserver in the same container (or probably better another docker service) to serve the DNS requests and point them to the right place (dnsmasq should work as you mentioned with some tweaks).

So here is my (untested) idea: First configure all your services to use a local domain name (like 'local') via the aliases. e.g.:

networks:
     default:
        aliases:
        - watismyip_clear.local

Then set up a dnsmasq container (could also share the network with the wireguard service) with conf like so:

#dont use hosts nameservers
no-resolv

# listen specifically on 127.0.0.1
listen-address=127.0.0.1

# send queries for local domain to docker
server=/local/127.0.0.11

# send queries for all else to upstream dns
server=9.9.9.9

Then in your wg0.conf, DNS=127.0.0.1, which will set your resolv.conf to

# Generated by resolvconf
nameserver 127.0.0.1

So what will happen is that all the DNS queries will go to your dnsmasq server. The dnsmasq server will in turn forward queries for the .local domain names to docker for resolution, or do the default upstream for everything else.

Definitely adds some complexity that is not ideal but I think it will work. Unfortunately it does not look like there is a way to do conditional nameservers in resolv.conf directly or it would make this a lot simpler.

ghostserverd commented 4 years ago

Ah that's a good tip about setting a local TLD.

I set up dnsmasq in the wireguard container, and set the alias with the .local TLD. I can

curl watismyip_clear.local:5555

from within the watismyip_vpn container which is super neat!

Now I just need to do some digging to see if dnsmasq can support an address without a TLD (might not be possible, but it would be nice to not have to alias each container).

I did have to disable the MASQUERADE rule in order for it to work. With the rule in place, even when I manually specified the DNS address (e.g. dig @127.0.0.11 watismyip_vpn.local) the request times out.

What exactly is that MASQUERADE rule for? Is it just masking the docker subdomain in outgoing packets? Or is there more to it than that? Is it safe to disable?

Thanks for your help!

cmulk commented 4 years ago

The MASQUERADE rule is really meant for the "server" side so it is ok for you to disable in this case. It does a NAT for all outbound traffic to the Internet (out eth0). Technically I'm not sure why it would have an effect on the local DNS lookups to 127.0.0.11, but docker does some interesting NAT stuff of its own for that built-in DNS so there may be some interference there.

ghostserverd commented 4 years ago

All right. I've made some progress.

I updated my fork of this repo to do the following:

  1. Scrape the wireguard interface file for its DNS server and write /etc/dnsmasq.conf with that as the final fallback DNS address.
  2. If docker.sock is mounted, write /etc/hosts with any service in the network (this is really just for backwards comptability)
  3. If LOCAL_TLD is set (e.g. to local) write /etc/dnsmasq.conf to use 127.0.0.11 for that TLD. Also, write /etc/resolv.conf to search $LOCAL_TLD so that containers can access the addresses of the services without having to know to append .local to match the rule in /etc/dnsmasq.conf. This will require aliases with the TLD in each of the containers that need to be accessible from within the wireguard network.
  4. If SERVICE_NAMES is set (a list of services to make available from within the wireguard network), write each service name individually to /etc/dnsmasq.conf to force 127.0.0.11 as the DNS server for each service address. This is nice because you don't have to write an alias for each service to make available, but you do need to list out all of the services anyway.

The last three are really mutually exclusive. Only one of the mechanisms should be used. Of the last two, I'm not really sure which one I prefer, but they're both better than mounting docker.sock in my opinion.

Thanks for your help on this! I'm going to resolve this issue now since I think I've come up with acceptable mechanisms for making docker DNS resolve from within containers using the wireguard network without having to mount docker.sock.