Docker-proxy process is listening on IPv4 and IPv6 but only forwarding IPv4 traffic

[beamerblvd]

• [x] This is a bug report
• [ ] This is a feature request
• [x] I searched existing issues before opening this one

I apologize if I'm reporting this against the wrong project. There are lots of different projects to report against, and it's possible I got it wrong. Event though I'm using Compose, I'm certain this is not a Compose bug. Compose appears to be doing all the right things. The bug appears to be in the docker-proxy process.

I'm using Docker Compose to run a DNS server (PowerDNS) within a container. Here is the config:

version: "2.4"
networks:
  dnsnet:
    driver: bridge
    driver_opts: 
      com.docker.network.bridge.name: "dnsbr0"
    ipam:
      driver: default
      config:
        - 
          subnet: 192.168.193.0/24
          gateway: 192.168.193.1
  power-dns:
    image: "my_image"
    restart: on-failure
    networks:
      dnsnet:
        ipv4_address: 192.168.193.170
    ports:
      - "x.x.x.x:53:53/tcp"
      - "x.x.x.x:53:53/udp"
      - "aaaa::ffff:53:53/tcp"
      - "aaaa::ffff:53:53/udp"

The virtual machine is assigned the IPv4 address x.x.x.x and the IPv6 address aaaa::ffff.

I can ping both x.x.x.x and aaaa::ffff from another machine in the same datacenter (~1.5ms round trip) and another machine in a different datacenter (~70ms round trip), so the addresses are unquestionably routable.

Expected behavior

I should be able to dig @x.x.x.x and dig @aaaa::ffff and receive responses without issue from any machine in the world that is connected with both IPv4 and IPv6.

Actual behavior

From the host machine (running on Ubuntu 18.04 on DigitalOcean), I can dig @x.x.x.x and dig @aaaa::ffff without issue. From the other machine (bbbb::ffff) in the same datacenter, I can still dig @x.x.x.x, but dig @aaaa::ffff times out. The behavior is the same from a machine in a different datacenter.

The first thing I checked was lsof:

$ sudo lsof -i -n
docker-pr 7258      root    4u  IPv6  97854      0t0  TCP [aaaa::ffff]:domain (LISTEN)
docker-pr 7272      root    4u  IPv4  97877      0t0  TCP x.x.x.x:domain (LISTEN)
docker-pr 7285      root    4u  IPv4  97919      0t0  UDP x.x.x.x:domain 
docker-pr 7290      root    4u  IPv6  98382      0t0  UDP [aaaa::ffff]:domain

That all looks correct. So next I checked a tcpdump, first of a ping:

$ sudo tcpdump -n host "aaaa::ffff"
01:24:36.570272 IP6 bbbb::ffff > aaaa::ffff: ICMP6, echo request, seq 0, length 16
01:24:36.570322 IP6 aaaa::ffff > bbbb::ffff: ICMP6, echo reply, seq 0, length 16
01:24:37.574518 IP6 bbbb::ffff > aaaa::ffff: ICMP6, echo request, seq 1, length 16
01:24:37.574558 IP6 aaaa::ffff > bbbb::ffff: ICMP6, echo reply, seq 1, length 16

And now of a dig:

$ sudo tcpdump -n host "aaaa::ffff"
00:42:03.291922 IP6 bbbb::ffff.51642 > aaaa::ffff.53: 60840+ [1au] A? example.net. (49)
00:42:08.297904 IP6 bbbb::ffff.51642 > aaaa::ffff.53: 60840+ [1au] A? example.net. (49)
00:42:13.301566 IP6 bbbb::ffff.51642 > aaaa::ffff.53: 60840+ [1au] A? example.net. (49)

$ sudo tcpdump -i dnsbr0 -n host "192.168.193.170"
<nothing>

So there doesn't appear to be a reply and, importantly, the host machine DOES receive the packets (no external firewall issues) but the docker-proxy process never forwards the packets on to the container. Note that a dig to the IPv4 address shows up as expected in the dump:

$ sudo tcpdump -n host "x.x.x.x"
00:46:16.129744 IP y.y.y.y.55183 > x.x.x.x.53: 989+ [1au] A? example.net. (49)
00:46:16.131823 IP x.x.x.x.53 > y.y.y.y.55183: 989*- 1/0/1 A 1.2.3.4 (65)

$ sudo tcpdump -i dnsbr0 -n host "192.168.193.170"
00:46:16.129905 IP y.y.y.y.62620 > 192.168.193.170.53: 16666+ [1au] A? example.net. (49)
00:46:16.131569 IP 192.168.193.170.53 > y.y.y.y.62620: 16666*- 1/0/1 A 1.2.3.4 (65)

I also tried doing the ports differently:

    ports:
      - "53:53/tcp"
      - "53:53/udp"

This resulted in a different (and expected) lsof output but the same behavior and tcpdump results.

$ sudo lsof -i -n
docker-pr 6982      root    4u  IPv6  95863      0t0  TCP *:domain (LISTEN)
docker-pr 6995      root    4u  IPv6  95894      0t0  UDP *:domain 

Steps to reproduce the behavior

I believe everything I've provided above already establishes replication processes, but I'm happy to provide more information if needed.

Output of docker version:

$ sudo docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

$ sudo docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 8
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-43-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: lon1
ID: X7OD:RCPC:RRU3:UJNU:77WC:K6M6:UBVU:4FNF:CAGN:RUI6:W4AY:45J3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

DigitalOcean droplet, Ubuntu 18.04 base system, 2 CPUs, 2GB RAM (I know this is tiny, but I'm only running two tiny containers right now. I will resize this larger as I need more containers)

I have worked around this bug by taking a different approach: Enabling IPv6 with a subnet in /etc/docker/daemon.json, assigning a subnet to the dnsnet network, assigning a static public IPv6 address to the container, enabling proxy_ndp on eth0, and adding IPv6 neighbor proxying (ip -6 neigh add proxy) to eth0 for the static IPv6 address, but this is arduous, fragile, and not an ideal solution. Just a workaround.

[spgyip]

Maybe the problem is the docker-proxy being a L4 proxy, it can only handle ipv4-to-ipv4 or ipv6-to-ipv6 proxy.

So, it can't forward your ipv6 request to a ipv4 backend.

[beamerblvd]

That might be the case—I don't know. If that's the case, then 1) the documentation should say so and say how to configure it properly, and 2) docker-proxy should fail to start if you don't have it configured properly.

[citrin]

Proxying ipv4-to-ipv6 and vise versa is useful functionality and it should not be hard to implement.

Currently I use nginx which listens on IPv6 address and forwards request to IPv4 address used by docker-proxy (to proxy traffic inside IPv4 container), but it would be nice to have ability to do the same without additional proxy layer (and accept v6 connections directly by docker-proxy).

[avollmerhaus]

I've hat to work around this using haproxy.

[vlada-dudr]

Having same issue

[omawnakw]

I think, at least docker proxy should not listen on ipv6 (should not bind to both v4 and v6) when in dockerd global configuration "ipv6" set to "false" and should only bind ipv4. When you have these two lines in /etc/hosts on the host machine (which is default for almost any IPv6-enabled linux system):

127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
::1     localhost localhost.localdomain localhost6 localhost6.localdomain6

gethostbyname defaults to resolve to IPv6, so then curl http://localhost:8080/ (for container with proxy port 8080) will be timed out.