Support for incoming requests

[chr4]

Your solution seems to work fine for enabling outgoing ipv6 requests using NAT, but I encounter some issues regarding incoming requests. When connecting from another server to a docker-exposed port, the docker container will get the request from the NATed ipv4 address:

Requests from another server

ipv4: nc -4 example.com 25
ipv6: nc -6 example.com 25

Incoming requests

ipv4: connect from otherserver.net[xx.xx.xx.xx]
ipv6: connect from unknown[172.18.0.1]

Is this a known problem? This prevents the service in the docker container to actually know the connecting machines. For some services this is essential (e.g. verifying RDNS records for mail services)

I kind of assumed that it's possible to also implement this using ip6tables e.g. mirroring the FORWARD rules that docker places in iptables.

Can you comment on that? If you think this is doable (and makes sense), I'm willing to invest some time in a pull-request.

[robbertkl]

Hi, thanks for your report. This should happen. With my own containers, I get the original source address. This was indeed 1 of the goals for this project, as I use it with a mail server myself.

Can you tell me what Docker version you're using?

Also, can you try disabling the userland proxy (--userland-proxy=false) and see if this fixes the issue? I believe the ip6tables rules should take priority over the userland proxy, but perhaps somehow this isn't the case in your situation.

[chr4]

Thanks for your response! I'm using Docker version 1.12.1, build 23cf638. Setting --userland-proxy=false results in no ipv6 connections arriving at the target container at all (conncetion eventually times out).

Here's the complete output of my iptables setup, in case it helps. I cleared all other rules besides the one that docker and ipv6nat create:

# iptables -L -v
Chain INPUT (policy ACCEPT 245 packets, 15981 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   26  2697 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            
   13  1774 DOCKER     all  --  any    br-5c8eb6f9de35  anywhere             anywhere            
   13  1774 ACCEPT     all  --  any    br-5c8eb6f9de35  anywhere             anywhere             ctstate RELATED,ESTABLISHED
   13   923 ACCEPT     all  --  br-5c8eb6f9de35 !br-5c8eb6f9de35  anywhere             anywhere            
    0     0 ACCEPT     all  --  br-5c8eb6f9de35 br-5c8eb6f9de35  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 194 packets, 50779 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-5c8eb6f9de35 br-5c8eb6f9de35  anywhere             172.18.0.2           tcp dpt:submission
    0     0 ACCEPT     tcp  --  !br-5c8eb6f9de35 br-5c8eb6f9de35  anywhere             172.18.0.2           tcp dpt:urd
    0     0 ACCEPT     tcp  --  !br-5c8eb6f9de35 br-5c8eb6f9de35  anywhere             172.18.0.2           tcp dpt:smtp

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  br-5c8eb6f9de35 docker0  anywhere             anywhere            
    0     0 DROP       all  --  docker0 br-5c8eb6f9de35  anywhere             anywhere            
   26  2697 RETURN     all  --  any    any     anywhere             anywhere  

# iptables -L -v -t nat
Chain PREROUTING (policy ACCEPT 19 packets, 1255 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6   332 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 6 packets, 332 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 28 packets, 1733 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 28 packets, 1733 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere            
   13   923 MASQUERADE  all  --  any    !br-5c8eb6f9de35  172.18.0.0/16        anywhere            
    0     0 MASQUERADE  tcp  --  any    any     172.18.0.2           172.18.0.2           tcp dpt:submission
    0     0 MASQUERADE  tcp  --  any    any     172.18.0.2           172.18.0.2           tcp dpt:urd
    0     0 MASQUERADE  tcp  --  any    any     172.18.0.2           172.18.0.2           tcp dpt:smtp

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
    0     0 RETURN     all  --  br-5c8eb6f9de35 any     anywhere             anywhere            
    0     0 DNAT       tcp  --  !br-5c8eb6f9de35 any     anywhere             anywhere             tcp dpt:submission to:172.18.0.2:587
    0     0 DNAT       tcp  --  !br-5c8eb6f9de35 any     anywhere             anywhere             tcp dpt:urd to:172.18.0.2:465
    0     0 DNAT       tcp  --  !br-5c8eb6f9de35 any     anywhere             anywhere             tcp dpt:smtp to:172.18.0.2:25

# ip6tables -L -v
Chain INPUT (policy ACCEPT 9 packets, 600 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION  all      any    any     anywhere             anywhere            
    0     0 DOCKER     all      any    br-5c8eb6f9de35  anywhere             anywhere            
    0     0 ACCEPT     all      any    br-5c8eb6f9de35  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all      br-5c8eb6f9de35 !br-5c8eb6f9de35  anywhere             anywhere            
    0     0 ACCEPT     all      br-5c8eb6f9de35 br-5c8eb6f9de35  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 22 packets, 1992 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all      any    any     anywhere             anywhere          

# ip6tables -L -v -t nat
Chain PREROUTING (policy ACCEPT 1 packets, 80 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    80 DOCKER     all      any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 1 packets, 80 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      any    any     anywhere            !localhost            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all      any    !br-5c8eb6f9de35  fd00:dead:beef::/48  anywhere            

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all      br-5c8eb6f9de35 any     anywhere             anywhere       

[robbertkl]

Sorry for my late reply, just noticed you had replied and my GitHub e-mail notification did not come through.

Your results with the userland proxy turned off confirm that the connections you were seeing were indeed proxied by the userland proxy, and therefore had an incorrect source address. Usually the userland proxy would be automatically overruled by the ip6tables rules created by ipv6nat, but looking at your rules it seems they are missing.

From the rules I concluded that you are not running the docker daemon with --ipv6, but instead you have created a user-defined network with IPv6 ("Option B" from my README). I can see that the rules for the network have been created successfully, but the rules for your published ports are missing.

Worried that something might have changed in the latest release, I've been trying to reproduce your issue myself. I've setup a machine with Docker 1.12.1, build 23cf638, and ran the following commands:

docker network create --ipv6 --subnet=fd00:dead:beef::/48 mynetwork
docker run --rm --name ipv6nat -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --privileged --net host robbertkl/ipv6nat
docker run --rm -it --net mynetwork -p 80:80 robbertkl/php
docker exec -it ipv6nat ip6tables -L -v
docker exec -it ipv6nat ip6tables -L -v -t nat

However, with my attempts, I did see the correct rules for the published port, so I'm unable to reproduce your issue.

Can you provide me with the following information:

• The docker run command of 1 of the containers with the published ports
• In what order do you start the containers? Do you first start the ipv6nat container and then the container with published ports, or the other way around?
• Can you run the exact above commands that I've used in my attempt to reproduce the issue and see if that gives the correct ip6tables rules?

The rules that are missing are:

• In table filter, chain DOCKER, there should be an ACCEPT rule for each published port
• In table nat, chain POSTROUTING, there should be a MASQUERADE rule for each published port
• In table nat, chain DOCKER, there should be a DNAT rule for each published port

Perhaps with this information we can find the cause. Also, the output of ip6tables -L is a bit unreadable; the output of ip6tables-save gives you all tables in a better format (docker exec -it ipv6nat ip6tables-save).

[chr4]

I've rebootet my server, killed all docker containers and ran all the commands you've mentioned in the previous post, and the port is correctly forwarded, incoming requests are forwarded correctly.

I think I've tracked down the problem! I've specified the -p parameter (using docker-compose.yml) with "*:25:25", forcing it to only listen on ipv4. I'm not sure why I used this notation, but I remember it was on purpose and probably due to a bug or something in an earlier docker/ docker-compose release. You can't see the difference in the docker ps output (it always dislays 0.0.0.0:25), but it actually works if I instead just use 25:25.

Thanks for your support!

[robbertkl]

Ah, that makes sense. I'll see if I can improve to support your case as well. Thanks for letting me know the cause!

[kiwixz]

Hi, I got the same problem. Unfortunately I'm already specifying ports as "80:80". I got similar output from docker inspect as shown here, so I guess it's not the binding that is wrong...

Could you suggest any way I could dive deeper to find why docker-ipv6nat does not create rules for my published ports ?

[robbertkl]

Hi @kiwixz, you could try running docker-ipv6nat with the -debug flag and also check the output of ip6tables-save to see if your ports are showing up. Also I suggest going through the steps in the README to make sure you've gone through every step. Hope this helps!

[kiwixz]

Thanks for the prompt reply, I indeed scrupulously followed the README and enabled debug, cleanup and retry flags.

Ports (or even container IPs) are not showing up in logs nor iptables output.

logs

ip6nat_1       | 2020/12/08 18:48:20 docker-ipv6nat is running in debug mode
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -j RETURN
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -j RETURN
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t nat -A PREROUTING 1 -m addrtype --dst-type LOCAL -j DOCKER
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t nat -A OUTPUT 1 -m addrtype --dst-type LOCAL -j DOCKER ! -d ::1
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 1 -j DOCKER-ISOLATION-STAGE-1
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 1 -j DOCKER-USER
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 3 -o docker0 -j DOCKER
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 4 -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 5 -i docker0 ! -o docker0 -j ACCEPT
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A FORWARD 6 -i docker0 -o docker0 -j ACCEPT
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t nat -A DOCKER 1 -i docker0 -j RETURN
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t nat -A POSTROUTING 1 -s fd00:1::/80 ! -o docker0 -j MASQUERADE
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t nat -A POSTROUTING 1 -o docker0 -m addrtype --dst-type LOCAL -j MASQUERADE
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o docker0 -j DROP
ip6nat_1       | 2020/12/08 18:48:20 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2

ip6tables-save

# Generated by xtables-save v1.8.2 on Tue Dec  8 19:44:17 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER-USER - [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-USER -j RETURN
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
COMMIT
# Completed on Tue Dec  8 19:44:17 2020
# Generated by xtables-save v1.8.2 on Tue Dec  8 19:44:17 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o docker0 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:1::/80 ! -o docker0 -j MASQUERADE
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d ::1/128 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Tue Dec  8 19:44:17 2020

I use nftables behind the scene, but it's supposedly backward-compatible.

[robbertkl]

What are your:

• docker daemon arguments or network create command
• container create/run command (or docker-compose YAML)

Are you sure it's connecting to the correct Docker socket/endpoint? Can you try running without -retry?

[kiwixz]

I'm pretty sure it's the good endpoint as I have only one dockerd, and it fixes containers' outgoing connections.

Removing -retry does not fix it.

I use docker from debian buster (stable) official repo:

Docker version 18.09.1, build 4c52b90
docker-compose version 1.21.0, build unknown

---

/etc/docker/daemon.json

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:1::/80",
  "log-driver": "journald"
}

---

docker-compose.yml (I removed unrelated stuff)

version: "3.3"

networks:
  nginx:

services:
  ip6nat:
    image: robbertkl/ipv6nat
    restart: always
    network_mode: host
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    cap_add:
      - NET_ADMIN
    command: ["-debug", "-cleanup"]

  nginx:
    build: nginx
    restart: always
    networks:
      - nginx
    volumes:
      ...
    ports:
      - "80:80/tcp"
      - "443:443/tcp"

[kiwixz]

Okay I found the problem. docker-ipv6nat was not recognizing the containers with a custom network, as it seems like they dont support IPv6 by default. Maybe you could add some logs in debug mode telling us why a container is not considered if that is even possible.

The solution is in docker-compose.yml:

version: "2.4"

networks:
  nginx:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "fd00:1:1::/80"

You have to use a compose file of version 2.x as IPv6 is not yet supported in the 3.x (see here)... Also don't forget to use a different IP subnet than the daemon.

[robbertkl]

Great you've figured it out! Thanks for the info and suggestions.