ipv6nat stop working after docker update

[celogeek]

Hi,

I have got this error since my docker upgrade:

ipv6nat_1              | Try `iptables -h' or 'iptables --help' for more information.
ipv6nat_1              | 2019/05/25 21:20:31 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory

My docker version:

docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:36:01 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false

My nat table:

sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE *** (all my rules here)

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
DNAT *** (all my ipv4 dnat here)
# Warning: iptables-legacy tables present, use iptables-legacy to see them

I have disabled ipv6 for now. Can I do something to fix this?

thanks

[robbertkl]

I'm not sure where those errors come from, ipv6nat runs ip6tables, not iptables. It seems something else is going on here.

I can take a look, but please give me your run command (or docker-compose section) for ipv6nat and output of both iptables-save and ip6tables-save.

[celogeek]

iptables-save:

# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-5c0b4b729dfe -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-5c0b4b729dfe -j DOCKER
-A FORWARD -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -i br-5c0b4b729dfe -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.4/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.5/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.8/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.12/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-5c0b4b729dfe -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 11:16:23 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-5c0b4b729dfe -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.4/32 -d 10.20.0.4/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.5/32 -d 10.20.0.5/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.8/32 -d 10.20.0.8/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.12/32 -d 10.20.0.12/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-5c0b4b729dfe -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.4:993
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.5:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.6:587
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.6:465
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.6:25
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.8:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.10:51415
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.12:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.20:443
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.20:80
COMMIT
# Completed on Sun May 26 11:16:23 2019

ip6tables-save:

# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019

docker-compose.yml:

networks:
  default:
    driver: bridge
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: 10.20.0.0/24
        - subnet: fd00:dead:beef::/48

services:
  ipv6nat:
    image: robbertkl/ipv6nat
    command: ["-retry", "-cleanup"]
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro
    network_mode: host
    cap_add:
      - NET_ADMIN
      - SYS_MODULE

[robbertkl]

And this output is with ipv6nat running? Or did it exit at this time? I'm asking because you're using --cleanup which removes all the rules on exit.

Can you do a docker-compose up -d --force-recreate ipv6nat and after that send me the docker logs?

[celogeek]

I have removed the command section and run the recreate:

ipv6nat_1              | 2019/05/26 11:25:07 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
ipv6nat_1              |
ipv6nat_1              | Try `iptables -h' or 'iptables --help' for more information.

[celogeek]

The ipv6nat wasn't running because it fails at the start with that log.

[robbertkl]

OK, it craps out trying to detect hairpin mode in the IPv4 firewall. Could you try:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat

[celogeek]

sudo docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
# Generated by iptables-save v1.6.2 on Sun May 26 12:14:56 2019
*nat
:PREROUTING ACCEPT [8392:426047]
:INPUT ACCEPT [2635:96038]
:OUTPUT ACCEPT [3987:318688]
:POSTROUTING ACCEPT [9744:648697]
COMMIT
# Completed on Sun May 26 12:14:56 2019

[robbertkl]

Can you try the same but with --privileged instead?

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat

[celogeek]

The output is empty. When I run docker-ipv6nat again:

/ # ./docker-ipv6nat
2019/05/26 15:50:19 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory

[robbertkl]

So when you're running iptables-save from within a privileged, net=host ipv6nat container it shows no IPv4 rules, but when you run iptables-save on the host it does show IPv4 rules?

I've just tried the same on a test host running Docker 19.03.0-beta3, and I still see the IPv4 rules from within the container, so no way to reproduce it on my end.

[celogeek]

Sorry I run the ip6tables command. Here the output:

# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*nat
:PREROUTING ACCEPT [3580:160229]
:INPUT ACCEPT [562:19464]
:OUTPUT ACCEPT [792:64734]
:POSTROUTING ACCEPT [1927:130179]
COMMIT
# Completed on Sun May 26 16:20:42 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*filter
:INPUT ACCEPT [3180:338688]
:FORWARD ACCEPT [50554:39971344]
:OUTPUT ACCEPT [3327:460405]
COMMIT
# Completed on Sun May 26 16:20:42 2019

[robbertkl]

Sorry, I'm a bit confused, this output is different than before again. Can you please post all 3 outputs:

1. iptables-save (on the host)
2. docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
3. docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat

All commands use iptables, not ip6tables. The problem is your ipv6nat instance seems not to see the IPv4 rules, even though they're available on the host. I cannot reproduce this myself, however.

[celogeek]

Despite my provider give me only a /128 block, it seems I can have a /64 block anyway. They are not limiting the ipv6 assignation on my /64 block.

I have reconfigured my docker to handle ipv6 block /80. All my containers have a public IP address now, and it seems to connect to the host with ipv6 on a container port (SMTP for example) redirect properly but in a weird way. HOST ipv6 -> Host docker ipv4 -> Container IPV4

Also, an issue is that they are no ipv6 isolation of the container. If I have a port listing (80) on a container but I don't open it on the host (don't have the "ports"), I can't reach it with ipv4, but I can with ipv6. Also, ports redirect doesn't work well: ports: 2222:22, give ipv4: 2222, ipv6: 22

The ipv6 nat is far easier to configure.

Here the output I have (sorry for the change): 1.

# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a040ef404950 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a040ef404950 -j DOCKER
-A FORWARD -i br-a040ef404950 ! -o br-a040ef404950 -j ACCEPT
-A FORWARD -i br-a040ef404950 -o br-a040ef404950 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.27/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.28/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.15/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.21/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.18/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a040ef404950 ! -o br-a040ef404950 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a040ef404950 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 16:29:11 2019
# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-a040ef404950 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.27/32 -d 10.20.0.27/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.28/32 -d 10.20.0.28/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.15/32 -d 10.20.0.15/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.21/32 -d 10.20.0.21/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.18/32 -d 10.20.0.18/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-a040ef404950 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.27:51415
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.28:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.15:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.19:587
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.19:465
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.19:25
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.21:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.18:993
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.10:443
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.10:80
COMMIT
# Completed on Sun May 26 16:29:11 2019

2.

# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*nat
:PREROUTING ACCEPT [4531:202007]
:INPUT ACCEPT [731:25374]
:OUTPUT ACCEPT [990:81507]
:POSTROUTING ACCEPT [2369:161300]
COMMIT
# Completed on Sun May 26 16:29:40 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*filter
:INPUT ACCEPT [3977:421198]
:FORWARD ACCEPT [85777:73490326]
:OUTPUT ACCEPT [4113:547670]
COMMIT
# Completed on Sun May 26 16:29:40 2019

3.

# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*nat
:PREROUTING ACCEPT [4554:202959]
:INPUT ACCEPT [735:25502]
:OUTPUT ACCEPT [992:81688]
:POSTROUTING ACCEPT [2374:161665]
COMMIT
# Completed on Sun May 26 16:29:56 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*filter
:INPUT ACCEPT [3995:422953]
:FORWARD ACCEPT [86786:74429505]
:OUTPUT ACCEPT [4131:549825]
COMMIT
# Completed on Sun May 26 16:29:56 2019

[celogeek]

well, ipv6 sucks with docker. I will disable ipv6 on my host. It is not really useful. If that evolves, I may reenable it. Or I may look out for an alternative.

[robbertkl]

Yeah, it does suck big time. However, I've been running it with ipv6nat for years without any issues. Not sure what your issue is; your container not being able to reach the (IPv4) firewall rules seems to be the cause of ipv6nat crashing, but why this is happening I don't know. Are you by any chance using SELinux? I can't reproduce it myself, so it's hard to debug.

[celogeek]

I'm on debian/buster, the latest kernel on this system. I also have the latest (non-beta) version of docker/docker-compose. It seems to hide iptables rules indeed inside the container which cause the issue here.

[robbertkl]

Yeah, I'm curious to know what's causing it. Even on 19.03.0-beta3 I could not reproduce it, so it must not be a Docker thing. Perhaps it's something kernel-related.

[celogeek]

Can you test it with debian/buster ?

[celogeek]

Maybe it would be possible and safer to run your companion on the host directly? Something like a service that starts with docker?

[celogeek]

The kernel:

Linux ns363135 4.19.0-5-amd64 #1 SMP Debian 4.19.37-3 (2019-05-15) x86_64 GNU/Linux

[robbertkl]

Running on the host is a great idea, that should work since it should be able to access the firewall there.

I'll try to do some tests with buster / later kernels soon.

[robbertkl]

OK @celogeek, just did 2 clean Debian VM installs (1 stretch @ 4.9.0-9, 1 buster @ 4.19.0-5) and installed Docker stable (18.09.6).

Then tried reading iptables from within a (privileged, net=host) container on both:

• on stretch it worked
• on buster it did not (no rules)

It seems the problem is buster using nftables (see https://wiki.debian.org/nftables). I managed to get it working by switching to legacy version by running:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
reboot

After the reboot, I can see the rules from inside the container again.

[celogeek]

Nice catch!

Did you try directly on the host? without a container?

[robbertkl]

Yes, you can either switch to legacy iptables like above, or run ipv6nat directly on the host. On the host it will use the iptables/ip6tables commands that are linked to the special nftables compatibility versions.

This same approach would also work from within the container (auto translating iptables to nftables), so I'm working on a fix for the Docker image to support this. I just need to figure out how to reliably detect which is used. After I'm done with this fix, you should be able to run it again as a container, without any changes.

[robbertkl]

Hi @celogeek , just released a new version of the docker images with support for nftables, so it should work again with your previous setup, unchanged. Could you test this?

[celogeek]

Yes of course. I will need to reactivate ipv6 on my server. I do that as soon as possible.

I guess you have try in debian/buster/nfptables condition ?

[robbertkl]

Yep, tried it and as far as I could see it starts just fine.

[celogeek]

test done it doesn't work.

here the error I got:

ipv6nat_1              | 2019/05/28 18:50:57 running [/usr/local/bin/ip6tables -t filter -I DOCKER 1 -d fd00:dead:beef::2 ! -i br-f016aef740c8 -o br-f016aef740c8 -p tcp -m tcp --dport 22 -j ACCEPT --wait]: exit status 1: iptables: Invalid argument. Run `dmesg' for more information.

in dmesg:

[  557.022382] x_tables: ip6_tables: tcp match: only valid for protocol 6

Maybe used ip6tables-nft instead?

[robbertkl]

Yeah, I'm seeing this same error now. Seems like a bug in the compat scripts, because we're obviously using protocol 6. Doing an ip6tables-translate did not generate an error.

You're right about ip6tables-nft, this is what I wanted to use, however I'm using Alpine Linux as a base image and their iptables is running behind.

[robbertkl]

Wait, this is great, there's a Alpine ip6tables 1.8.2-r1 package in edge now, built just YESTERDAY 😄

I'll go check it out.

[robbertkl]

I got a bit further by using 1.8.1-r1 and ip6tables-nft, but now I ran into an issue with go-iptables, I reported it here: https://github.com/coreos/go-iptables/issues/59

There's not really a workaround with the current version, so I'll fix the issue to get it working. Should be done soon, I'll let you know. I'll reopen this issue.

[robbertkl]

@celogeek Could you please test again with the latest release? Thanks!

[celogeek]

yes sure. testing it right now.

[celogeek]

It works !!! congrat.

Everything is back to normal now.

I will take a vps with ipv6 connectivity and play with real ipv6 config (no nat). It seems to be not so easy to do a firewall with docker / ufw / ipv6.

This would be a good challenge ;)

A very great companion for docker would be something like this tools, that disable incoming connection and redirect port to the one inside but with real ipv6 ip addr.

As I said before if I have a real ipv6 on a docker, it is accessible directly without following the rules of dockers (port open/close or redirected).

[robbertkl]

Good to hear it's working! I'll close the issue now.