ipv6 request get's logged with docker's internal private ipv4 address

[thesix]

If you have a question, DO NOT SUBMIT a new issue. Please ask the question on the Q&A Group: https://groups.google.com/forum/#!forum/nginx-proxy

Tried that, but didn't work. Google says that "either this group doesn't exist or you don't have the right to see it" (translated from German).

I have nginx-proxy running with a bunch of websites (aka https://plagi.at/). All theses websites have been dual stack for years. Basically they still are reachable via ip (ipv6) and legacy ip (ipv4) but all ip connections get logged using docker's internal private ip address whereas legacy ip get's logged correctly using the client's address.

Has anyone seen this? Has anyone a solution for this?

I am running docker 18.09.6 on a debian stretch host using nginx-proxy:latest together with letsencrypt-nginx-proxy-companion.

Here are two lines of log output (docker logs nginx-proxy) to make my point. Line on was triggered by me using wget and definitely having ipv6 connection. Line two was someone else.

plagi.at 172.17.0.1 - - [01/Jun/2019:16:16:13 +0200] "GET / HTTP/1.1" 200 (...)
plagi.at 60.aaa.bbb.ccc - - [01/Jun/2019:16:21:47 +0200] "GET / HTTP/1.1" 301 (...)

Regards, J.

Edit: fixed docker version; 18.0.9.

[DatAres37]

You can try to run the reverse proxy in network mode "host". Not sure if this is 100% compatible with the letsencrypt companion tho.

[zoh48gz04]

I've got the same problem. Running nginx-proxy in network mode "host" results in "::1" as remote IP.

[artiomchi]

Can confirm, after spending hours trying to get IPv6 working with my docker server (weird network config), I've finally got native IPv6 working on there.. and I realise that it was all for "naught", since the IPs I get is the docker network's "gateway" IP, instead of anything relevant.

Would love to have this sorted, if possible, please!

[jantoine1]

This seems related to #401, although I can't even get IPv4 addresses to come through. I'm still only getting the internal gateway IP.

[cwildfoerster]

@thesix I have seem to get it working, using https://ipv6-test.com/validate.php to validate it's a real IPv6 address, because i'm currently IPv4 only. Below is my current docker-compose.yml. Please note, im building my own nginx-proxy with some of the open Pull Requests already merged. Key is to use the ipv6nat container and to create your own IPv6 enabled network. I think docker-compose versions > 2 need to create the network externally, because the enable_ipv6 key has been removed. Also per ipv6nat readme, make sure you're using a fc00::/7 IPv6 nework... and to configure the real_ip module.

Everything else is dead standard, no custom /etc/docker/daemon.json. Maybe make sure to have net.ipv6.conf.default.forwarding=1 and net.ipv6.conf.all.forwarding=1 enabled.

Logs for my personal IPv4 and ipv6-test.com

87.78.178.XXX - - [25/Jan/2020:18:11:28 +0000] "GET / HTTP/2.0" 200 36637 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
2001:41d0:701:1100::29c8 - - [25/Jan/2020:18:11:41 +0000] "GET / HTTP/1.0" 200 36637 "http://ipv6-test.com/validate.php" "ipv6-test.com validator"

My conf.d/realip.conf, which basically is pretty default, just using private networks:

set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
set_real_ip_from 192.168.0.0/16;
set_real_ip_from fc00::/7;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

version: '2.1'

services:
  nginx-proxy:
    build:
      context: ./nginx-proxy
      dockerfile: Dockerfile.alpine
    container_name: nginx-proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./certs:/etc/nginx/certs:ro
      - ./dhparam:/etc/nginx/dhparam:ro
      - ./default.conf:/etc/nginx/vhost.d/default
      - ./conf.d:/etc/nginx/conf.d 
      - ./logs:/var/log/nginx
    environment:
      - ENABLE_IPV6=true
      - RESOLVERS=1.1.1.1 1.0.0.1
      - DEFAULT_HOST=www.example.com
      - SEPERATE_LOGS_PER_VHOST=true
    networks: 
      nginx-proxy-network:
        aliases: 
          - nginx-proxy

  ipv6nat:
    depends_on:
      - nginx-proxy
    image: robbertkl/ipv6nat
    container_name: ipv6nat
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro

  default_web:
    build: ./default_web
    container_name: default_web
    restart: unless-stopped
    environment:
      - VIRTUAL_HOST=www.example.com
    networks:
      nginx-proxy-network:
        aliases:
          - default_web

networks:
  nginx-proxy-network:
    name: nginx-proxy-network
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: br-nginx-proxy
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: 172.22.1.0/24
        - subnet: fd4d:6169:6c63:6f77::/64

docker info

Client:
 Debug Mode: false

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 56
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-74-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.789GiB
 Name: xxx
 ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[starcraft66]

you need to configure ipv6 in your docker daemon or docker-compose network definition and give all of your containers ipv6 addresses and access services using those IPs otherwise docker will perform 6<->4 NAT.

[thesix]

Dear all,

I consider this matter closed since it cannot be fixed by nginx-proxy. My preferred solution is to use ipv6nat (as mentioned above by @cwildfoerster) in combination with traefik and completely forget about nginx.

Below you find an example on how to create all this with one docker-compose.yml file. Before you can use this you MUST set "userland-proxy": false in /etc/docker/daemon.json and restart docker. You also MUST change all appearances of 'domain.tld' with a suitable domain and make sure you have access to it's DNS in order to create missing entries.

If unsure how things will work for/with letsencrypt I recommend to un-comment the last command argument for the traefik container; this will use the letsencrypt staging environment.

Regards, J.

version: "3"

services:

  traefik:
    image: traefik:v2.1
    hostname: "traefik"
    container_name: "traefik"
    command:
      # - "--api.insecure=true"
      - "--api=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      # - "--log.level=DEBUG"
      - "--log.level=INFO"
      - "--accesslog=true"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--entryPoints.adminer.address=:8080"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.email=your-email@domain.tld"
      - "--certificatesResolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=web"
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`traefik.domain.tld`)"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.usersfile=/basicauth/usersfile"
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=httpsalways"
      - "traefik.http.middlewares.httpsalways.redirectscheme.scheme=https"

    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/path/to/letsencrypt:/letsencrypt"
      - "path/to/basicauth:/basicauth:ro"
    networks:
      - ipv6nat
    restart: unless-stopped

  whoami:
    image: "containous/whoami"
    hostname: "host.domain.tld"
    container_name: "whoami"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`host.domain.tld`)"
      - "traefik.http.routers.whoami.tls.certresolver=letsencrypt"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.entrypoints=websecure"
    networks:
      - ipv6nat
    restart: unless-stopped

  ipv6nat:
    image: "robbertkl/ipv6nat"
    container_name: "ipv6nat"
    hostname: "ipv6nat"
    entrypoint: "/docker-ipv6nat"
    command: "-cleanup -debug"
    network_mode: host
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro
    restart: unless-stopped

networks:
  ipv6nat:
    external: true

[matthijskooijman]

For future reference, it seems that dockerd has very recently gained experimental support for doing ipv6 NAT, which is AFAIU essentially the same as what the ipv6nat container referenced above does, but then built into docker. Once that is stabilized, it should allow for zero-effort ipv6 support. See https://github.com/moby/moby/pull/41622