search

microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.44k stars 822 forks source link

No IPv6 connectivity in docker container on win11/wsl2/mirrored #10663

Closed ruben-herold closed 3 months ago

ruben-herold commented 1 year ago

Windows Version

Microsoft Windows [Version 10.0.22621.2428]

WSL Version

2.0.5

Are you using WSL 1 or WSL 2?

Kernel Version

5.15.133.1-microsoft-standard-WSL2

Distro Version

Ubuntu 22.04

Other Software

Docker Desktop Version: 4.24.2

Repro Steps

Expected Behavior

Container can reach ipv6 target, system nat outgoing ipv6 connection like ipv4

Actual Behavior

no ipv6 is reachable

Diagnostic Logs

No response

keith-horton commented 1 year ago

HI there. can you please collect traces: https://github.com/microsoft/WSL/blob/master/diagnostics/collect-wsl-logs.ps1

There are some known limitations accessing the Windows Host from the Linux container. Are you trying to access an Internet resource from Linux/Docker?

Docker has a separate address than what is mirrored?

ruben-herold commented 1 year ago

We will collect the data in the next days because I'm out of office.

We try to reach ipv6 on the internet or via vpn (both did not work). If I enter the ubuntu on wsl i can do ping -6 blog.fefe.de for example. If I start a container and try to ping from the container, it failed via IPv6, but the container got an IPv6 address from the docker pool.

Seems that there is some kind of problem with wsl and mirrored network and docker-desktop. On linux it is very simple, configure docker to use ipv6 (https://ungleich.ch/u/blog/how-to-enable-ipv6-in-docker/) and do masquerade the network used in docker.

shigenobuokamoto commented 1 year ago

if you need masquerade, add this to daemon.json.

    "experimental": true,
    "ip6tables": true
ruben-herold commented 12 months ago

hi,

so I got a system for testing. Fresh windows 11 pro 23H2 with wsl2 2.0.9.0 and Docker Desktop v4.25.1. my .wslconfig:

[wsl2]
localhostForwarding=true

[experimental]
autoMemoryReclaim=gradual #
networkingMode=mirrored
dnsTunneling=true
firewall=true
autoProxy=true
sparseVhd=true
hostAddressLoopback=true

Docker config:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true,
  "fixed-cidr-v6": "fd00:ec2::/64",
  "ip6tables": true,
  "ipv6": true
}

IPv6 in wsl works without any problem:

PS C:\Users\admin> wsl
root@DESKTOP-TC7MMN9:/mnt/c/Users/admin# ping -6 -c 2 blog.fefe.de
PING blog.fefe.de(2001:4d88:3508::fefe:b106 (2001:4d88:3508::fefe:b106)) 56 data bytes
64 bytes from 2001:4d88:3508::fefe:b106 (2001:4d88:3508::fefe:b106): icmp_seq=1 ttl=55 time=17.6 ms
64 bytes from 2001:4d88:3508::fefe:b106 (2001:4d88:3508::fefe:b106): icmp_seq=2 ttl=55 time=19.6 ms

--- blog.fefe.de ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 17.551/18.591/19.632/1.040 ms
root@DESKTOP-TC7MMN9:/mnt/c/Users/admin#

But in docker:

PS C:\Users\admin> docker run -it alpine /bin/sh
/ # ping -6 -c 2 blog.fefe.de
PING blog.fefe.de (2001:4d88:3508::fefe:b106): 56 data bytes

--- blog.fefe.de ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
/ #

I have no idea how the docker desktop integrates wsl2 and windows, cause I normaly use linux.

ruben-herold commented 11 months ago

@shigenobuokamoto @keith-horton any idea? Or is it a docker problem?

shigenobuokamoto commented 11 months ago

@ruben-herold i tried it.

$ wslinfo --networking-mode
mirrored

$ cat /etc/docker/daemon.json
{
    "experimental": true,
    "ip6tables": true,
    "fixed-cidr-v6": "fd00:ec2::/64",
    "ipv6": true
}

$ docker run --rm alpine ping -6 -c 2 blog.fefe.de
PING blog.fefe.de (2001:4d88:3508::fefe:b106): 56 data bytes
64 bytes from 2001:4d88:3508::fefe:b106: seq=0 ttl=44 time=264.986 ms
64 bytes from 2001:4d88:3508::fefe:b106: seq=1 ttl=44 time=267.971 ms

--- blog.fefe.de ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 264.986/266.478/267.971 ms

ping seems to work. but,,, i installed Docker-CE with apt, not Docker Desktop.

if ip6tables is enabled, you can find masquerade entry in iptables.

$ sudo ip6tables -tnat -nvL POSTROUTING
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   442 MASQUERADE  all      *      !docker0  fd00:ec2::/64        ::/0
ruben-herold commented 11 months ago

@shigenobuokamoto thx good idea. I tried this but this also failed. I can ping from docker IPv6 Adresses but can't do any curl request ore something like that:

In wsl it worked:

root@DESKTOP-TC7MMN9:/mnt/c/Users/admin# ping -c1 -6 blog.fefe.de
PING blog.fefe.de(2001:4d88:3508::fefe:b106 (2001:4d88:3508::fefe:b106)) 56 data bytes
64 bytes from 2001:4d88:3508::fefe:b106 (2001:4d88:3508::fefe:b106): icmp_seq=1 ttl=55 time=30.5 ms

--- blog.fefe.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 30.514/30.514/30.514/0.000 ms
root@DESKTOP-TC7MMN9:/mnt/c/Users/admin# curl -6 -I http://blog.fefe.de
HTTP/1.1 200 Here you go
Server: Gatling/0.17
Content-Type: text/html; charset=utf-8
Content-Length: 13016
Last-Modified: Sat, 25 Nov 2023 13:45:36 GMT
Date: Sat, 25 Nov 2023 15:59:18 GMT
X-Frame-Options: DENY
Content-Security-Policy: frame-ancestors 'none';

In docker running on wsl:

root@DESKTOP-TC7MMN9:/mnt/c/Users/admin# docker run -it alpine:latest
/ # ping -c 1 -6 blog.fefe.de
PING blog.fefe.de (2001:4d88:3508::fefe:b106): 56 data bytes
64 bytes from 2001:4d88:3508::fefe:b106: seq=0 ttl=54 time=19.046 ms

--- blog.fefe.de ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.046/19.046/19.046 ms
/ # apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/7) Installing ca-certificates (20230506-r0)
(2/7) Installing brotli-libs (1.0.9-r14)
(3/7) Installing libunistring (1.1-r1)
(4/7) Installing libidn2 (2.3.4-r1)
(5/7) Installing nghttp2-libs (1.57.0-r0)
(6/7) Installing libcurl (8.4.0-r0)
(7/7) Installing curl (8.4.0-r0)
Executing busybox-1.36.1-r2.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 22 packages
/ # curl -v -I -6 http://blog.fefe.de
*   Trying [2001:4d88:3508::fefe:b106]:80...

If I do an tcpdump on the wsl I can see:

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:05:59.390023 IP6 2001:470:9d91:6:c9b1:cc94:a002:40ed.54848 > 2001:4d88:3508::fefe:b106.80: tcp 0
17:06:00.453982 IP6 2001:470:9d91:6:c9b1:cc94:a002:40ed.54848 > 2001:4d88:3508::fefe:b106.80: tcp 0
17:06:02.533922 IP6 2001:470:9d91:6:c9b1:cc94:a002:40ed.54848 > 2001:4d88:3508::fefe:b106.80: tcp 0
17:06:06.613864 IP6 2001:470:9d91:6:c9b1:cc94:a002:40ed.54848 > 2001:4d88:3508::fefe:b106.80: tcp 0

On my gateway I cann see that the packets go out and are send back to the windows machine. So looks for me like something in the windows firewall. or stack.

ruben-herold commented 11 months ago

@keith-horton @shigenobuokamoto also I found out that in docker on wsl all tools seems to prefer IPv4 over IPv6... so if I do a ping without -6 it tried IPv4 first....

Seems it has todo with the network fd00 is normaly for ULA and this leads do this. If I take a nother IPv6 range IPv6 will be prevered over IPV4 but still curl will not work..

shigenobuokamoto commented 11 months ago

@ruben-herold @keith-horton i was able to reproduce it.

in the current WSL (2.0.0 - 2.0.11), nft chains WSLOUTPUT and WSLPOSTROUTING have been added. this chain is applied only to ip (=IPv4), it seems that the source port conversion when transmitting from the container network to the outside via WSL is not applied to IPv6. it is possible to rewrite the table ip -> inet and forward from container via IPv6. as a temporary workaround, please execute the following command in WSL and then try curl from the container.

sudo nft add table ip6 filter
sudo nft add chain ip6 filter WSLOUTPUT '{type filter hook output priority filter; policy accept;}'
sudo nft add rule  ip6 filter WSLOUTPUT counter meta mark set 0x00000001
sudo nft add table ip6 nat
sudo nft add chain ip6 nat WSLPOSTROUTING '{type nat hook postrouting priority srcnat - 1; policy accept;}'
sudo nft add rule ip6 nat WSLPOSTROUTING 'oif "eth0" udp sport 1-65535 meta mark != 0x00000001 counter masquerade to :60600-60900'
sudo nft add rule ip6 nat WSLPOSTROUTING 'oif "eth0" tcp sport 1-65535 meta mark != 0x00000001 counter masquerade to :60600-60900'

edited: in the case of inet, it seems to interfere with the rules provided by /init, so change to ip6.

ps. the reason why IPv4 is preferred over ULA lies in the specifications of IPv6 itself.

ruben-herold commented 11 months ago

@shigenobuokamoto @keith-horton

Thx @shigenobuokamoto I can confirm that the nft commands help. Yes the ULA is standard I found it some minutes later and switched to NAT64 space for the docker containers...

How can this be fixed permanent?

What is the simplest way to give vstudio or vscode on windows access to the docker in wsl? As described at te start of this issue we have developers running windows who develop microservices in dot net core to run in linux containers.

They need the posibilty to run the containers from their IDE and must be able to debug the code.. Do you have any idea?

shigenobuokamoto commented 11 months ago

@ruben-herold WSLOUTPUT/WSLPOSTROUTING chains are probably defined by /init, so the only real solution is to wait for microsoft to resolve this issue.

as a temporary measure, it may be possible to deal with this by setting nft rule from systemd or .wslconfig.

ruben-herold commented 11 months ago

@shigenobuokamoto yes seems so. Is there some documentation available how the wsl interact with the windows system in mirrored mode?

I run into the next problem I started a container with nging -p80:80 but it is only reachable from wsl not from the windows host system.

shigenobuokamoto commented 11 months ago

@ruben-herold

I run into the next problem I started a container with nging -p80:80 but it is only reachable from wsl not from the windows host system.

this is another issue in WSL(mirror mode): WSL cannot relay localhost accesses from Windows. FYI; https://github.com/microsoft/WSL/issues/10494#issuecomment-1741605674

127.0.0.1        127.0.0.1
Windows -------> WSL -------> others(e.g. containers)
                    X<-------

a temporary workaround for this is to exclude traffic from loopback0 (= pseudo-local loopback shared with Windows) from nat.

i have prepared this response and IPv6 support for your reference. it is a temporary measure until the issue is resolved.

network-mirrored.service:

[Unit]
Wants=network-pre.target
Before=network-pre.target shutdown.target

[Service]
User=root
ExecStart=/bin/sh -ec "\
  nft add chain ip nat PREROUTING '{ type nat hook prerouting priority dstnat; policy accept; }';\
  nft insert rule ip nat PREROUTING iif loopback0 counter accept comment mirrored;\
  nft add table ip6 filter;\
  nft add chain ip6 filter WSLOUTPUT '{type filter hook output priority filter; policy accept;}';\
  nft add rule  ip6 filter WSLOUTPUT counter meta mark set 0x00000001 comment mirrored;\
  nft add table ip6 nat;\
  nft add chain ip6 nat WSLPOSTROUTING '{type nat hook postrouting priority srcnat - 1; policy accept;}';\
  nft add rule ip6 nat WSLPOSTROUTING 'oif "eth0" udp sport 1-65535 meta mark != 0x00000001 counter masquerade to :60600-60900' comment mirrored;\
  nft add rule ip6 nat WSLPOSTROUTING 'oif "eth0" tcp sport 1-65535 meta mark != 0x00000001 counter masquerade to :60600-60900' comment mirrored;\
"
ExecStop=/bin/sh -ec '\
  for chain in "ip nat PREROUTING" "ip6 filter WSLOUTPUT" "ip6 nat WSLPOSTROUTING";\
  do\
    handle=$(nft -a list chain $chain | sed -En "s/^.*comment \\"mirrored\\" # handle ([0-9]+)$/\\1/p");\
    for n in $handle; do nft delete rule $chain handle $n; done;\
  done;\
'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

edited: in the case of inet, it seems to interfere with the rules provided by /init, so change to ip6.

usage:

sudo cp network-mirrored.service /etc/systemd/system
sudo systemctl --now enable network-mirrored
ruben-herold commented 11 months ago

@shigenobuokamoto soory seems I must reinstall my test system so it needs a bit to test. After last testing I can't get wsl to start even after complete remove and reinstall. It hangs on start. Logs attached WslLogs-2023-11-28_12-59-20.zip

keith-horton commented 11 months ago

Thank you. For some reason the ETL was completely empty. Did you start the trace before trying to open WSL?

We have a new & better script for capturing traces: can you wsl --shutdown, start this trace by running the ps1, repro the issue once or twice, then stop the trace? https://github.com/microsoft/WSL/blob/master/diagnostics/collect-networking-logs.ps1

keith-horton commented 7 months ago

Thanks for reporting this. It looks like @shigenobuokamoto is correct - the rules we created for traffic exiting the Linux container only applied to IPv4 traffic, we didn't correctly make the corresponding IPv6 rules. I'm fixing that right now :)

Thank you for reporting this!

keith-horton commented 7 months ago

Thanks again. This should be fixed in the next WSL pre-release.

CatalinFetoiu commented 3 months ago

closing since the issue is fixed