microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.23k stars 808 forks source link

WSL 2.0: `networkingMode=mirrored` makes Docker unable to forward ports #10494

Open halimsamy opened 11 months ago

halimsamy commented 11 months ago

Windows Version

Microsoft Windows [Version 10.0.22631.2338]

WSL Version

2.0.0.0

Are you using WSL 1 or WSL 2?

Kernel Version

5.15.123.1-1

Distro Version

Ubuntu 20.04.6

Other Software

Docker version 24.0.6, build ed223bc

Repro Steps

Expected Behavior

To forward the port and be able to connect to my containers

Actual Behavior

Doesn't forward the port, so I cannot connect to my containers.

Diagnostic Logs

No response

driver1998 commented 11 months ago

Interestingly, the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.

Machine A:

Machine B:

greenhat616 commented 11 months ago

Same issue


Maybe related issues: https://github.com/docker/for-win/issues/13686

giovannicandido commented 11 months ago

Same problem here

lengthofrope commented 11 months ago

Yup, it's unfortunate but I have the same issue. I am on the release channel with windows version 10.0.22621.2359

over-star commented 11 months ago

Same problem here

speller commented 11 months ago

In my case, Docker ports are not exposed to the host Windows machine at all after the upgrade to 2.0 even if I set networkingMode=NAT which is the default value. I can't see opened ports in the tcpview.exe utility. Previously, I was able to see them opened by the wslhost.exe process (or similar name).

Switching to NAT helped me.

Yukari316 commented 11 months ago

same here

tuxiaobei-scu commented 11 months ago

+1

bkfino commented 11 months ago

+1

ptr1120 commented 11 months ago

+1

halimsamy commented 11 months ago

@benhillis Any ideas/updates here?

DukeCrimson commented 11 months ago

same here

tumugin commented 11 months ago

+1

Aerglonus commented 11 months ago

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

halimsamy commented 11 months ago

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

I think that's a workaround that would work... but I am waiting for someone from the WSL team to answer us here, did they get to know that is the issue, and if they have any plans to fix this?

lengthofrope commented 11 months ago

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

That seems to work. Nice find.

zcobol commented 11 months ago

ignorePorts is misleading. What it does is making in this case port 8080 available to request inside the container only, but is not forwarded to Windows side. Look for more details at https://learn.microsoft.com/en-us/windows/wsl/wsl-config

If you run get-nettcpConnection -LocalPort 8080 there's nothing listening on that port.

After starting the web server with docker run -d -p 8080:80 nginx:alpine requests inside the WSL container works:

zcobol@debian:~$ curl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:41:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes

Requests from Windows side timeout with:

curl: (28) Failed to connect to localhost port 8080 after 21276 ms: Couldn't connect to server

From another server it work, as mentioned by @driver1998 :

tux@raspi:~$ curl -I 192.168.1.105:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:43:45 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes
mew1033 commented 11 months ago

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125 I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

halimsamy commented 11 months ago

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125 I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

It seems like it's generally a problem with WSL itself. I am looking for a fix soon, since the new network mode is so much useful but it's not useable (usefully) in the current state.

imacarthur741 commented 11 months ago

I ma having the same issue with apache. No changes other than added --experimental but now nothing works.

(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down [no listening sockets available, shutting down](apache2.service: Control process exited, code=exited, status=1/FAILURE)

root@ACER-Nitro:/usr/sbin# lsof -nP -iTCP -sTCP:LISTEN COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-r 308 systemd-resolve 14u IPv4 22507 0t0 TCP 127.0.0.53:53 (LISTEN) mysqld 385 mysql 21u IPv4 32886 0t0 TCP 127.0.0.1:33060 (LISTEN) mysqld 385 mysql 23u IPv4 27396 0t0 TCP 127.0.0.1:3306 (LISTEN) miniserv. 1867 root 5u IPv4 52863 0t0 TCP *:10000 (LISTEN)

zohaibhassan156 commented 11 months ago

Same problem

zohaibhassan156 commented 11 months ago

ignoredPorts=8025,1025 work for me too. I had to use mailpit in docker

docker run -d \
--name=mailpit \
--restart unless-stopped \
-p 8025:8025 \
-p 1025:1025 \
axllent/mailpit
shigenobuokamoto commented 11 months ago

there seem to be two issues why Docker containers cannot connect from Windows.

  1. ignorePorts works Docker Desktop, probably

  2. the port forwarding does work from another machine on the same network as host. Just not on the host machine itself. docker-ce that installed on Linux

temporary measures for 2 .... disable iptables and use docker-proxy

/etc/docker/daemon.json

{
    "iptables": false
}

when using mirrored, the behavior seems to be different from the previous localhostforwarding.

use docker-proxy(listen on Linux)

windows  --->  linux(WSL)
  localhostforwarding (until)
ok    127.0.0.1   --->     127.0.0.1(lo)
ok    127.0.0.1   <---     127.0.0.1(lo)

  netowrkingMode=mirrored
ok    127.0.0.1   --->     127.0.0.1(loopback0)
ok    127.0.0.1   <---     127.0.0.1(loopback0)

interface is different, but the behavior remains the same.

use iptables(listeon on container)

windows  --->  linux(WSL) ---> Docker container
  localhostforwarding (until)
ok    127.0.0.1   ---> 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  --->  172.x.x.x(eth0)
ok    127.0.0.1   <--- 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  <---  172.x.x.x(eth0)

  netowrkingMode=mirrored
ok    127.0.0.1   ---> 127.0.0.1(loopback0)/127.0.0.1(br-xxxx) --->  172.x.x.x(eth0)
                                          ~~~~~~~~~
ng                                                           <- - -  172.x.x.x(eth0)

via localhostforwarding(until), source address(Windows) was the docker network gateway (=pointing to linux).

via mirrored, source address is 127.0.0.1. for this reason that packet is returned to 127.0.0.1(localhost on container) and does not reach Windows. in the case of access from another node(PC), there is no problem because source is his address.

i-curve commented 11 months ago

Same problem

jogiji commented 11 months ago

Same problem :) was on the verge of reinstalling the whole system..

ohroy commented 11 months ago

Any update?

halimsamy commented 11 months ago

ignorePorts (with iptables disabled) seems to work but it doesn't allow you to connect to the container from the Windows Machine. However, it doesn't work with Redis (running inside docker for some reason, so I didn't have enough time to look it up).

NasdaqScott commented 11 months ago

+1 for priority. There is a large contingent of corporate devs who need docker and mirrored mode (vpn/sec reqs).

ignoredPorts really isn't an ideal solution. Hopefully we can get an official update on this item as it's already quite popular.

rasta-mouse commented 11 months ago

Came across this issue today when starting my RabbitMQ container. Adding my +1 for visibility.

fanthos commented 11 months ago

I tested with disable iptables for docker in /etc/docker/daemon.json and use docker-proxy instead. It works on my device.

Another thing, when I set autoMemoryReclaim to gradual, all containers won't start. So I have to set it to dropcache.

My working .wslconfig

[wsl2]
macAddress=02:00:01:02:00:01
memory=8GB

[experimental]
sparseVhd=true
dnsTunneling=true
networkingMode=mirrored
autoMemoryReclaim=dropcache

My /etc/docker/daemon.json content:

{
  "hosts": [
    "tcp://0.0.0.0:2375",
    "unix:///var/run/docker.sock"
  ],
  "iptables": false
}
DukeCrimson commented 11 months ago

Using ignorePorts does not work on my laptop. I mean I still can not visit docker service port from Windows. I only installed docker in my wsl2 and didn't use docker desktop client in Windows. Is that the reason?

caaat233 commented 11 months ago

使用ignorePorts 在我的笔记本电脑上不起作用。 我的意思是我仍然无法从 Windows 访问 docker 服务端口。 我只在我的wsl2中安装了docker,并没有在Windows中使用docker桌面客户端。 是这个原因吗?

添加下面配置到wslconfig

[experimental]
autoMemoryReclaim=gradual
networkingMode=mirrored
dnsTunneling=true
firewall=false
autoProxy=false
hostAddressLoopback=true

并且添加 "iptables": false/etc/docker/daemon.json 就好了

felipecrs commented 11 months ago

@caaat233 it works indeed, thanks a lot! All it takes is to put {"iptables": false}.

keith-horton commented 10 months ago

Thanks for the work around. We are engaged with the Docker team to have better integration.

NasdaqScott commented 10 months ago

@keith-horton Why is this being closed when you are still actively working with Docker? Is there another issue that I can track? Is this workaround an official workaround suggested by MS? :) Are there any negative/downsides from turning iptables off in Docker w/ WSL?

keith-horton commented 10 months ago

Sorry, it was close for internal tracking of issues. If it's more appropriate, we can keep this open as well.

keith-horton commented 10 months ago

" Are there any negative/downsides from turning iptables off in Docker w/ WSL?"

Sorry, I don't know Docker well enough to explain their various options. Do they have a forum to ask this?

mew1033 commented 10 months ago

Sorry, it was close for internal tracking of issues. If it's more appropriate, we can keep this open as well.

Does that mean a fix has been developed and we're just waiting on a new version? If so, awesome! Thank you!

keith-horton commented 10 months ago

Sorry, we are working with the Docker developers so their solution works well with WSL. We don't yet have a fix from our side.

shigenobuokamoto commented 10 months ago

this might be closer to the previous behavior. (instead daemon.json)

$ nft insert rule ip nat PREROUTING iif loopback0 counter accept

generally, packets from local loopback are not routed. when mirrored, PREROUTING chain works because it routes packets from loopback0. so let the packets from loopback0 escape from PREROUTING chain.

however, this requires additional configuration. if WSL has set rules for nftables in advance..... loopback0.service.gz

halimsamy commented 10 months ago

Interestingly, the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.

Machine A:

  • Change networkingMode to mirrored
  • docker run -d -p 8080:80 nginx:alpine
  • Go to localhost:8080
  • Cannot connect
  • Set Windows firewall rules to allow 8080 inbound
  • Go to localhost:8080
  • Cannot connect

Machine B:

  • Go to A:8080
  • works

I first removed Docker Desktop and installed Docker directly on WSL, which allowed the containers to start. However, I encountered an issue where containers exposing ports were not accessible from the Windows machine or any device on the network.

Interestingly, when I started a Node server, it worked perfectly, and I could access it from both Windows and any device on the network. To resolve the port accessibility issue, I set "iptables": false in /etc/docker/daemon.json.

As a side note, I switched to legacy iptables to make VSCode work with the WSL plugin using the following command:

sudo update-alternatives --config iptables
BlueTree242 commented 10 months ago

Disabling iptables and using docker-proxy is a bad idea after all as docker-proxy has overhead

panoshu commented 10 months ago

使用ignorePorts 在我的笔记本电脑上不起作用。 我的意思是我仍然无法从 Windows 访问 docker 服务端口。 我只在我的wsl2中安装了docker,并没有在Windows中使用docker桌面客户端。 是这个原因吗?

添加下面配置到wslconfig

[experimental]
autoMemoryReclaim=gradual
networkingMode=mirrored
dnsTunneling=true
firewall=false
autoProxy=false
hostAddressLoopback=true

并且添加 "iptables": false/etc/docker/daemon.json 就好了

it works for me

mendhak commented 9 months ago

So the two workarounds I see so far, both have their negatives but worth listing anyway. Correct any of my misunderstandings.

Option1 - without docker desktop

Edit %USERPROFILE%\.wslconfig file (create it, if doesn't exist) and add this section

[experimental]
networkingMode=mirrored
hostAddressLoopback=true

Restart WSL, then in your distro, edit /etc/docker/daemon.json and add

{
"iptables": false
}

Then sudo systemctl restart docker.

With this, both normal port usage like python3 -m http.server and docker port usage like docker run -p "8080:8080" --rm -t mendhak/http-https-echo:26 are accessible from Windows side. The disadvantage is that iptables is now disabled.

Option 2 - with Docker Desktop

No longer necessary, update to the latest Docker Desktop and it should start working.

Edit %USERPROFILE%\.wslconfig file (create it, if doesn't exist) and add this section

[experimental]
networkingMode=mirrored
hostAddressLoopback=true
ignoredPorts = 8000,8080

Restart WSL and ensure Docker Desktop is running. With this, the normal port usage stops working if it uses a port listed above. eg the Python http.server isn't accessible over 8000 anymore (at least in my testing). Instead use a different port, like python3 -m http.server 8001. Docker ports will work as normal as long as the port is listed in the ignoredPorts above, like docker run -p "8080:8080" --rm -t mendhak/http-https-echo:26

The disadvantage here is you have to know which ports you'll be using with Docker. And it doesn't look like ignoredPorts accepts ranges of ports either so it can get pretty tedious.

And ignoredPorts only applies to docker containers it seems, not simple python http servers, unless I messed up? Did anyone run into the ignoredPorts issue as I did, for non-Docker applications?

AzureZeng commented 9 months ago

Docker Desktop 4.26.0 Release Notes For Windows * Added support for WSL mirrored mode networking (requires WSL v2.0.4 and up). * Added missing signatures on DLL and VBS files.

mendhak commented 9 months ago

Thank you for posting @AzureZeng

Would appreciate a few more people checking to confirm, but I just did a quick test and it seems to be working better than before, hopefully no other issues appear 🤞

In %USERPROFILE%\.wslconfig I only put this and nothing else:

[experimental]
networkingMode=mirrored

and updated to Docker Desktop 4.26.0

I am now able to

python3 -m http.server

and also

docker run -p 8080:8080 --rm -t mendhak/http-https-echo:31

Both are visible from the Windows side in a browser.

felipecrs commented 9 months ago

Ok, that's cool, but I hope this issue does not get closed already. What about people who are not using Docker Desktop? (which is not free and neither open source)

robpomeroy commented 9 months ago

Re the Docker update to 4.26.0 - WSL networking is working correctly for me in bridged mode. My Docker containers have stopped binding to my preferred NIC's IP though. Ports are exposed on localhost, but not my LAN IP. Attempting to bind containers to IP:port failed. So this may be a deal-breaker for me. (I don't want to create a custom bridge at the moment, if there's a chance the experience can be more seamless, with future Docker updates.)

AzureZeng commented 9 months ago

Ok, that's cool, but I hope this issue does not get closed already. What about people who are not using Docker Desktop? (which is not free and neither open source)

See above, set networkingMode=mirrored and hostAddressLoopback=true in .wslconfig, then set "iptables": false in /etc/docker/daemon.json.

I think there is not more convenient solution than this, unless Docker team want to improve the experience that running Docker daemon directly in WSL.

felipecrs commented 9 months ago

Yes, I'm aware of that and that's what I am using. BTW, I don't need to set hostAddressLoopback=true and apparently it still works fine.

But I do expect some answer from either the WSL team or Docker as to if this is the recommended/final approach or not.