microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.23k stars 808 forks source link

Very slow network speeds #8171

Open paul-vd opened 2 years ago

paul-vd commented 2 years ago

Version

Microsoft Windows [Version 10.0.22000.556]

WSL Version

Kernel Version

5.10.60.1

Distro Version

Ubuntu 20.04

Repro Steps

  1. Install branch latest version of windows 11 (just did a complete reinstall of my computer)
  2. Install everything wsl2 related from the docs (I did this twice to ensure it was not a faulty setup)
  3. Install NVM from the docs
  4. Try to download a node env (very slow speeds)
  5. Try to download the same tar using curl instead (also very slow speeds)

Downloading this from windows is almost instant.

Speedtest in my windows environment:

image

Speedtest in my WSL environment

$ speedtest
Retrieving speedtest.net configuration...
Testing from REDACTED...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by REDACTED [28.19 km]: 44.639 ms
Testing download speed
Download: 788.47 Mbit/s
Testing upload speed
Upload: 137.57 Mbit/s

Expected Behavior

I would expect it to not be much slower than my host machine.

Actual Behavior

~4 hours for 14mb makes it quite unusable...

image

Diagnostic Logs

Let me know if I should include the networking diagnostic from windows as well, they are long, so I did not add it.

WSL Networking Summary:

paul@PAUL-DESKTOP:~/temp/wsl/diagnostics $ ./networking.sh
If you see 'ping: socket: Operation not permitted' errors, run this command as root.
+ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal
+ uname -a
Linux PAUL-DESKTOP 5.10.60.1-microsoft-standard-WSL2 #1 SMP Wed Aug 25 23:20:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:c0:01:6a:1b:ed brd ff:ff:ff:ff:ff:ff
3: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 76:4b:c8:32:c2:9a brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:7c:a2:af brd ff:ff:ff:ff:ff:ff
    inet 172.30.22.56/20 brd 172.30.31.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe7c:a2af/64 scope link
       valid_lft forever preferred_lft forever
+ ip route show
default via 172.30.16.1 dev eth0
172.30.16.0/20 dev eth0 proto kernel scope link src 172.30.22.56
++ ip route show
++ awk '/default/ { print $3 }'
+ gateway=172.30.16.1
+ ping -c 4 172.30.16.1
PING 172.30.16.1 (172.30.16.1) 56(84) bytes of data.

--- 172.30.16.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3097ms

+ ping -c 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=18.6 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=58 time=19.0 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=58 time=18.8 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=58 time=18.7 ms

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 18.639/18.811/19.028/0.143 ms
+ traceroute 1.1.1.1
./networking.sh: line 28: traceroute: command not found
+ cat /etc/resolv.conf
nameserver 8.8.8.8
+ type curl
+ curl -m 5 -v https://microsoft.com
*   Trying 104.215.148.63:443...
* TCP_NODELAY set
* Connected to microsoft.com (104.215.148.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=microsoft.com
*  start date: Jan 24 04:39:56 2022 GMT
*  expire date: Jan 19 04:39:56 2023 GMT
*  subjectAltName: host "microsoft.com" matched cert's "microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 06
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55dc5d401880)
> GET / HTTP/2
> Host: microsoft.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 301
< date: Sat, 19 Mar 2022 09:44:03 GMT
< location: https://www.microsoft.com/
< server: Kestrel
< content-length: 0
< strict-transport-security: max-age=31536000; includeSubDomains
<
* Connection #0 to host microsoft.com left intact
paul-vd commented 2 years ago

So the issue seems to be originating from my network adapter, when I connect using my WIFI network adapter (built-in) everything works as expected, but when I switch to my ASUS XG-C100F 10G SFP+ Network Adapter the issue starts.

I'm unsure how to further debug this problem to understand why it's happening on my network adapter, I'm open to any suggestions or even a debug session to maybe isolate the issue?

brian-rey-dev commented 2 years ago

Did you try with this? It absolutely worked for me

https://townsyio.medium.com/wsl2-how-to-fix-download-speed-3edb0c348e29

Qormix commented 2 years ago

I also experienced inconsistent and slow network speed with my ASUS XG-C100C.

Disabling IPv6 on the network adapter gave me expected network speeds in WSL2

scovetta commented 1 year ago

@paul-vd What version of Windows/WSL are you running? There are a few reports of this issue going away in 22H2 (around version 10.0.22623.730 or later).

wsl --version should give enough information.

paul-vd commented 1 year ago

@scovetta this was "resolved" after disabling IPv6.

Here is my version info:

WSL version: 0.70.0.0 Kernel version: 5.15.68.1 WSLg version: 1.0.45 MSRDC version: 1.2.3575 Direct3D version: 1.606.4 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.22621.674

calixteman commented 1 year ago

I managed to "resolve" my super slow network issues in wsl2 in disabling ipv6. My network device is a Marvell FastLinQ Edge 10Gbit Network Adapter and wsl --version outputs:

WSL version: 0.70.4.0
Kernel version: 5.15.68.1
WSLg version: 1.0.45
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.674
eugepemi commented 1 year ago

I tried everything and finally I could solve it:

Finally I tried to disable IPv6 from the vEthernet of WSL and in W11 doesn't appears listed in the control panels, reading GitHub I found out that is because is a hidden device, from PowerShell I managed to disable it and right now WSL is has regular network speeds, the command:

Disable-NetAdapterBinding -Name "vEthernet (WSL)" -ComponentID ms_tcpip6 -IncludeHidden

jkugiya commented 1 year ago

I solved the problem by trying the following (Windows 11 Home).

Disable-NetAdapterLso -IncludeHidden -Name "vEthernet (WSL)"
cgraf-spiria commented 1 year ago

Just restarting the WSL network adapter worked for me. Had 5Mbps upload speed, after restart back to normal speeds (>800Mbps)

In Administrator Powershell: Restart-NetAdapter -Name "vEthernet (WSL)" -IncludeHidden

Zerogoki00 commented 1 year ago

I have the same problem:

Testing download speed................................................................................
Download: 2.15 Mbit/s
Testing upload speed......................................................................................................
Upload: 287.43 Mbit/s

Nothing helped me, only reboot solved it. And it randomly appears again

Rowanto commented 1 year ago

For me restarting the WSL adapter does not work. What worked is deactivating the wifi adapter in windows network control panel and activating it again, then it gained full speed, until it worked again.

mbarberry commented 1 year ago

This works for me:

1- Press the Windows Key and type “Control Panel” and hit enter 2- Click “Network and Internet” 3- Click “View network status and tasks” 4- Click “Change adapter settings” 5- Find the “vEthernet (WSL)” adapter and click properties 6- Click “configure” and open the advanced tab

Note: 2 and 3 might be under "Network and Sharing Center" in the Control Panel

Select “Large Send Offload Version 2 (IPv4)” and change the drop-down to disabled. To the same for “Large Send Offload Version 2 (IPv6)”.

Once this is done click “OK” and you are all done. You may experience a slight downtime in connection while this is saved.

peterfelts commented 1 year ago

For those of you that can't easily disable the "Large Send Offload Version 2 (IPv4)" (or IPv6) because the WSL adapter is hidden in the control panel, here are the steps I took to disable this setting via powershell: (full disclosure: I have zero understanding/comprehension of what this setting does... so I'm essentially pressing buttons at random)

In PowerShell:

get-netadapter

This will give you a list of network adapters on your system:

image

Next, run the following command to view the advanced properties of the "vEthernet (WSL)" adapter:

Get-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" | Format-Table -AutoSize

Here, you can see that this setting had a value of 1, which I speculate maps to "enabled".

image

Next, change the value to "Disabled":

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled.

image

I applied this change and network speed in my WSL environment seemed to have improved.

peterfelts commented 1 year ago

I woke up this morning and found that my Windows box had been "updated". I also saw that my WSL speeds were slow again. I opened up this thread and followed the steps in my previous post and immediately saw the download speeds increase:

image

This is really bad. This suggests that updates are wiping out the workout that people are using.

davorpa commented 1 year ago

For those of you that can't easily disable the "Large Send Offload Version 2 (IPv4)" (or IPv6) because the WSL adapter is hidden in the control panel, here are the steps I took to disable this setting via powershell: (full disclosure: I have zero understanding/comprehension of what this setting does... so I'm essentially pressing buttons at random)

In PowerShell:

get-netadapter

This will give you a list of network adapters on your system: image

Next, run the following command to view the advanced properties of the "vEthernet (WSL)" adapter:

Get-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" | Format-Table -AutoSize

Here, you can see that this setting had a value of 1, which I speculate maps to "enabled".

image

Next, change the value to "Disabled":

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled.

image

I applied this change and network speed in my WSL environment seemed to have improved.

Thanks, the only response that works for us since we are running WSL on Windows 11 Pro where vEthernet adapters are hidden. So I executed:

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" -IncludeHidden -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled

... and voilà.... from 0-60kb/s to fullspeed

cgraf-spiria commented 1 year ago

Looks like disabling TCP Segmentation Offload inside WSL/Linux also works:

sudo apt install ethtool

sudo ethtool -K eth0 tso off

sudo ethtool -k eth0 | grep 'tcp-segmentation-offload'

Credit: https://github.com/microsoft/WSL/issues/4901#issuecomment-1580750148

That fixed my upload speed being really slow.

zgeor commented 9 months ago

This is getting silly, I've had to "fix" this countless times now. Which is only possible due to this thread. So I had to modify the above instructions by @peterfelts and @davorpa as there seems to have been a new change. The adapter is now hidden from get-netadapter, and might have a different name.

I've made the assumption that the adapter will start with vEthernet and have WSL somewhere else following in the name. With that in mind here is a combined query to get the properties to check your setup

get-netadapter -IncludeHidden | Where-Object Name -like "vEthernetWSL" | Get-NetAdapterAdvancedProperty -IncludeHidden | Format-Table -Autosize That would list the advanced properties of your vEthernet WSL adapter.

To check only for LSOv2, you can run just the below

get-netadapter -IncludeHidden | Where-Object Name -like "vEthernetWSL" | Get-NetAdapterAdvancedProperty -IncludeHidden | Where-Object DisplayName -like "Large Send Offload Version 2*"

Now to disable tjhe properties for both ipv4 and ipv6

get-netadapter -IncludeHidden | Where-Object Name -like "vEthernetWSL" | Get-NetAdapterAdvancedProperty -IncludeHidden | Where-Object DisplayName -like "Large Send Offload Version 2*" | Set-NetAdapterAdvancedProperty -DisplayValue Disabled

marcoaureliocardoso commented 8 months ago

Had to include "| Format-Table -AutoSize" to get the full adapter name: get-netadapter -IncludeHidden | Format-Table -AutoSize | Where-Object Name -like "vEthernet (WSL (Hyper-V firewall))" | Get-NetAdapterAdvancedProperty -IncludeHidden | Where-Object DisplayName -like "Large Send Offload Version 2*" | Set-NetAdapterAdvancedProperty -DisplayValue Disabled

N1TROGUE commented 8 months ago

Had to include "| Format-Table -AutoSize" to get the full adapter name: get-netadapter -IncludeHidden | Format-Table -AutoSize | Where-Object Name -like "vEthernet (WSL (Hyper-V firewall))" | Get-NetAdapterAdvancedProperty -IncludeHidden | Where-Object DisplayName -like "Large Send Offload Version 2*" | Set-NetAdapterAdvancedProperty -DisplayValue Disabled

I had to replace Set-NetAdapterAdvancedProperty -DisplayValue Disabled with ForEach-Object { Set-NetAdapterAdvancedProperty -InputObject $_ -DisplayValue Disabled } making it: Get-NetAdapter -IncludeHidden | Where-Object Name -like "vEthernet (WSL (Hyper-V firewall))" | Get-NetAdapterAdvancedProperty -IncludeHidden | Where-Object DisplayName -like "Large Send Offload Version 2*" | ForEach-Object { Set-NetAdapterAdvancedProperty -InputObject $_ -DisplayValue Disabled }, otherwise I would get the following error six times in row: "Set-NetAdapterAdvancedProperty: The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input."

corfor commented 8 months ago

I refactored that a bit to be more verbose

# Get all network adapters, including hidden ones
Write-Host "Retrieving all network adapters, including hidden ones..."
$adapters = Get-NetAdapter -IncludeHidden

# Filter the adapters to find the one with a specific name pattern
Write-Host "Filtering for network adapter with name like 'vEthernet (WSL (Hyper-V firewall))'..."
$targetAdapter = $adapters | Where-Object Name -like "vEthernet (WSL (Hyper-V firewall))"

# Check if the target adapter is found
if ($targetAdapter) {
    Write-Host "Network adapter found. Retrieving advanced properties..."
    # For the found adapter, retrieve its advanced properties, including hidden ones
    $advancedProperties = $targetAdapter | Get-NetAdapterAdvancedProperty -IncludeHidden

    # From these properties, filter to find those with a display name matching a specific pattern
    Write-Host "Filtering for properties with display name like 'Large Send Offload Version 2*'..."
    $targetProperties = $advancedProperties | Where-Object DisplayName -like "Large Send Offload Version 2*"

    # Check if the target properties are found
    if ($targetProperties) {
        Write-Host "Properties found. Disabling 'Large Send Offload Version 2'..."
        # For each property that matches the criteria, set its value to 'Disabled'
        foreach ($property in $targetProperties) {
            Set-NetAdapterAdvancedProperty -InputObject $property -DisplayValue Disabled
            Write-Host "Property '$($property.DisplayName)' is now disabled."
        }
    } else {
        Write-Host "No properties matching the criteria were found."
    }
} else {
    Write-Host "No network adapter matching the criteria was found."
}
khuongphp commented 8 months ago

This should be answer: https://github.com/microsoft/WSL/issues/4901#issuecomment-1845315463

ericblade commented 7 months ago

This should be answer: #4901 (comment)

I'm sorry, but networkingmode=mirrored only breaks my networking completely, and I already have all the other features advertised on it working

igmogo-ku commented 7 months ago

I solved the problem by trying the following (Windows 11 Home).

Disable-NetAdapterLso -IncludeHidden -Name "vEthernet (WSL)"

Hi,

this command solves the problem for me. Unfortunately, I have to rerun the command after every reboot. Is there any way to make this change permanent?

Thanks and regards

ericblade commented 7 months ago

None of these get me going consistently over 200mbps with speedtest-cli in the bare wsl, or over 30mbps in docker containers. Network mode=mirrored results only in no networking working, as far as I can tell.

optimbro commented 7 months ago

For those of you that can't easily disable the "Large Send Offload Version 2 (IPv4)" (or IPv6) because the WSL adapter is hidden in the control panel, here are the steps I took to disable this setting via powershell: (full disclosure: I have zero understanding/comprehension of what this setting does... so I'm essentially pressing buttons at random)

In PowerShell:

get-netadapter

This will give you a list of network adapters on your system: image

Next, run the following command to view the advanced properties of the "vEthernet (WSL)" adapter:

Get-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" | Format-Table -AutoSize

Here, you can see that this setting had a value of 1, which I speculate maps to "enabled".

image

Next, change the value to "Disabled":

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL)" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled.

image

I applied this change and network speed in my WSL environment seemed to have improved.

This worked for me... I had to change some of the values though. Here's the full:

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL (Hyper-V firewall))" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled -IncludeHidden

from

image

to

image

ericblade commented 7 months ago

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL (Hyper-V firewall))" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled -IncludeHidden

This one got me from around 150-200mbps down to 350mbps down, and i'm now hitting my upload cap at close to 50mbps in bare WSL, still poking in docker at about 30mbps down, 1.5mbps up

optimbro commented 7 months ago

Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL (Hyper-V firewall))" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled -IncludeHidden

This one got me from around 150-200mbps down to 350mbps down, and i'm now hitting my upload cap at close to 50mbps in bare WSL, still poking in docker at about 30mbps down, 1.5mbps up

try this once

PASTE ON POWERSHELL

Restart WSL Network Adapter

Restart-NetAdapter -Name "vEthernet (WSL (Hyper-V firewall))" -IncludeHidden
Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL (Hyper-V firewall))" -DisplayName "Large Send Offload Version 2 (IPv4)" -DisplayValue Disabled -IncludeHidden
Set-NetAdapterAdvancedProperty -Name "vEthernet (WSL (Hyper-V firewall))" -DisplayName "Large Send Offload Version 2 (IPv6)" -DisplayValue Disabled -IncludeHidden

PASTE WSL Distro (Ubuntu)

upload speed

sudo apt install ethtool
sudo ethtool -K eth0 tso off
sudo ethtool -k eth0 | grep 'tcp-segmentation-offload'
optimbro commented 7 months ago

What I have experienced is when I create new WSL2 instance then everything is fine. Download/Upload speeds are normal.

When I move this another drive, then the network speed becomes slow, while upload speed is same but download speed becomes very slow from 200mbps to 2-8mbps.

luizbezerra-pinheiro commented 5 months ago

Looks like disabling TCP Segmentation Offload inside WSL/Linux also works:

sudo apt install ethtool

sudo ethtool -K eth0 tso off

sudo ethtool -k eth0 | grep 'tcp-segmentation-offload'

Credit: #4901 (comment)

That fixed my upload speed being really slow.

Thank you!!!! A thousand times thank you!! My download speed was OK (600 MBits/s) but my upload was like 1 MBit/s now it it also around 600, thank you again!!

4cademy commented 3 months ago

I have tried so many things by now. The one that fixed it for me was:

Admin PowerShell: wsl --shutdown Disable-NetAdapterLso -IncludeHidden -Name "vEthernet *"