Very Slow Download of Packages on Sourceforge

JerwinModavi commented 2 years ago

Brief description of your issue

Very slow download speed of Avidemux.Avidemux which was downloaded from the following link: https://sourceforge.net/projects/avidemux/files/avidemux/2.8.0/avidemux_2.8.0_win64.exe/download

The installer is only 31.6 MB and it took several minutes to download on a 1 Gbps Ethernet Internet connection.

Packages downloading from other sources download much faster so this could potentially be a Sourceforge related issue.

Steps to reproduce

winget install Avidemux.Avidemux

Expected behavior

Download faster

Actual behavior

Downloaded very slowly

Environment

Windows Package Manager v1.2.11601
Windows 11 latest version at time of writing
Avidemux.Avidemux Version 2.8.0.220109

ItzLevvie commented 2 years ago

It took me less than 15 seconds on 4G LTE (50 Mbps) so maybe there's no closest mirror servers near where you live so you were probably redirected to a mirror server that's far away from you.

Sometimes it's possible for SourceForge to connect you to the wrong mirror servers (i.e. if you're in EU or US, you might be redirected to downloading from China or Japan).

degam-gh commented 2 years ago

I've had this issue for several packages as well, since winget seems to use the Microsoft delivery optimization algorithm by default(?), which tend to waste a lot of time under some circumstances on my corporate machine. You may try to bypass by setting the network downloader to wininet: https://docs.microsoft.com/de-de/windows/package-manager/winget/settings#downloader

JerwinModavi commented 1 year ago

Still experiencing this issue. This time with upgrading WinSCP also downloaded from Sourceforge.

ItzLevvie wrote:

It took me less than 15 seconds on 4G LTE (50 Mbps) so maybe there's no closest mirror servers near where you live so you were probably redirected to a mirror server that's far away from you.

Sometimes it's possible for SourceForge to connect you to the wrong mirror servers (i.e. if you're in EU or US, you might be redirected to downloading from China or Japan).

I'm located in the US - Mountain Standard Time Zone region. I take it the selection of the mirror is out of my hands.

degam-gh wrote:

I've had this issue for several packages as well, since winget seems to use the Microsoft delivery optimization algorithm by default(?), which tend to waste a lot of time under some circumstances on my corporate machine. You may try to bypass by setting the network downloader to wininet:

Thanks for the suggestion. I followed the directions from the link below and changed my winget settings to use wininet. Will see next time if there is an improvement. https://learn.microsoft.com/en-us/windows/package-manager/winget/settings#downloader

stearless commented 1 year ago

Thanks for the suggestion. I followed the directions from the link below and changed my winget settings to use wininet. Will see next time if there is an improvement.

The issue is still present, changing to wininet doesn't fix the problem, and as far as I noticed, it's only affected by downloads from sourceforge, and it's not true that faster (closer) mirror is not available, if you try to manually download the same app from sourceforge download is very quick, so it's a winget problem

Karl-WE commented 1 year ago

Hi I am also seeing this for a long time now.

Best to reproduce with the community package

apache.openoffice

Using the exact same link that's presented from winget and using this link to DL the file via MS Edge works like a blast. (1 GBit) Via winget it's limiting to 0.9-1.2 MBit according to Task Manager.

I am not seeing this with other DL sources.

Windows: Windows.Desktop v10.0.25267.1000 Systemarchitektur: X64 Paket: Microsoft.DesktopAppInstaller v1.19.3132.0

Protokolle: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir

What could I provide for better investigation? @ItzLevvie

Edit: While repro I am using DNS from cloudflare (DNS over https) but can repo it with other on other Windows PCs, networks, involving other ISP and DNS like Telecom Germany, Telefónica or Vodafone.

ItzLevvie commented 1 year ago

and it's not true that faster (closer) mirror is not available

cc @stearless: It depends on which app you're trying to download from SourceForge.

Some apps published on SourceForge only has limited amounts of mirrors and some of them won't be closest to your location.

What could I provide for better investigation?

cc @Karl-WE: No idea. I tested on Shadow PC - Power Upgrade on AMD EPYC 7545P with NVIDIA RTX A4500 (1 Gbps) & Azure VM (4.6 Gbps) and both were able to download in about 2 to 3 seconds with wininet which matches what I see on Microsoft Edge & on Google Chrome.

I only see slower speeds with Delivery Optimization (DO) for SourceForge.

stearless commented 1 year ago

and it's not true that faster (closer) mirror is not available

cc @stearless: It depends on which app you're trying to download from SourceForge.

Some apps published on SourceForge only has limited amounts of mirrors and some of them won't be closest to your location.

It makes no sense what you are saying, downloading manually via the same link downloads apache.openoffice for example almost instantly, but wininet or DO take minutes literally, so it means there is in fact mirror available just Winget isn't able to use it, unlike some other package managers

ItzLevvie commented 1 year ago

It makes no sense what you are saying, downloading manually via the same link downloads apache.openoffice for example almost instantly

~~Oops. I didn't realize that my trackpad wiped a lot of useful text in that comment before I posted it but I'm now back on my PC.~~

There's actually multiple situations for SourceForge:

1) A closest mirror is available but isn't used which is done through load balancing by SourceForge — to ensure the amount of people on those mirrors are spread equally for reliability and that there aren't a lot of people hogging down one of the mirrors because of too much people. 2) No alternative mirrors available where everyone is placed only on one mirror/server. 3) Lots of mirrors available and you're placed to the closest one.

If an end user is using WinGet you won't be able to easily tell which situation you're in so in order to find out which mirror you're connected to you will have to use Fiddler (or something else) before WinGet starts downloading the package. WinGet logs exposes the Download URL but doesn't expose which URLs you're redirected to.

Sometimes for other URLs outside of SourceForge you may have to switch between Delivery Optimization (DO) & WinINet because both of them has its own advantages and disadvantages. Other package managers uses different downloading methods compared to WinGet so you'll always see different results compared to WinGet.

What happens if you forced the mirror you're closest to instead of https://sourceforge.net/projects/openofficeorg.mirror/files/4.1.13/binaries/en-GB/Apache_OpenOffice_4.1.13_Win_x86_install_en-GB.exe/download in WinGet by downloading the manifest file from https://cdn.winget.microsoft.com/cache/manifests/a/Apache/OpenOffice/4.113.9810/0917-Apache.OpenOffice.yaml, changing the Download URL, and then manually test by running winget install --manifest .\0917-Apache.OpenOffice.yaml in Command Prompt or PowerShell?

EDIT: With PowerShell (7.4.0-daily20221224.1)'s Invoke-WebRequest on a 130 Mbps connection with the forced mirror — I do see a lot of speed differences which theoretically proves that the issue either stems from SourceForce's side or through the HTTP APIs.

I'm not sure if PowerShell's Invoke-WebRequest is also powered by WinINet but doing the same test on WinGet is much more faster as well. Delivery Optimization (DO) on WinGet also works properly on that URL. I did roughly 20 downloads on both Delivery Optimization (DO) & WinINet to make sure that it wasn't just me being lucky there.

Karl-WE commented 1 year ago

@ItzLevvie Happy New Year.

What would you propose to setup for reproduction. I believe your findings and being thankful for your intense testing.

Yet my results are exact opposite here. Downloading apache.openoffice will always load that slow via WinGet (only)

What logs could I provide from fiddler or winget that will be helpful to understand why there is this discrepancy? Any special settings on fiddler?

microsoft / winget-cli