search

jjjake / internetarchive

A Python and Command-Line Interface to Archive.org
GNU Affero General Public License v3.0
1.62k stars 218 forks source link

Upload speed problem #422

Open soredake opened 3 years ago

soredake commented 3 years ago

I dunno where problem resides, but asking anything on archive.org forums is useless, so i'll post it there.

I have a problem with upload speed being very low, using internetarchive in WSL2/Ubuntu or with vpn improves it, but i want to know what causes this.

internetarchive installed with pip.

Poorly measured (with windows task manager on windows and nload in WSL2) speed: zsh/msys2/no vpn 200-300kb/s (avg 200kb/s) zsh/msys2/proton nl vpn 600-1200kb/s (1100kb/s)

bash/wsl ubuntu 20.04/no vpn: 1200-3000kb/s (avg 1400kb/s) bash/wsl ubuntu 20.04/proton nl vpn: 1300-3000kb/s (avg 2200kb/s)

I've upload from HDD, but using SSD or even NVMe does not change anything, upload from browser have same speed problem. I've never had any problems with uploading anything in linux with either ia or browser.

Windows 10 Pro 21H1

maxz commented 3 years ago

Is this something that you are experiencing constantly? What is the normal upload bandwidth that you would expect to have with your connection?

The Internet Archive's ingestion speeds can vary a lot, depending on the current load.

It is rather strange that using WSL2 would improve the speed.

soredake commented 3 years ago

Is this something that you are experiencing constantly?

Yes, constantly, with every file, small or big, i dunno why vpn or wsl helps with this, also don't know why linux does not have problems, maybe bug in windows network drivers.

What is the normal upload bandwidth that you would expect to have with your connection?

More than 200-300kb/s at least, on linux my speeds were usually okay (2000-3000kb/s)

jjjake commented 3 years ago

Thanks for the report @soredake. We've been getting a lot of similar reports from other users recently, and we think it's an issue with our ISP. We're still sorting it out, but I'll keep you posted. Thanks again for the report, and my apologies for the trouble.

soredake commented 3 years ago

Upgraded to windows 11, no changes.

soredake commented 2 years ago

I've reinstalled my os, now i have problems even with WSL2 speed, after ~10 seconds it drops from 9 MB/s to 1-2 MB/s. Tried 3.0.0 release, master, even using Wi-Fi adapter, nothing changed.

soredake commented 2 years ago

But, after upgrading to latest commit on windows (not in WSL2) i now get full speed, but, still only for 10 seconds, but that's at least something.

soredake commented 2 years ago

Uploading from site interface varies from 200 KB/s to 2.5 MB/s.

soredake commented 2 years ago

Tried vpn, nothing changed.

soredake commented 2 years ago

Tested vpn for a few days with wsl2, now it helps me keep speed at decent speeds (5-9 MB/s) most of the time.

soredake commented 8 months ago

In the end, this problem is not related to windows, it's some kind of throttling from archive.org side, the only workaround i found on the internet is to use USA vpn when this happens, and it works (most of the time for now).