search

melbahja / got

Got: Simple golang package and CLI tool to download large files faster 🏃 than cURL and Wget!
MIT License
722 stars 46 forks source link

Stuck at 99.34% #47

Open tylerweitzman opened 2 years ago

tylerweitzman commented 2 years ago

Hi, I get the following

got -c 128 https://datashare.ed.ac.uk/download/DS_10283_3443.zip
  99.34% ▕████████████████████████▉▏ 12 GB/12 GB @ 0 B/s

I'm using a VM with 8 vCPUs, so it's 16 threads per core. It downloaded @40-90MB/s at the beginning and then once it reached 99.45% it slows down to 0 B/s. What's going on?

melbahja commented 2 years ago

Maybe you're IP address get blocked, 128 is too many concurrent connections.

melbahja commented 2 years ago

I would recommend that u always keep the -c value below 10.

tylerweitzman commented 2 years ago

I actually tried even 1000 connections. It was interesting because I was able to get up to 90 MB/s on that whereas with wget I get 400Kb. When I use 10 connections with got its around 3-5 MB/s. It kept up on 90 up until 90% and only then I started seeing it go down to 0

Is there any way that we could recover from rate limits that are delayed? For instance, start with 1000 connections, and then decrease to 10 later

On Tue, Apr 26, 2022 at 9:13 PM Mohamed El Bahja @.***> wrote:

I would recommend that u always keep the -c value below 10.

— Reply to this email directly, view it on GitHub https://github.com/melbahja/got/issues/47#issuecomment-1110425825, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP3R7X3PVS57BIZVZZH2ETVHCBBRANCNFSM5UNHZICA . You are receiving this because you authored the thread.Message ID: @.***>

-- --- Tyler Weitzman LinkedIn http://linkedin.com/in/tylerweitzman BlackSMS http://blacksms.info/

zQueal commented 2 years ago

I actually tried even 1000 connections. It was interesting because I was able to get up to 90 MB/s on that whereas with wget I get 400Kb.

Because you're entirely monopolizing the throughput from the server by using so many connections, which as melbahja pointed out, you've probably been IP banned as an anti-DDoS measure...or maybe the server admin noticed the uptick in simultaneous connections and rate limited the number of connections from 1 host.

Is there any way that we could recover from rate limits that are delayed?

Something like this has nothing to do with got. It's entirely up to the server you're pulling from what kind of bandwidth is acceptable, and what is not. You weren't very nice to the server, so being unbanned is probably not going to happen.

melbahja commented 2 years ago

Adding X-Rate-Limit-Limit support will help prevent issues like this.