search

lay295 / TwitchDownloader

Twitch VOD/Clip Downloader - Chat Download/Render/Replay
MIT License
2.68k stars 261 forks source link

Considering increasing the limit of download threads #982

Closed jyeric closed 6 months ago

jyeric commented 7 months ago

Checklist

Write your feature request here

Currently, the limit of download threads is set to 10 per task. Can the limit be increased so that a faster download speed can be gained?

It seems that I got a low speed from my place to twitch server. Adding more threads can increase the speed to download vod.

By the way, I'm not sure more threads will be considered abusing to the server. If this attempt will lead to 403, maybe I should give up.

superbonaci commented 6 months ago

Did you try patching the code and trying? You can find it here:

https://github.com/lay295/TwitchDownloader/blob/1afcc3ce2166ab128cb6ca0463fae46acc2e59b5/TwitchDownloaderCore/VideoDownloader.cs#L242C13-L242C96

Twitch servers are designed to serve video at normal playback speed, if you want to download from a terabit network may not be as fast as possible.

superbonaci commented 6 months ago

The 10 limit is for restarted threads, but you can have as many simultaneous download threads as you want. Did you try?

krowgoat commented 6 months ago

What are threads anyway? Actual question.

superbonaci commented 6 months ago

What are threads anyway? Actual question.

In theory threads are connections you make to Twitch to download the .ts parts. Each part can be in a different server, so if you download only one (-t 1) at the same time, the speed could be much lower, for example the hard drive is shared and they server the file slowly. So if you download several parts at the same time you can have higher download speed, but all depends how the Twitch servers work.

Generally speaking the more threads the better, your own hard drive can be a limitation also. If you abuse they can ban your account or your IP so it's up to you to use as many as possible.

ScrubN commented 6 months ago

The 10 limit is for restarted threads, but you can have as many simultaneous download threads as you want. Did you try?

What are threads anyway? Actual question.

Threads are the number of simultaneous connections to Twitch servers.

Back when the program used the WebClient class, threads were much more important because WebClient had a maximum bandwidth of around 10Mbps. Now we use HttpClient, which does not have a maximum bandwidth so threads are much less important, but there are still scenarios where more threads is useful, such as:

The problem is finding a good middle ground of allowing users to set an appropriate thread count without banning their IP and blaming the program for it. I might increase the VOD thread cap to 20 in the GUI, but with upcoming Kick support threads becomes more of an issue. I have found that above 4 threads, I almost always receive HTTP 429 from Kick, and there was already an issue report from someone trying to download a Kick chat with 50 threads and getting their IP banned.

krowgoat commented 6 months ago

Thanks!

On Friday, March 15th, 2024 at 12:47 PM, Scrub @.***> wrote:

The 10 limit is for restarted threads, but you can have as many simultaneous download threads as you want. Did you try?

  • The GUI has a maximum of 10 threads.
  • Restarted threads is either threads * 1.5 or 10, whichever is larger.

What are threads anyway? Actual question.

Threads are the number of simultaneous connections to Twitch servers.

Back when the program used the WebClient class, threads were much more important because WebClient had a maximum bandwidth of around 10Mbps. Now we use HttpClient, which does not have a maximum bandwidth so threads are much less important, but there are still scenarios where more threads is useful, such as:

  • downloading at lower qualities (file size so small the overhead of making requests and disk IO becomes noticeable)
  • downloading in regions where Twitch throttles individual connections

The problem is finding a good middle ground of allowing users to set an appropriate thread count without banning their IP and blaming the program for it. I might increase the VOD thread cap to 20 in the GUI, but with upcoming Kick support threads becomes more of an issue. I have found that above 4 threads, I almost always receive HTTP 429 from Kick, and there was already an issue report from someone trying to download a Kick chat with 50 threads and getting their IP banned.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

ScrubN commented 6 months ago

I completely forgot to up the threads, my apologies.