Slowed Download Speed?

[nguyen-anthony]

Checklist

• [X] I have checked the issue page for duplicates
• [X] I am running the latest version (download here)

Edition

Unsure

Describe your issue here

Just curious if there's any kind of issue with speed to download vods? Downloading a VOD of roughly 3-4 hours in length usually took about 5 minutes. Lately it's been about 20-30 minutes. I've checked on my end if there's been any changes to speeds in my network, but I'm not seeing anything. Curious if this is an application thing or a server issue by chance.

Thanks,

[ScrubN]

There is a configurable network limiter in the application as of 5 versions ago. It can be configured in the settings of the windows desktop application or via an arg in the CLI.

[ronan-thibaudau]

The network limiter is expressed in kb/sec and doesn't seem to go high (i didn't bother checking exact value but 30 000 works 40 000 doesn't save settings i assume signed int?). It would be nice to be able to disable the rate limit as i was downloading way above 30MB/sec/thread before

Edit: just checked the code and line 102 of the config page casts to int : Settings.Default.MaximumBandwidthKb = (int)NumMaximumBandwidth.Value; as well as in the settings config themselves line 188 :

2048

These should be changed to UInt64 and also ideally have a bool setting to disable throttling completely, the old unthrottled behavior worked very well even with concurrent downloads (ex: downloading 2000 clips in 1 go with 20-50 in parallel) and helped cap the connection

[nguyen-anthony]

There is a configurable network limiter in the application as of 5 versions ago. It can be configured in the settings of the windows desktop application or via an arg in the CLI.

Thank you for this info. I've adjusted the rate and will see if this helps. The default value is much slower than what my ISP provides me, so I'm assuming this was it. Will confirm and then close.

[ScrubN]

I am unable to reproduce 40,000 KB/s not saving.

Additionally we store the download speed in an Int32 which is enough to represent a limiter of 2.147 TB/s. If we used an Int16 then that would only cover 32.767MB/s, but we use an Int32.

It would be nice to be able to disable the rate limit

I will probably do this.

[ronan-thibaudau]

I am unable to reproduce 40,000 KB/s not saving.

Additionally we store the download speed in an Int32 which is enough to represent a limiter of 2.147 TB/s. If we used an Int16 then that would only cover 32.767MB/s, but we use an Int32.

It would be nice to be able to disable the rate limit

I will probably do this.

You're 100% right i had a brain fart there, now i'm confused as to why it fails, also i just tried again, 40 000 works, but typing 100 000 and just unfocusing resets it to the value typed before, can you repro this? If not i'll gladly try to debug it locally

[ScrubN]

The current version has a limit of 40,960KiB/s (40 MiB/s). I am not sure how fast the bandwidth limiter can theoretically process so I just put a large number. I can increase this in the next update, probably to 1Gbps (119.2 MiB/s). There is absolutely no way that twitch servers will provide more than gigabit to a single connection.

[ronan-thibaudau]

The current version has a limit of 40,960KiB/s (40 MiB/s). I am not sure how fast the bandwidth limiter can theoretically process so I just put a large number. I can increase this in the next update, probably to 1Gbps (119.2 MiB/s). There is absolutely no way that twitch servers will provide more than gigabit to a single connection.

I don't know if they'd cap above 1GB/s but i'm pretty sure i was saturating my 1gb/s connection and there are up to 8gb/s connections available for home use here, so no particular reason to limit it at all? It's a nice feature to have added for people who want stable internet while they download but i don't see what there is to gain by capping it / not allowing bypassing the limiter alltogether as i now download very slow compared to older versions and 40mbit/sec is way way lower than the speed i used to download at. (Full VODs used to take nearly no time to get to 100%, and then a while for the copy).

[nguyen-anthony]

Is there any desire to make this optional or remove the default value? Admittedly we didn't know this feature existed (so thanks y'all for responding here). The original issue is resolved here but feel free to keep this open for further discussion on increasing the limit and whatnot. Thanks everyone.

[ronan-thibaudau]

I think it would be better optional and opt in instead of opt out, default expectation i would have using this is no limit unless i ask for one, another sensible (but more work to implement) would be an opt out default value of ~80% tested internet speed but i think the throttling as a whole is a pretty niche need / the OS does a decent job at sharing between processes to begin with? I'm not sure what use case it really covers but if it's here to stay it would be nice to be able to at least opt out.

[ScrubN]

Toggle added in 774fbdc5021c9a1339efe3a419a3f15b72109f6b

[ronan-thibaudau]

Not the same issue but falls under the same title so not sure if i should make a new issue, i pulled the last version with this commit and still have a low download speed when downloading a single VOD, it goes just fine when downloading many VODs (no bandwidth cap to twitch) and seems to be caused by the limit of threads set to 4 (not sure where, i assume parallel tasks in the settings is not where you set threads for VOD download but for how many VODs to handle in parallel but i can't find anywhere in the UI to set it?).

Here too i'm not sure what the throttling code aims to achieve by default vs just letting it run by itself, the single VOD i tested on had 1023 chunks and i capped at ~100mb/sec, if i completely remove the semaphore and let things run, it goes just fine and i get this confirming twitch will happily go way past the speeds you expected (100% caps my bandwidth) :

thats a 10X speed increase, and if i upgraded to 8gbit which is available here it would be an 80x speed increase,i really think it's too bad to miss out on it (it may be worth capping still by default to avoid saturating the connection because even if the OS does a decent job at sharing, here it's 1023 calls for a single video so while computer stayed stable it did stutter my twitch viewing, but i think setting this value on first run to an overrideable default of Min(4,Max(CpuCores * 2, 40)) is reasonable / would let most people with fast connections cap out with the default setting while not affecting negatively people with a lower end CPU?

[ScrubN]

The SemaphoreSlim count is controlled by the number input on the video download page. Enqueued tasks will mirror the thread count from that page.

By removing the SemaphoreSlim, you made 1023 http requests to the same endpoint in a fraction of a second. If you weren't on Twitch/Amazon's API radar before, you probably are now 😆. I have heard from other users with faster internet than me that a single thread downloading a 1080p60 video is able to saturate over 250mbps, so assuming you're downloading the same resolution video I don't see why you were only getting 100mbps with 4 threads. The reason the download thread count is still exposed to the user is because with smaller files (160p30, 360p30 video) the bottleneck becomes disk writes and the latency of sending & receiving the http requests. To be specific, I have found from my testing that a single thread for 160p30 video tops out around 4mbps.

[ronan-thibaudau]

Oh thanks it makes more sense now that perf changed if that comes from the single video section as i re installed the program and just copy/pasted my datetime patterns a couple weeks back and i never use the single downloaders but long ago i used them so i may have set it there back in the day. I'm not sure if there's anything i could provide that would help figure out why it tends to cap out ~100mbps on 4 threads while the connection isn't bottlenecked, maybe different location's servers have different per chunk speed limit on twitch's side? If you can think of something i could do to test let me know. I'm sick and not thinking at my best so while i did ponder making it 64 threads instead of removing the semaphore for the test i wanted to check without any throttling and had already forgotten by then there were that many chunks.... you make a fair point there, i doubt anyone noticed as the test prolly lasted ~5 seconds and i'm not worried about looking atypical i just hope i didn't make one engineer's day worst by wasting 10mins looking up an incident and figuring out an idiot like me prolly screwed up their download hehe. Thanks for the help and fixes!:)