Adaptive formats are rate limited despite deciphering N

[azihassan]

I noticed that downloading with an itag other than 18 or 22 causes a slowdown, even if the N parameter is deciphered. What's strange is that if I abort the slow download and resume it again, the rate limiting no longer applies.

[azihassan]

I ran a few tests using the HTML and base.js files from Firefox. The video URL I found in Firefox's network analyser was similar to the one that youtube-d returned (with --output-url or --verbose) :

I suspect that the range argument is what's causing it to be rate limited. Running cURL with and without -C - shows that there's a difference in download speeds. While there's still no range parameter in the URL, -C - adds a range header to the request :

[azihassan]

I did some tests and realized that certain videos (or formats ?) don't have this behaviour. Here's an example where the download speed is limited to 110-ish kB/s :

One difference I notice is that in this case, curl follows a couple of redirects before downloading the file.

[azihassan]

libcurl doesn't send a Range header if CURLOPT_RESUME_FROM is set to 0, command-line cURL doesn't either. As established in the earlier comments, a missing range parameter or Range header triggers rate-limiting measures for adaptive formats. To get around this limitation, a Range: bytes=0- header needs to be manually added to the request if the destination file doesn't exist.

[azihassan]

I did some more tests and noticed that rate limiting isn't applied when requesting small ranges. Beyond a certain range, youtube starts rate limiting adaptive format videos for some reason. In a 200 MB video, the range cut-off was at 11 MBs :

Not sure if 11 MBs is calculated as a percentage of 200 MB or if it's a hard limit regardless of the video size though.

[azihassan]

I changed ParallelDownloader to take this into consideration. Trying to apply it to RegularDownloader proved to be a major refactoring, so I gave up on it. Instead I added a ChunkedDownloader that acts like a serial version of ParallelDownloader, meaning that it only downloads one chunk at a time. This will bypass the 11 MB limitation with the added benefit of not spamming the youtube servers with parallel connections. But this comes at the cost of introducing a chunk concatenation step when the download is finished. I'll test drive these changes for a few days to make sure they're stable before merging.