Avoid bandwith throttling

[error39]

Hola Ricardo,

I Wondered if there's a way of implementing an option for circumventing a specific kind of bandwidth throtling?

Let me explain: I noticed quite a few providers, for example soundcloud and mixcloud, will serve the first few megabytes of a stream at maximum speed, but after this the speed drops just to the amount needed for streaming the served media at its bitrate. Like this it takes a long time to download certain media with youtube-dl. How I used to circumvent this is by stopping & resuming the download for example 20 times for a 100mb file, in case they serve the first 5mb unthrottled.

Technically this works very well, but it's not very handy in a practical way. So I created my little helper script where I use youtube-dl to generate the download-url, and let Curl download the actual file, by downloading the first XXkb block, than the second XXkb block, etc... using the range header start & end values, and a while loop.

This works out great for me, but I wonder if it would be very difficult to implement this in the downloader classes of youtube-dl? With an optional switch to activate this 'chunk-downloading', or even the ability for contributors to implement it in the extractors - if the specific service is known for using these 'first part fast' throttling mechanisms - would really benefit the download speed for a lot of users, in my humble opinion :)

So, my questions are: Do you think it's a good idea to implement this kind of functionality and how difficult will it be to do so? If you consider implementing this in the future, I like to help in any way I can to make this possible.

Thanks for creating this great 'little' tool, I use it regularly and am very happy it exists!

Saludos,

Erik

[rg3]

Thanks for your kind words on behalf of the current youtube-dl maintainers. As for the feature request, wait for their reply.

Best regards.

[error39]

I would like to look into this myself, as there's only a limited amount of time maintainers have to look into feature requests like this, I guess. Anyone who can give me a headstart with some direction to which files/functions are actually responsible for handling the actual download of the video or audio file? One more question; Is there a guide/documentation on how to compile the python code yourself on a linux system?

Thanks!

edit: Found the developer notes myself: https://github.com/rg3/youtube-dl/#developer-instructions

[yan12125]

A good suggestion. For most cases (including soundcloud), downloading happens in downloader/http.py.

[jaimeMF]

Note that until it gets implemented in the http downloader you may want to try aria2 with --external-downloader aria2c, it uses multiple connections which could help in your case.

[error39]

@yan12125 Thanks for your help, I will try to see if/how it's possible to implement this kind of functionality in http.py (I'm a python n00b, but it's always fun learning.. :) )

@jaimeMF It speeds up my download, but it opens multiple connections at the same time, so it's not exactly what I'm after (It increases the server load, which is one reason download accelerators are not so popular among server admins, and chances are they'll try to block those methods if possible) But nevertheless thanks a lot for the valuable tip! It might help others looking for a way to make downloads happen faster.

[Degru]

Something like this would be great! Perhaps an option so that if speed drops below a certain rate, re-start the download from the current point. This happens for me on Youtube. I download on a 400mbps connection and get 3MB/s downloads at times for some videos after the first couple hundred megabytes.

[ghost]

Hi. Seems to me that youtube just started doing this! Noticed today, wasn't present like 2 days ago.

First burst is at full speed (ie. 9.25MiB, not Mbps) then there are 2 bursts of 1.04 MBit/s each lasting 1 sec happening every 2.5 sec (equivalent of an average of 400Kbps/sec) - EDIT: according to nload.

EDIT2: first burst gets 0.5MiB of the file (or, 1.1% of a 45.78MiB file) - I'm assuming this is true for any file, although this particular one was only the audio part (a .webm) of a video.

Before today, it used to download at full speed constantly (ie. 100Mbps/sec).

EDIT:3: looks like if I restart it 5 times (ie. start youtube-dl, C-c after like 7 sec(ie. let it run just as long as the first half meg gets downloaded at full speed then C-c), repeat 5 times) it unlocks the full speed the 6th time. (same file above got transferred in 1 sec from 75% to 100% after this happened).

EDIT4: I've just retested edit3 on 3 new videos(but only the audio part, because that's what I needed) and there was only one repeat needed: ie. first try got stalled after first burst, I C-c, rerun youtube-dl, then it downloads the remainder at full speed.

here's full output for edit4

```bash $ downaudio https://www.youtube.com/watch?v=1PtC7D9euHU [youtube] 1PtC7D9euHU: Downloading webpage !! host= www.youtube.com [youtube] 1PtC7D9euHU: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] 1PtC7D9euHU: Extracting video information !! host= r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a [download] 6.7% of 7.43MiB at 9.70MiB/s ETA 00:00^C ERROR: Interrupted by user real 0m6.358s user 0m3.523s sys 0m0.163s $ downaudio https://www.youtube.com/watch?v=1PtC7D9euHU [youtube] 1PtC7D9euHU: Downloading webpage !! host= www.youtube.com [youtube] 1PtC7D9euHU: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] 1PtC7D9euHU: Extracting video information [download] Resuming download at byte 523264 !! host= r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a [download] 100% of 7.43MiB in 00:00 [ffmpeg] Correcting container in "As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a" [ffmpeg] Post-process file As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a exists, skipping real 0m5.602s user 0m4.045s sys 0m0.363s $ downaudio https://www.youtube.com/watch?v=trH6Uw_Rs-E [youtube] trH6Uw_Rs-E: Downloading webpage !! host= www.youtube.com [youtube] trH6Uw_Rs-E: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] trH6Uw_Rs-E: Extracting video information !! host= r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a [download] 5.5% of 9.09MiB at 9.71MiB/s ETA 00:00^C ERROR: Interrupted by user real 0m6.187s user 0m3.467s sys 0m0.157s $ downaudio https://www.youtube.com/watch?v=trH6Uw_Rs-E [youtube] trH6Uw_Rs-E: Downloading webpage !! host= www.youtube.com [youtube] trH6Uw_Rs-E: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] trH6Uw_Rs-E: Extracting video information [download] Resuming download at byte 523264 !! host= r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a [download] 100% of 9.09MiB in 00:00 [ffmpeg] Correcting container in "Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a" [ffmpeg] Post-process file Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a exists, skipping real 0m5.766s user 0m4.054s sys 0m0.411s $ downaudio https://www.youtube.com/watch?v=nKxjG9F0iw8 [youtube] nKxjG9F0iw8: Downloading webpage !! host= www.youtube.com [youtube] nKxjG9F0iw8: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] nKxjG9F0iw8: Extracting video information !! host= r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com [download] Destination: VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a [download] 6.0% of 8.27MiB at 9.67MiB/s ETA 00:00^C ERROR: Interrupted by user real 0m6.694s user 0m3.474s sys 0m0.183s $ downaudio https://www.youtube.com/watch?v=nKxjG9F0iw8 [youtube] nKxjG9F0iw8: Downloading webpage !! host= www.youtube.com [youtube] nKxjG9F0iw8: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] nKxjG9F0iw8: Extracting video information [download] Resuming download at byte 523264 !! host= r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com [download] Destination: VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a [download] 100% of 8.27MiB in 00:00 [ffmpeg] Correcting container in "VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a" [ffmpeg] Post-process file VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a exists, skipping real 0m5.734s user 0m4.097s sys 0m0.357s $ type downaudio downaudio is aliased to `time youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio' $ cat /etc/hosts|grep r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.210 r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com $ dig r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com r7.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.210 r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com $ cat /etc/hosts|grep r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com 145.236.72.15 r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com $ dig r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com r4.sn-h5bt5guxax3vh-c0q6.googlevideo.com. r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com 145.236.72.15 r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com $ cat /etc/hosts|grep r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com 145.236.72.83 r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com $ dig r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com r8.sn-h5bt5guxax3vh-c0qd.googlevideo.com. r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com 145.236.72.83 r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com ```

EDIT5: Just figured out that --socket-timeout 1 works flawlessly, especially since I've encountered this video which wouldn't decide to unlock full speed even after like 20 restarts! Thank you for this option, btw! Amazing! \ I say video but I'm only downloading the audio part. EDIT6: Also need --retries infinite or it will end after the default of 10. However, there is a jump from 60.8% to 100% even though the output seems to be of correct size: -rw-r--r-- 1 xftroxgpx xftroxgpx 28768063 08.01.2018 11:18 'VERY STABLE GENIUS-u-yLGFuu2dc.opus' I'll take that to mean that after 34 retries it unlocked full speed.

actual console output

```bash $ downaudio --socket-timeout 1 --retries infinite https://www.youtube.com/watch?v=u-yLGFuu2dc [youtube] u-yLGFuu2dc: Downloading webpage !! host= www.youtube.com [youtube] u-yLGFuu2dc: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] u-yLGFuu2dc: Extracting video information !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 1.8% of 27.89MiB at 9.79MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of inf)... [download] Resuming download at byte 523264 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 3.6% of 27.89MiB at 9.92MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 2 of inf)... [download] Resuming download at byte 1046528 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 5.4% of 27.89MiB at 9.89MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of inf)... [download] Resuming download at byte 1569792 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 7.2% of 27.89MiB at 9.86MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 4 of inf)... [download] Resuming download at byte 2093056 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 8.9% of 27.89MiB at 9.84MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 5 of inf)... [download] Resuming download at byte 2616320 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 10.7% of 27.89MiB at 9.90MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 6 of inf)... [download] Resuming download at byte 3139584 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 12.5% of 27.89MiB at 9.79MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 7 of inf)... [download] Resuming download at byte 3662848 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 14.3% of 27.89MiB at 9.61MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 8 of inf)... [download] Resuming download at byte 4186112 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 16.1% of 27.89MiB at 9.73MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 9 of inf)... [download] Resuming download at byte 4709376 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 17.9% of 27.89MiB at 9.85MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 10 of inf)... [download] Resuming download at byte 5232640 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 19.7% of 27.89MiB at 9.98MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 11 of inf)... [download] Resuming download at byte 5755904 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 21.5% of 27.89MiB at 9.87MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 12 of inf)... [download] Resuming download at byte 6279168 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 23.3% of 27.89MiB at 9.67MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 13 of inf)... [download] Resuming download at byte 6802432 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 25.1% of 27.89MiB at 9.83MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 14 of inf)... [download] Resuming download at byte 7325696 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 26.8% of 27.89MiB at 9.91MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 15 of inf)... [download] Resuming download at byte 7848960 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 28.6% of 27.89MiB at 9.77MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 16 of inf)... [download] Resuming download at byte 8372224 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 30.4% of 27.89MiB at 9.89MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 17 of inf)... [download] Resuming download at byte 8895488 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 32.2% of 27.89MiB at 9.86MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 18 of inf)... [download] Resuming download at byte 9418752 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 34.0% of 27.89MiB at 9.87MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 19 of inf)... [download] Resuming download at byte 9942016 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 35.8% of 27.89MiB at 9.97MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 20 of inf)... [download] Resuming download at byte 10465280 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 37.6% of 27.89MiB at 9.81MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 21 of inf)... [download] Resuming download at byte 10988544 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 39.4% of 27.89MiB at 9.88MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 22 of inf)... [download] Resuming download at byte 11511808 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 41.2% of 27.89MiB at 9.85MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 23 of inf)... [download] Resuming download at byte 12035072 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 42.9% of 27.89MiB at 9.84MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 24 of inf)... [download] Resuming download at byte 12558336 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 44.7% of 27.89MiB at 9.75MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 25 of inf)... [download] Resuming download at byte 13081600 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 46.5% of 27.89MiB at 9.93MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 26 of inf)... [download] Resuming download at byte 13604864 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 48.3% of 27.89MiB at 9.93MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 27 of inf)... [download] Resuming download at byte 14128128 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 50.1% of 27.89MiB at 9.80MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 28 of inf)... [download] Resuming download at byte 14651392 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 51.9% of 27.89MiB at 9.64MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 29 of inf)... [download] Resuming download at byte 15174656 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 53.7% of 27.89MiB at 9.78MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 30 of inf)... [download] Resuming download at byte 15697920 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 55.5% of 27.89MiB at 9.90MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 31 of inf)... [download] Resuming download at byte 16221184 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 57.3% of 27.89MiB at 9.87MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 32 of inf)... [download] Resuming download at byte 16744448 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 59.1% of 27.89MiB at 9.76MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 33 of inf)... [download] Resuming download at byte 17267712 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 60.8% of 27.89MiB at 9.88MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 34 of inf)... [download] Resuming download at byte 17790976 !! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com [download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm [download] 100% of 27.89MiB in 00:00 [ffmpeg] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.opus Deleting original file VERY STABLE GENIUS-u-yLGFuu2dc.webm (pass -k to keep) real 0m44.638s user 0m6.094s sys 0m1.016s $ type downaudio downaudio is aliased to `time youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio' $ cat /etc/hosts|grep r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com 145.236.72.12 r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com $ dig r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com r1.sn-h5bt5guxax3vh-c0q6.googlevideo.com. r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com 145.236.72.12 r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com $ type dig dig is hashed (/home/xftroxgpx/bin/dig) ```

Note that downloading the video itself, is not as bad: seems to go fullspeed for the first few seconds, then limited to 20Mbit/sec). It's the audio which gets stalled bad.

[ghost]

Partial workaround for automatically bypassing stalls while downloading the audio part from youtube, just add args: --socket-timeout 1 --retries infinite to youtube-dl (as per my edit5&6 above)

EDIT1: I guess it doesn't always work (or they were quick to update against the workaround above? xD EDIT3: actually, I can only surmise that different *.googlevideo.com servers have differing behaviours, also see my edits in my prev. post)

``` $ downaudio https://www.youtube.com/watch?v=rPEU2KJwY6Q [youtube] rPEU2KJwY6Q: Downloading webpage !! host= www.youtube.com [youtube] rPEU2KJwY6Q: Downloading video info webpage !! host= www.youtube.com !! host= www.youtube.com [youtube] rPEU2KJwY6Q: Extracting video information !! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com !! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm [download] 1.0% of 50.51MiB at 9.87MiB/s ETA 00:05[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of inf)... [download] Resuming download at byte 523264 !! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com !! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm [download] 2.0% of 50.51MiB at 9.99MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 2 of inf)... [download] Resuming download at byte 1046528 !! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com !! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm [download] 3.0% of 50.51MiB at 9.95MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of inf)... [download] Resuming download at byte 1569792 !! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com !! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com [download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm [download] 4.0% of 50.51MiB at 9.87MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 4 of inf)... [download] Resuming download at byte 2093056 !! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com ERROR: unable to download video data: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop. The last 30x error message was: Found real 0m9.234s user 0m3.877s sys 0m0.223s $ type downaudio downaudio is aliased to `time youtube-dl --socket-timeout 1 --retries infinite --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio' $ youtube-dl --version 2018.01.14 $ cat /etc/hosts|grep r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.208 r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com $ dig r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com r5.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.208 r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com $ cat /etc/hosts|grep r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.205 r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com $ dig r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com r2.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com 31.46.22.205 r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com ```

EDIT2: some bash workaround for edit1: $ (exit 1) || while test "$?" -ne "0"; do downaudio https://www.youtube.com/watch?v=rPEU2KJwY6Q; done

EDIT4: alright, I've decided to stop using youtube altogether, replacing it with reading books.

[sjgdk]

This is not a workaround but my own findings.

I downloaded NewPipe from F-droid and standalone apk Snaptube to test on this slow downloading issue on m4a and dash video 1080p.

Snaptude speed is like youtube-dl, slow.

But Newpipe downloading speed is superb. Not sure why, I couldn't really read any code now as a newbie in programming.

[zouhair]

It seems they are systematically throttling all DASH streams, I doubt there can be a solution client side.

[norcaljohnny]

Just something worth mentioning. If you load a video or audio directly into an html5 player, then download is instant. I am considering quickly creating a php page/js form to handle the dl and conversion of mp3. I believe it is faster than some other options I was left with.

[zouhair]

Even for DASH 1080p+ ?

[coolguruji]

First i thought my server provider is throttling my bandwidth speed but they told me - "No, we are not" Then i tried with multiple speedtest via command line and i saw that download speed is awesome of my server. Now i started to google about it on web. As i can say know, Youtube.com maybe throttling our requests and slowing it down. I download webm/m4a format and download speed is 80 to 250 kbps right now, however it should be 20 to 40 Mbps. using aria2c is not giving much boost.

Now the question is how can we fix it? Please suggest any idea!

[zouhair]

As far as I understand it (don't take my word for it) this is a server side problem. Youtube seems to be throttling everything but the "best" stream. As far as I know, there is not much we can do on the client side, and I hope to be wrong.

[ghost]

The download on Youtube is now very annoying. But I have an idea how to handle this. How about if youtube-dl includes a list of VPN servers that you would define yourself in a config file for the download?

[zouhair]

VPN won't solve a thing, this is a server side thing.

[ghost]

Sure it is a server problem. The idea is that youtube-dl fooled the server, the download would be downloaded by different clients, depending on the number of VPN servers that you defined in the config list

[porjo]

I'm seeing the same throttling with audio-only downloads (-f worstaudio) in the last week. Downloads that previously would take seconds now take upwards of several minutes, regardless of what Internet connection I use. As an earlier commentor suggested, using --external-downloader aria2c appears to work around this.