search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.23k stars 9.93k forks source link

[YouTube] Randomly slow youtube download speed #29326

Closed triplesixman closed 2 years ago

triplesixman commented 3 years ago

Checklist

Verbose log

root@server:~# youtube-dl https://youtu.be/8PecfdkEM2Y --source-address 64.31.22.34 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/8PecfdkEM2Y', u'--source-address', u'64.31.22.34', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-15-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] 8PecfdkEM2Y: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953531&ei=GzzLYMyiKojn8wSQ-42YBA&ip=64.31.22.34&id=o-AMDSXCc14P0ndQaRCkihVxb1SwdClDAhUbq8xP8nD2ss&itag=313&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&mh=e_&mm=31%2C26&mn=sn-ab5l6n67%2Csn-vgqsrnee&ms=au%2Conr&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=video%2Fwebm&ns=gA0RPx_S5d5eK1Qi3zH6xdcF&gir=yes&clen=202533160&dur=299.160&lmt=1617981974189982&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5532432&n=0B2xjorVelV0Xu6Hq&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAI7W7dT0pYaOQgxn1mHYX3js6NByrqiykD9fsPJs3kAXAiApHiHXVdDMO1k6OKyg2sAb1PMyMO1jfgtZV5R-7frcpw%3D%3D&sig=AOq0QJ8wRgIhAPdD-seXFXT-yOEqoIqCQfPnRqMLASvU8SbymG5TPpNhAiEA1pVFpS3hDYqTVe1ia5sDOi9RaPf3BCuT94XB-vICq_E='
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm
[download]  18.4% of 193.15MiB at 77.46KiB/s ETA 34:44Terminated

While the download was in progress, I ran the exact same command in another terminal, another folder, and the download was completed in a few seconds:

root@server:~/test# youtube-dl https://youtu.be/8PecfdkEM2Y --source-address 64.31.22.34 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/8PecfdkEM2Y', u'--source-address', u'64.31.22.34', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-15-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] 8PecfdkEM2Y: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953574&ei=RjzLYLyWBIHGhwak2IzwBg&ip=64.31.22.34&id=o-AFgzn9Zdn1KSGbuO09ZkpRma9GzWqWUApkavXXN93_F6&itag=313&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&mh=e_&mm=31%2C29&mn=sn-ab5l6n67%2Csn-ab5szne7&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=video%2Fwebm&ns=hoSqVT_3ust7ILej5iYoT40F&gir=yes&clen=202533160&dur=299.160&lmt=1617981974189982&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5532432&n=l6PFqwM4uREk1JKwP&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAJU5426qtqf6BwLiB48OKkcK_ATe_S9jDPYAVbttM7T1AiBoVGwb1ZBagaiUyKeVGLv562cloZeh5xBT2lFZx61gyQ%3D%3D&sig=AOq0QJ8wRAIgWSAyj1JyqoTHFWMdJ04gjcIDJ8tFryw5sNsf5soVaPQCIE81_29FoDoAOeRv2_hdcnBi2-4XxvlGvbX9YNajLi6y'
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm
[download] 100% of 193.15MiB in 00:04
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953574&ei=RjzLYLyWBIHGhwak2IzwBg&ip=64.31.22.34&id=o-AFgzn9Zdn1KSGbuO09ZkpRma9GzWqWUApkavXXN93_F6&itag=251&source=youtube&requiressl=yes&mh=e_&mm=31%2C29&mn=sn-ab5l6n67%2Csn-ab5szne7&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=audio%2Fwebm&ns=hoSqVT_3ust7ILej5iYoT40F&gir=yes&clen=5324207&dur=299.201&lmt=1617980855722369&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5531432&n=l6PFqwM4uREk1JKwP&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIqCBWf7PmHH8y1wnG8QvB-0vxKzRG26qCAIWdgOAT1PAiEAwTSl4J0e9L7emiYUDKV_YjfApo2gchge3iVfrYH76lo%3D&sig=AOq0QJ8wRQIgODhcEL0uT0u1nXP41IARrB63CfmpDmUzl6HhPwrXOTwCIQD-OUE152N7yzYXcgU_tAPCP0YdRdfVyFlHE4kIYyyoew=='
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm
[download] 100% of 5.08MiB in 00:00
[ffmpeg] Merging formats into "Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm' -i 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm' -c copy -map '0:v:0' -map '1:a:0' 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.temp.webm'
Deleting original file Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm (pass -k to keep)
Deleting original file Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm (pass -k to keep)

Description

Hello,

Important: The problem is random, maybe 1 chance on 8 to produce it. You have to download several videos in a row (about 10) to notice it.

Since a few weeks, randomly a youtube video can be slowed down to 48 Kio/s, so it takes 5-10 minutes to download a short video of 5 minutes instead of 4, 5 seconds, often the download does not succeed and stops after a few minutes.

This happens on several servers, several internet providers as well as with my private connection.

I even managed to launch a second download in parallel when the first one was taking time, the second one finished in 1 seconds, the first one in 5 minutes and was interrupted before the end. Same video, same connection, same command. (tested only with ipv4 because I don't have ipv6 on my servers or with my internet connection).

Attached is an extraction of the results with --dump-pages. ko.txt ok.txt

lunainvictum commented 3 years ago

Dont know if it helps, but since downloading with youtube-dl didnt work anymore (because of the speed limit), i started 2 days ago using clipgrab that i always used on windows.

I downloaded the whole rambalac channel with clipgrab 3.9.6, hundreds of high quality 4k-8k videos, over 1,2TB. And had no single speed drop down. Non stop 110MB/s (1gig fiber).

And thats wonder me. Because if i try the same with youtube-dl, i get speed limited.

And what wonder me the most, is that clipgrab use youtube-dl for downlaoding. 2021-06-06 to be exact.

Maybe it helps you, if you guys conntact the guy from clipgrab why it is working with his tool? I know him from linuxforen.de, because 10-12 years ago, he was/is a user from the forum, wich presented this tool and asked for help and so on.

tfdahlin commented 3 years ago

I believe I'm successfully calculating the updated value of n in my experimental branch for pytube if anybody wants to reference the code. However, this calculated value of n is different from the value present on stream URLs with the ratebypass parameter, which seems a little unusual. The calculated value of n doesn't seem to lead to errors while downloading.

Furthermore, when I substituted the value of n from the URL with the ratebypass parameter into other URLs, there don't seem to be any errors while downloading.

I have not yet tested whether this solves the throttling problem. Any help with doing so is greatly appreciated.

RhFortuna commented 3 years ago

I don't know if this can help anything, but ETA was from the start about the size of the video itself (~ one hour). "best" in this case is only 360p. youtube-dl 2021.06.06 - Xubuntu 20.04.2

rhfor@rhfor-PC:~/Downloads$ youtube-dl -f best https://youtu.be/Lg8ONTGiUOc --sub-lang pt-BR --sub-format srt
[youtube] Lg8ONTGiUOc: Downloading webpage
[youtube] Lg8ONTGiUOc: Downloading MPD manifest
[download] Destination: Cosmos com Carl Sagan - Episódio 01 - Shores of Cosmic Ocean-Lg8ONTGiUOc.mp4
[download]  68.8% of 165.35Mib at 43.61KiB/s ETA 20:11

PS: I downloaded the episode 2 in 10 seconds.

Youtube-dlThrottled commented 3 years ago

this youtube download throtteling has to stop. I bought a new hdd to watch my youtube offline and what happened? downloadspeed at 70 kbs!!! That takes decades. It is terrible. Everyone has the right to watch his videos offline. It is the only way to download playlists and more, automatically.

Youtube is also able to block youtube-dl completely. Youtube already did this during the music copyright thing. Didn't matter which youtube-dl version you used, it didn't work, blocked nothing worked. It can be done anytime by google.

ghost commented 3 years ago

this is so annoying don't you think?

ghost commented 3 years ago

i just tried downloading a video and this happened:

[youtube] Downloading login page
[youtube] Looking up account info
WARNING: Unable to look up account info: HTTP Error 400: Bad Request
[youtube] P9vmxl4sXQM: Downloading webpage
[youtube] P9vmxl4sXQM: Downloading MPD manifest
[download] Destination: /Users/[ommited]/Data/downloads/YouTube/NA _NA/NA- Shinigami Eyes By Toddsson _P9vmxl4sXQM.f137.mp4
[download]  34.0% of 38.14MiB at 20.11KiB/s ETA 21:20

this is getting worse and worse

tfdahlin commented 3 years ago

I believe I'm successfully calculating the updated value of n in ~my experimental branch for~ pytube if anybody wants to reference the code. However, this calculated value of n is different from the value present on stream URLs with the ratebypass parameter, which seems a little unusual. The calculated value of n doesn't seem to lead to errors while downloading.

Furthermore, when I substituted the value of n from the URL with the ratebypass parameter into other URLs, there don't seem to be any errors while downloading.

~I have not yet tested whether this solves the throttling problem. Any help with doing so is greatly appreciated.~

A pytube user tested the changes I made to calculate n based on the js in the base.js file, and did not see a slowdown in 100+ videos. They ran into other errors, but I believe those errors are unrelated to the new code for ciphering n. It looks like there are ~9-10 unique function calls in the javascript for computing n, but most of them are pretty simple.

The weird part was modifying the list that was passed as a reference to the function calls to get it to calculate correctly. Effectively, the way that js code works is as follows:

  1. Split n into an array of characters.
  2. Create an array of integers, strings, anonymous functions, references to n, and null (calling this array c for clarity).
  3. Replace all instances of null in c with references to c itself.
  4. Call the functions in c with arguments of different elements of c (notably, c itself gets left- and right-shifted by some of these functions. This is why the references to c are placed in the array). These calls are all made sequentially within the try section of the try-catch block.
  5. Because c has references to the array made from n, the actual array of n gets changed and can be un-split into a new string value of n.

I don't have the time to figure out how to incorporate this into youtube-dl myself, but maybe somebody else can use this info to write a fix.

shoxie007 commented 3 years ago

A pytube user tested the changes I made to calculate n based on the js in the base.js file, and did not see a slowdown in 100+ videos. They ran into other errors, but I believe those errors are unrelated to the new code for ciphering n. It looks like there are ~9-10 unique function calls in the javascript for computing n, but most of them are pretty simple.

The weird part was modifying the list that was passed as a reference to the function calls to get it to calculate correctly. Effectively, the way that js code works is as follows:

1. Split `n` into an array of characters.

2. Create an array of integers, strings, anonymous functions, references to `n`, and `null` (calling this array `c` for clarity).

3. Replace all instances of `null` in `c` with references to `c` itself.

4. Call the functions in `c` with arguments of different elements of `c` (notably, `c` itself gets left- and right-shifted by some of these functions. This is why the references to `c` are placed in the array). These calls are all made sequentially within the `try` section of the `try-catch` block.

5. Because `c` has references to the array made from `n`, the actual array of `n` gets changed and can be un-split into a new string value of `n`.

I don't have the time to figure out how to incorporate this into youtube-dl myself, but maybe somebody else can use this info to write a fix.

Thank you SOO much @tfdahlin for your efforts. I'm gonna try my hand at this. Glad that it can be implemented in Python. But does your code have any dependencies on packages which are not part of Youtube-DL? Anything for decrypting the Javascript for example?

danny-wu commented 3 years ago

But does your code have any dependencies on packages which are not part of Youtube-DL?

pytube is dependency-free.

tfdahlin commented 3 years ago

But does your code have any dependencies on packages which are not part of Youtube-DL?

danny-wu is correct, pytube has no package dependencies.

Anything for decrypting the Javascript for example?

A while back I wrote a parser to replace the regex that was being used to identify the bounds of javascript objects, and was able to use that for parsing the different pieces of the javascript. From there, I use regexes to identify individual functions within the arrays, and map them to the relevant ciphering function

StarSkyCaster commented 3 years ago

Hi, all. I solved this issue by using https://github.com/iTaybb/pySmartDL My example code:

import time
from pySmartDL import SmartDL
from urllib3 import PoolManager
from os.path import exists

LINK = "ANY_LINK"
SAVE_AS = "ANY_FILENAME"

try:
    pool = PoolManager()
    response = pool.request(
        "GET", LINK, preload_content=False)
    file_size = int(response.headers.get(
        "Content-Length"))
    # `4009252` is optimal for 100Mbit/s
    threadsVideo = int(file_size / 4009252)
except Exception as e:
    print("Exception", e)
    threadsVideo = 50
print("threadsVideo", threadsVideo)
obj = SmartDL(LINK, SAVE_AS,
              threads=threadsVideo)
obj.start(blocking=False)
while not obj.isFinished():
    # Doing something ...
    # print("Speed: %s" % obj.get_speed(human=True))
    # print("Already downloaded: %s" % obj.get_dl_size(human=True))
    # print("Eta: %s" % obj.get_eta(human=True))
    # print("Progress: %d%%" % (obj.get_progress()*100))
    # print("Progress bar: %s" % obj.get_progress_bar())
    # print("Status: %s" % obj.get_status())
    # print("\n"*2+"="*50+"\n"*2)
    time.sleep(0.5)

if obj.isSuccessful():
    print("downloaded file to '%s'" % obj.get_dest())
    print("download task took %ss" % obj.get_dl_time(human=True))
    print("File hashes:")
    print(" * MD5: %s" % obj.get_data_hash('md5'))
    print(" * SHA1: %s" % obj.get_data_hash('sha1'))
    print(" * SHA256: %s" % obj.get_data_hash('sha256'))
else:
    print("There were some errors:")
    for e in obj.get_errors():
        print(str(e))
secondsBeforeTempVideoSaved = 0
while secondsBeforeTempVideoSaved < 10:
    if not exists(SAVE_AS):
        print("Video not saved, waiting 500ms ...")
        time.sleep(0.5)
        secondsBeforeTempVideoSaved += 0.5
    else:
        break
print("Video downloaded")
ghost commented 3 years ago

can confirm it's still an issue image

tzarebczan commented 3 years ago

It went away for us for 2 weeks and is now back. Any other updates? Would be willing to put a bounty in crypto for this.

rebane2001 commented 3 years ago

If somebody wants to fix this in the next couple of days (so we have time to archive unlisted videos before the 23rd), I'd also be willing to drop a bit into @tzarebczan's bounty pot.

liamengland1 commented 3 years ago

try the workaround in yt-dlp. @tzarebczan @rebane2001

yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K

see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07

danny-wu commented 3 years ago

+1, no issues with yt-dlp's workaround. Archiving thousands of videos fine without throttle.

I am switching to dlp now, the additional features and active community development has won me over.

Just do the same. It's mostly compatible, was a drop in replacement for me.

On Wed, 14 Jul 2021, 3:08 am LE, @.***> wrote:

try the workaround in yt-dlp. @tzarebczan https://github.com/tzarebczan @rebane2001 https://github.com/rebane2001

yt-dlp "your video link here" --extractor-args youtube:player_client=android --throttled-rate 100K

see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-879256177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCF6E2LUZSNML7TKBDTXRXHNANCNFSM463R5FEQ .

-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva  https://facebook.com/canva  https://au.linkedin.com/company/canva  https://instagram.com/canva

tzarebczan commented 3 years ago

Glad to hear that @danny-wu . Any issues with formats or download failures otherwise?

I've also been in contact with the DLP dev on Discord, he's great. Sent him a nice tip via paypal last week for his support and continued efforts. Anyone using it should consider it also: https://ko-fi.com/pukkandan

shoxie007 commented 3 years ago

It went away for us for 2 weeks and is now back. Any other updates? Would be willing to put a bounty in crypto for this.

That's the spirit!! A bounty would work wonders.

@tfdahlin appears to have solved the problem with his pytube project. I'm just waiting for someone to implement in youtube-dl. If you don't mind using another repo, then use pytube to download the actual video files. https://github.com/pytube/pytube/issues/1033

MisterSheeple commented 3 years ago

try the workaround in yt-dlp. @tzarebczan @rebane2001

yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K

see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07

I can vouch for this workaround as well. Switched to using it a few hours ago and already a mind-blowing difference. Downloading is so much more efficient now. Perfect!

ghost commented 3 years ago

try the workaround in yt-dlp. @tzarebczan @rebane2001 yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07

I can vouch for this workaround as well. Switched to using it a few hours ago and already a mind-blowing difference. Downloading is so much more efficient now. Perfect!

Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?

spschoen commented 3 years ago

Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?

Likely because it's not a proper solution. It's a workaround until someone implements decoding the n parameter, which is the real problem.

Evernow commented 3 years ago

Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?

Likely because it's not a proper solution. It's a workaround until someone implements decoding the n parameter, which is the real problem.

A workaround that works is better than no solution at all imo, but I suppose anyone who thinks that way can just use the fork, it's a drop-in replacement after all.

ghost commented 3 years ago

Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?

Likely because it's not a proper solution. It's a workaround until someone implements decoding the n parameter, which is the real problem.

I see, that makes sense

skwolvie commented 3 years ago

I have a 100 MBPS connection but the fix with y_dl is downloading at 1 MBPS. 

yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K

If i change the rate to something like 500k it retries and dies.

The fix is amazing it is better than 76 KBPS rate for some random videos with youtube_dl. Is there any other way to increase this?

thanks!

rautamiekka commented 3 years ago 
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K

yt-dlp != yt-dl.

liamengland1 commented 3 years ago

@skwolvie report it here https://github.com/yt-dlp/yt-dlp/issues/new

Obscure2020 commented 3 years ago

I can confirm I am experiencing this same issue. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.

danny-wu commented 3 years ago

Just use yt-dlp. My setup is more complicated but it was a drop in replacement for youtube-dl.

yt-dlp is a more actively maintained project at this point.

On Sun, 25 Jul 2021, 2:48 pm Obscure2020, @.***> wrote:

I can confirm I am experiencing this same issue, intermittently. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-886146212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCGHXMVAWILAH7N5CLTZOJQXANCNFSM463R5FEQ .

-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva  https://facebook.com/canva  https://au.linkedin.com/company/canva  https://instagram.com/canva

danny-wu commented 3 years ago

Don't worry if you have complicated batch scripts. Just switch "youtube-dl" to "yt-dlp" and keep everything else the same. It's been working flawlessly for me.

On Sun, 25 Jul 2021, 5:57 pm Danny Wu, @.***> wrote:

Just use yt-dlp. My setup is more complicated but it was a drop in replacement for youtube-dl.

yt-dlp is a more actively maintained project at this point.

On Sun, 25 Jul 2021, 2:48 pm Obscure2020, @.***> wrote:

I can confirm I am experiencing this same issue, intermittently. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-886146212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCGHXMVAWILAH7N5CLTZOJQXANCNFSM463R5FEQ .

-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva  https://facebook.com/canva  https://au.linkedin.com/company/canva  https://instagram.com/canva

diyaa59 commented 3 years ago

I have this too and I find if I set youtube-dl to resume and restart the script it would regain the speed!

bani6809 commented 3 years ago

unfortunately development on this project seems to have stalled. yt-dlp project is aggressively addressing this throttling problem.

Obscure2020 commented 3 years ago

Does yt-dlp have the same -U command-line auto update feature?

Evernow commented 3 years ago

Does yt-dlp have the same -U command-line auto update feature?

Yes.. Please read the documentation before asking questions.

Butterfly-Dragon commented 3 years ago

to be frank i have noticed that all i have to do to stop the 50 kb downloads for a day or two is visit the youtube page and redownload the cookies from there. So it's probably a problem of how the cookies are updated in the cookie file.

noembryo commented 3 years ago

unfortunately development on this project seems to have stalled. yt-dlp project is aggressively addressing this throttling problem.

Yes, but yt-dpl does not seem to support Python 2, so it seems I'm stuck with youtube_dl.. I manually added an exception to my projects, to go around this problem for now.

Evernow commented 3 years ago

Yes, but yt-dpl does not seem to support Python 2, so it seems I'm stuck with youtube_dl

Sounds like you have much bigger problems if you have to be staying on Python 2.

noembryo commented 3 years ago

Sounds like you have much bigger problems if you have to be staying on Python 2.

There are a lot of libraries that are not supported in Python 3 Also, big frameworks like Qt also stopped supporting various things that I need, in their updated versions.. Its a nightmare to support complicated apps while everybody changes compatibilities under your feet.. :angry:

woj-tek commented 3 years ago

OT: @noembryo Isn't virtenv a thing? https://python.land/virtual-environments/virtualenv

noembryo commented 3 years ago

OT: @noembryo Isn't virtenv a thing? https://python.land/virtual-environments/virtualenv

Well, my problem is not how to install different versions of Python/libraries. The problem arises when an app that uses something from Qt 4 that does not exists in Qt 5/6 (e.g. like Phonon) also uses a library that suddenly stops supporting Python 2.. :skull:

.. but lets not hijack the thread anymore.. Thank you for your time. :+1:

OmniSexualCofeeBean commented 3 years ago

Since there haven't been any takers, I thought to poke at it and write down what I found (while writing a simple transcoder for another scripting language and for myself - it already works, but it'd count as an external dependency). I will explain what Youtube's n token function does. This is not a patch and is only of interest to other devs if they wish to implement it themselves or improve the current interpreter. But if you wanted to know what useless job Google developers get paid for, continue reading!

Here's your keygen music while reading. Sort of.

Overview of the control flow (already known):

  1. The Youtube API provides you with n - your video access token
  2. If their new changes apply to your client (they do for "web") then it is expected your client will modify n based on internal logic. This logic is inside player...base.js
  3. n is modified by a cryptic function
  4. Modified n is sent back to server as proof that we're an official client. If you send n unmodified, the server will eventually throttle you.

This cryptic function is currently named "lha" at line 1162 of the current player. My explanation is based on this player. The function code is randomly generated with each player release. It consists of:

  1. input (argument "a"), takes n as string
  2. b = a.split("") this is where n will be modified letter by letter
  3. c - the really long array that's supposed to confuse you
  4. a lot of calls into random indexes of c[]. This is where magic happens
  5. return: new value of n that holds the key to Narnia. If anything goes wrong, an enhanced_except_ is returned along with original n
    1. This is interesting and did not exist in very early players. The developer who created this ran into problems and needed a way to debug them. They came up with the idea to return all values needed for debugging: n and the seed needed to generate the cryptic function: "enhanced_except_nJMB4-z-_w8_" + original_n - the random base64 letters are the parameters used by the code generator to create this function. Seed = nJMB4-z- (changed with each player) and w8 is probably the generator version (hadn't changed yet).

The c array contains 3 references to itself and 3 references to b. The b's are assigned at the time of array creation, the c's are filled in where the placeholder null was placed (indexes 3,4,9). Three different references to the same variable are supposed to make correlation harder? They didn't. Idk what's that useful for.

(4) Function calls look like this: c[39](c[3], c[22]) where:

What are the long numbers for? Like 1222607021 and -1989164346 - these are indexes into either b/c: simply something like 1222607021 % 52 = 25 aka c[25]

If the argument is a string, it is used for "translateA/B" functions (see below). Nothing is eval'ed or anything (even if you see javascript keywords)

Here's a visual representation (for player 2840754e, Aug 4/5th 2021) of what the shuffling of data looks like:

diag-4x

Arrows track objects of c between shuffles. If all arrows are straight on this diagram, it means a shuffle on b was performed instead (The above diagram describes this order: CCC, BBB, C, BB, C, BB, CC, B, CCCCC, B, CCCCC, BB, CCC, BBBBB). There're only 3 kinds of shuffle operations: swap two elements, reverse entire array and move all elements by E positions.

Now it's time to explain what kind of shuffle functions there are.

As you can see some functions are implemented TWICE with different code. It is beyond me why anyone would do this. I really hope it was a "puzzle task" for an intern and not a real attempt.

Alright, so you call some simple functions that shuffle both c and b, nothing complicated. While you could incrementally deduce and reduce the "randomized" index values inside the try-catch block without executing any code, the functions translateA and translateB require you to execute their code (either interpreter or reimplementation).

However you do NOT need to implement the switch-case statement. Whoever created this Rube-Goldberg machine was not as smart as they imagine. Let's take a look at the full function code for "translateA/B"

function(d, e) {
    for (var f = 64, h = []; ++f - h.length - 32;) {
        switch (f) {
            case 91:
                f = 44;
                continue;
            case 123:
                f = 65;
                break;
            case 65:
                f -= 18;
                continue;
            case 58:
                f = 96;
                continue;
            case 46:
                f = 95
        }
        h.push(String.fromCharCode(f))
    }
    d.forEach(function(l, m, n) {
        this.push(n[m] = h[(h.indexOf(l) - h.indexOf(this[m]) + m - 32 + f--) % h.length])
    }, e.split(""))
}

Here's the equivalent function with my comment:

function(d, e) {
// look, there's really no need for overengineering even if your IQ is below average
// and you can't write iterators to generate this, like I show below:
    h = ["0", "1" ... "a", "b", ... "A", ... "Y", "Z", "-", "_"]; // shortened array for readability
    d.forEach(function(l, m, n) {
        this.push(n[m] = h[(h.indexOf(l) - h.indexOf(this[m]) + 64) % h.length])
    }, e.split(""))
}
  1. At the end of the for-loop, f is always = 96
  2. IF and only IF there's a case 65: in the switch then the array is base64 in the following order: 0aA-_, otherwise it is Aa0-_ (A-Za-z0-9-_)
  3. Since m increases while f-- decreases and -32 remains constant, the expression + m - 32 + f-- can be replaced with a constant +64
  4. I don't wish to know how many hours/days the developer wasted to create and debug this useless control flow.

The translateA/B functions modify both N and themselves (this) hence modifying the LUT hence requiring an interpreter, I do not think it is possible to reduce this one (unlike everything else).

At the end of all these, in essence, trivial transformations, you get the new N. That's all this long-winded function does. I hope you had fun reading <3

The problem is the reimplementation of youtube-dl's basic interpreter because its current approach is a dead-end: I tried in the other scripting language and it works... with ever more hacks applied. Trust me and start with a basic parser. Or use a library, that's what they're for ;)

PS: If you want to have fun with Google engineers, let their cryptic function deliberately "crash" by sending the "enhanced_except" message to their endpoint. They track these messages to react to bugs. Why not send them a message? :sunglasses: Here's the command to get the current "enhanced_except" marker:

curl -L "https://youtube.com/$(curl -L https://youtube.com/ | grep -oP 's/player/([^/]+)/player_ias\.vflset/[\w_]+/base\.js' | head -n 1)" | grep -oP 'enhanced_except[\w_-]+'

Then all you need to do is modify and repeat the /videoplayback? call from F12 (Browser's Network tab): Replace value of &n with enhanced_except_..._YourMessage (don't use special symbols)

byebye~

PPS: Bonus containing 81 base.js players - collection-of-81-youtube-players.zip

shoxie007 commented 3 years ago

@OmniSexualCofeeBean Outstanding analysis! So I take it that Youtube-DL as it is currently designed cannot be modified/extended to accommodate this new change. What other package can one use, which Youtube-DL can import, to outsource this decryption algorithm to?

If you can come up with a complete working solution for us (maybe your own fork of youtube-dl?), @tzarebczan has put up a crypto bounty for anyone who cracks the problem. I'd be willing to add a small amount to this. I don't have much in my Bitcoin wallet. He didn't state an amount, but whatever it is, you can claim it if you can solve this problem for us. We would all be very grateful. Would it be worth your while?

NarutoUA commented 3 years ago

@shoxie007 youtube-dl has JSInterpreter which is used in signature decryption. The same can be done for "n" parameter

shoxie007 commented 3 years ago

@shoxie007 youtube-dl has JSInterpreter which is used in signature decryption. The same can be done for "n" parameter

If you read @OmniSexualCofeeBean 's analysis, the decryption happens at random DURING the streaming. The n value doesn't necessarily stay the same. Will JSInterpreter be able to handle this? I don't know if there is anything in youtube-dl that can intervene during a download and re-send a request with a different n value.

OmniSexualCofeeBean commented 3 years ago

What other package can one use, which Youtube-DL can import, to outsource this decryption algorithm to?

@shoxie007 any JS interpreter whether NodeJS or anything smaller. Alternatively one of the available Python JS interpreters. The problem is youtube-dl's own stance against dependencies because I do not see any other reason to write an own interpreter (like this basic JSInterp, but Python 2 & 3 support too). Surely someone may decide to rewrite it from scratch to expand the needed functionality. Actually my entire analysis is irrelevant if someone decided to use an external interpreter. My only goal was to understand their (flawed) logic.

Just to reiterate: youtube-dlp uses a different API call that currently does not require n manipulation.

decryption happens at random DURING the streaming. The n value doesn't necessarily stay the same.

No, it only happens once after getting the n value from the server. I don't know what led you to this idea. The following only meant that the lha function code (which is called to modify n) is changed with each different base.js player.

The function code is randomly generated with each player release.

shoxie007 commented 3 years ago

@OmniSexualCofeeBean Thanks for the clarification. I saw the arbitrary shuffling of data (CCC, BBB, C, BB, C, BB etc) and assumed that it would need a new n when this happens. A little confused. I am NOT adept with Javascript and the inner workings of streaming and media players. Hence the reason I haven't come up with a solution myself.

Butterfly-Dragon commented 3 years ago

dude... 😅 omni clearly said the shuffling happens before...

He just gave us a visual representation of a "really long array the is supposed to confuse you" because he likes to humour us and show us some of the needless passages that this silly function does. That's not a "visual" of the "stream".

i mean, i find it quite funny too: several passages clearly show the entire array stays completely untouched. 😂 That's some high level "nothing" that wastes processor time.

In the end what we have is an original value, divided, that goes through some code which rearranges it around. And then gets sent back to the origin.

That is all.

gatecrasher777 commented 3 years ago

FWIW, I published this innertube library yesterday. Its for nodejs, though. It does the n transform, url decipher & sapisid hashing (for logged-in accounts) by extracting the functions from the yt player. Over the past month I have tested it on 1000's of videos with no slowdowns. I know its not a fix for this project, but it may help some frustrated by this issue.

shoxie007 commented 3 years ago

@gatecrasher777 Many thanks for your efforts and the share. How would I get Youtube-DL to use it? Give us a rough idea, if you please.

In the ReadMe you state "To get the most from ytcog, all you need is an active age-verified YouTube account." Can your solution work without logging in? Will it at least download non-age-restricted videos? I would NOT want to download 1000 videos or more with a logged in account. I wouldn't want high bandwidth use and multiple automated requests associated with my account, for fear of what can happen.

gatecrasher777 commented 3 years ago

@shoxie007 - It works for most things without supplying a cookie. It recognizes that you are not logged in, though. Because of that it will likely give 403 on age-restricted video and it might start giving 429 if you use it a lot (although I have never seen a 429 on innertube requests, logged in or out). The test examples as they are don't use a cookie, and still work fine.

I really can't say how youtube-dl could use it. I'm planning to release a cli version in the future. But of course, it is only a youtube solution. youtube-dl does so much more.

triplesixman commented 3 years ago

After 2 months and 100 comments on this post, I switched to yt-dlp (https://github.com/yt-dlp/yt-dlp) a few weeks ago and I must say that everything works perfectly.

Quite surprising that the problem has been present for at least 4 months and is still ongoing, while youtube remains the most used platform by youtube-dl.

It's also surprising that we didn't get any comment from @dstftw .

Nevertheless, I don't close this post for those who are interested. Cya and thanks to the participants for their research work.