[youtube] Download speeds still throttled for some clients after range fix

superislam commented 1 year ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2023.02.17 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

download is slow when downloading as audio and fast as video

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', '-x', '--audio-quality', '0', '--add-metadata', 'https://www.youtube.com/watch?v=pJ0auP7dbcY']
[debug] Encodings: locale cp1256, fs utf-8, pref cp1256, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19044-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2022-11-03-git-5ccd4d3060-full_build-www.gyan.dev (setts), ffprobe 2022-11-03-git-5ccd4d3060-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[youtube] Extracting URL: https://www.youtube.com/watch?v=pJ0auP7dbcY
[youtube] pJ0auP7dbcY: Downloading webpage
[youtube] pJ0auP7dbcY: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] pJ0auP7dbcY: Downloading 1 format(s): 251
[debug] Invoking http downloader on "https://rr2---sn-uxaxjvhxbt2u-xhtz.googlevideo.com/videoplayback?expire=1677815867&ei=2xsBZMmQEYqrxwK3x7SQDQ&ip=156.209.131.29&id=o-AP6fDpGS2eg_Bpej8argoWMtHOiqLzZKBxGxsndnCgwr&itag=251&source=youtube&requiressl=yes&mh=fD&mm=31%2C29&mn=sn-uxaxjvhxbt2u-xhtz%2Csn-aigl6ney&ms=au%2Crdu&mv=m&mvi=2&pl=19&initcwndbps=191250&spc=H3gIhkkFhilEy37sYHuXiZnT_lMxF-k&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=143903469&dur=10988.181&lmt=1674553794508873&mt=1677793993&fvip=1&keepalive=yes&fexp=24007246&c=ANDROID&txp=5532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgM2Qu7Ls0Opq_HeGnsFCLfyVX4XY5BSuDoRiRAky6GrsCICR96HrEqa5CL66bTy0obVBQUfXX2YeGL2pJfx6fbY67&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhALwiU8fuCUDbk0OBCW9E-BFf_keBgT-yAzRRKat9DlTgAiAGLjZQ0zz8xIcZZ6Sc69t7DdMgkxY08pAU7LxFr8q3Ew%3D%3D"
[download] Resuming download at byte 2195705
[debug] File locking is not supported. Proceeding without locking
[download] Destination: كيف تنجح العلاقات مع ياسر الحزيمي ｜ بودكاست فنجان [pJ0auP7dbcY].webm
[download]   1.9% of  137.24MiB at   25.28KiB/s ETA 01:30:51
ERROR: Interrupted by user
't' is not recognized as an internal or external command,
operable program or batch file.

vphantom commented 1 year ago

Interesting, I'm stuck roughly around 25 KiB/s too today, especially for format 251. 🤔

dude22072 commented 1 year ago

Also been having issues with all youtube audio formats the past two days. Running a modified version of TheFrenchGhosty's archive script; comments, live chat, video formats, etc all download at a decent speed but audio is always under 30KiB/s despite speedtest showing me as above gigabit download speed.

gamer191 commented 1 year ago

duplicate of #6369 It will be fixed in the next release According to https://github.com/yt-dlp/yt-dlp/issues/6369#issuecomment-1450068901

Until then, you have many options, including using Daily builds or PiP installing from master

Vangelis66 commented 1 year ago

If I may just step in 😉 ...

While it might initially appear to the devs as indeed a duplicate of #6369, it actually isn't; most evil Google are monitoring this repo and have now come up with their "own fix", to nullify yt-dlp's committed fixes...

The latest yt-dlp daily build is 2023.03.01.19419; this was compiled from yt-dlp snapshot https://github.com/yt-dlp/yt-dlp/commit/5b28cef72db3b531680d89c121631c73ae05354f, which includes the "fixes" for #6369, 5038f6d713303e0967d002216e7a88652401c22a + f34804b2f920f62a6e893a14a9e2a2144b14dd23 ; yet:

yt-dlp -vf 140 "p7FCgw_GlWc" => 

[debug] Command-line config: ['--ffmpeg-location', '..\\FFmpeg', '--downloader-args', 'ffmpeg:-v 8 -stats', '-vf', '140', 'p7FCgw_GlWc']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2023.03.01.19419 [9b7cf00] (win_x86_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6 (setts), ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Extractor Plugins: AGB+NSIG (YoutubeIE)
[debug] Plugin directories: ['<redacted>\\yt-dlp\\yt-dlp-plugins\\YTAgeGateBypass.zip\\yt_dlp_plugins', '<redacted>\\yt-dlp\\yt-dlp-plugins\\YTNSigProxy.zip\\yt_dlp_plugins']
[debug] Loaded 1782 extractors
[youtube+AGB+NSIG] Extracting URL: p7FCgw_GlWc
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading webpage
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading android player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded client config
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading player 7862ca1f
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded player API JSON
[debug] [youtube+AGB+NSIG] Extracting signature function js_7862ca1f_105
[debug] Loading youtube-sigfuncs.js_7862ca1f_105 from cache
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig oPGW9c82dzKHz-U => 0-VAXYzMO_3yT
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig wR-57cpazAlyJ0k => JqOKDJTXM_pG8
[debug] [youtube+AGB+NSIG] Extracting signature function js_7862ca1f_101
[debug] Loading youtube-sigfuncs.js_7862ca1f_101 from cache
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] p7FCgw_GlWc: Downloading 1 format(s): 140
[debug] Invoking dashsegments downloader on "https://rr2---sn-4vguioxu-n3bl.googlevideo.com/videoplayback?expire=1677820864&ei=YC8BZPTXEISp1wLUwoK4Cg&ip=<redacted>&id=o-AN-mV5Wv9YUp9ezgeFxbuqle6Q96XRuvc6TGWrYpJyF6&itag=140&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-4vguioxu-n3bl%2Csn-nv47lnsk&ms=au%2Crdu&mv=m&mvi=2&pl=18&gcr=gr&initcwndbps=693750&vprv=1&mime=audio%2Fmp4&ns=f61flAuXtRGkA8HpLjYOoDQL&gir=yes&clen=10309815&dur=636.992&lmt=1637451611628671&mt=1677798791&fvip=4&keepalive=yes&fexp=24007246&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=5532434&n=JqOKDJTXM_pG8&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgQKzJR9gDv3-gj4E7c-ShxYqHSKT4rWQYptoFmAnWIysCIQDYsFq_cgVkIJbYw_BOpr66HJoPo1B5SAMQ5VO8x04fgw%3D%3D&sig=AOq0QJ8wRAIgMwdZJilDcFCVw7v1tBifs_obb7seo0K1h40l5QrEbLECIDrfO9Y5ESKlRjZXLeEIJ2OkNtoZ-UJ5uE7exVvgnYEw"
[dashsegments] Total fragments: 1
[download] Destination: Kanye West - Famous [p7FCgw_GlWc].m4a
[debug] File locking is not supported. Proceeding without locking
[download]   2.2% of ~   9.83MiB at   50.45KiB/s ETA 03:15 (frag 0/1)
ERROR: Interrupted by user
Terminate batch job (Y/N)?

With that very same build, some 6 hours ago, I could get ca. 2MiB/s on this very same sample download (dl time ca. 00:04) !

@gamer191, please reopen this issue - and pukkandan must hold off the imminent stable release, for it'll be BROKEN too, if this latest gimmick by Google 😡 isn't mitigated!

Kind regards, thanks in advance!

bashonly commented 1 year ago

I can reproduce the audio format throttling on master as @Vangelis66 describes

Vangelis66 commented 1 year ago

I can reproduce the audio format throttling on master

Most sadly, it isn't limited to just audio (though audio-only was the first I tried...); Google have now "broken" yt-dlp for video-only, too:

yt-dlp -vf 399 "p7FCgw_GlWc" => 

[debug] Command-line config: ['--ffmpeg-location', '..\\FFmpeg', '--downloader-args', 'ffmpeg:-v 8 -stats', '-vf', '399', 'p7FCgw_GlWc']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2023.03.01.19419 [9b7cf00] (win_x86_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6 (setts), ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Extractor Plugins: AGB+NSIG (YoutubeIE)
[debug] Plugin directories: ['<redacted>\\yt-dlp\\yt-dlp-plugins\\YTAgeGateBypass.zip\\yt_dlp_plugins', '<redacted>\\yt-dlp\\yt-dlp-plugins\\YTNSigProxy.zip\\yt_dlp_plugins']
[debug] Loaded 1782 extractors
[youtube+AGB+NSIG] Extracting URL: p7FCgw_GlWc
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading webpage
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading android player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded client config
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading player 7862ca1f
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded player API JSON
[debug] [youtube+AGB+NSIG] Extracting signature function js_7862ca1f_101
[debug] Loading youtube-sigfuncs.js_7862ca1f_101 from cache
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig vjEprBVjAvtOGh8 => -eKiTHxf4oqPE
[debug] [youtube+AGB+NSIG] Extracting signature function js_7862ca1f_105
[debug] Loading youtube-sigfuncs.js_7862ca1f_105 from cache
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig JxGjhc8vVETpT27 => 5KvUjGz9W_L_T
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] p7FCgw_GlWc: Downloading 1 format(s): 399[debug] Invoking dashsegments downloader on "https://rr2---sn-4vguioxu-n3bl.googlevideo.com/videoplayback?expire=1677823917&ei=TTsBZPyjFeKF6dsPvOiPmAo&ip=<redacted>&id=o-ALuYwiFyIpw-zI9gtOOVSDfLzFk4mBJGkHLoiuGvuTPf&itag=399&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-4vguioxu-n3bl%2Csn-nv47zn7y&ms=au%2Crdu&mv=m&mvi=2&pl=18&gcr=gr&initcwndbps=785000&vprv=1&mime=video%2Fmp4&ns=09Ur6WP4o29mc6iXerb6CIAL&gir=yes&clen=140099890&dur=636.902&lmt=1622078607328044&mt=1677801912&fvip=1&keepalive=yes&fexp=24007246&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=5436432&n=5KvUjGz9W_L_T&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgMc14--0-gj34yNGqB6ZJxhIqUMdJpceqkWc4_xQzSAoCIBodkacLAcGaeaAjzIF4oZVj9Ht0x2Cl73rbir9pPW5g&sig=AOq0QJ8wRgIhAJLG_QQ8TjKGXzs3etmnL7L1nDCqm6qN3ft7EeyR-yWwAiEA8mHUC-Fm4h0N3_YZSxSw2mRgkxAvwoTjiz236KGqFHc%3D"
[dashsegments] Total fragments: 14
[download] Destination: Kanye West - Famous [p7FCgw_GlWc].mp4
[debug] File locking is not supported. Proceeding without locking
[download]   0.1% of ~ 140.00MiB at   37.45KiB/s ETA 01:03:45 (frag 0/14)
ERROR: Interrupted by user
Terminate batch job (Y/N)?

... Guess the issue's title has to be changed once again 😠 ...

Vangelis66 commented 1 year ago

Download speeds still throttled for some formats after range fix

Well, I'd make it myself "for ALL formats", since now even the non-segmented ones (18 and/or 22) are being throttled (at least here, YMMV?):

yt-dlp -f 18 "p7FCgw_GlWc" => 

[youtube+AGB+NSIG] Extracting URL: p7FCgw_GlWc
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading webpage
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading android player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded client config
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading player 7862ca1f
[youtube+AGB+NSIG] p7FCgw_GlWc: Downloading web embedded player API JSON
[info] p7FCgw_GlWc: Downloading 1 format(s): 18
[download] Destination: Kanye West - Famous [p7FCgw_GlWc].mp4
[download]   4.4% of   12.33MiB at   83.07KiB/s ETA 02:25
ERROR: Interrupted by user

I really wonder what they came up with this time 😠 ... Perhaps new player 7862ca1f is somehow relevant?

bashonly commented 1 year ago

I still get unthrottled speeds with some videos/formats, e.g. the OP's example URL

$ yt-dlp -f bv pJ0auP7dbcY
[youtube] Extracting URL: pJ0auP7dbcY
[youtube] pJ0auP7dbcY: Downloading webpage
[youtube] pJ0auP7dbcY: Downloading android player API JSON
[info] pJ0auP7dbcY: Downloading 1 format(s): 313
[dashsegments] Total fragments: 642
[download] Destination: 20221116_UCwjLh640nGXSGa9iHRS31ag_pJ0auP7dbcY__.webm
[download]   1.1% of ~   6.27GiB at   13.15MiB/s ETA 07:42 (frag 7/642)^C
ERROR: Interrupted by user

gamer191 commented 1 year ago

OP isn't on master. Theoretically Vangelis66 should have created a new issue, but it's probably easier to just stick to this issue now
Vangelis66's url is an age-restricted music video. I'm not at home right now, but it wouldn't surprise me if only music videos, or only age-restricted videos, are throttled on master
@bashonly are you sure that the format is relevant, as opposed to my assumption that some videos are fully throttled and others aren't throttled at all?

Vangelis66 commented 1 year ago

Non age-gated music clip, non-segmented format:

yt-dlp -f 22 "o8Y_uR6ZDxo" => 
[youtube+AGB+NSIG] Extracting URL: o8Y_uR6ZDxo
[youtube+AGB+NSIG] o8Y_uR6ZDxo: Downloading webpage
[youtube+AGB+NSIG] o8Y_uR6ZDxo: Downloading android player API JSON
[info] o8Y_uR6ZDxo: Downloading 1 format(s): 22
[download] Destination: ROY KNOX x WTCHOUT - Shadows (Feat. Svniivan) [NCS Release] [o8Y_uR6ZDxo].mp4
[download]  43.9% of   31.68MiB at    1.19MiB/s ETA 00:14
ERROR: Interrupted by user

i.e. not throttled!

Same clip, audio-only format:

yt-dlp -f ba "o8Y_uR6ZDxo" => 

[youtube+AGB+NSIG] Extracting URL: o8Y_uR6ZDxo
[youtube+AGB+NSIG] o8Y_uR6ZDxo: Downloading webpage
[youtube+AGB+NSIG] o8Y_uR6ZDxo: Downloading android player API JSON
[info] o8Y_uR6ZDxo: Downloading 1 format(s): 251
[dashsegments] Total fragments: 1
[download] Destination: ROY KNOX x WTCHOUT - Shadows (Feat. Svniivan) [NCS Release] [o8Y_uR6ZDxo].webm
[download]  76.9% of ~   4.03MiB at    1.07MiB/s ETA 00:00 (frag 0/1)
ERROR: Interrupted by user

i.e. not throttled!

So, gamer191 is on to something 👍 ...

gamer191 commented 1 year ago

NoCopyrightSounds don't count as music videos. Try with a Vevo video

bashonly commented 1 year ago

are you sure that the format is relevant, as opposed to my assumption that some videos are fully throttled and others aren't throttled at all?

It does seem to specifically be age-gated videos.

No throttling for this VEVO, non-age-gated music video: https://www.youtube.com/watch?v=bRChz-OYi9o

While all of the actually age-gated music videos in this playlist are throttled: https://www.youtube.com/watch?v=&list=PLl5Tkoh1UlTgdGlIbtcdn8S9vPMEdXaXB

Vangelis66 commented 1 year ago

Try with a Vevo video

yt-dlp -f ba "XXYlFuWEuKI" => 

[youtube+AGB+NSIG] Extracting URL: XXYlFuWEuKI
[youtube+AGB+NSIG] XXYlFuWEuKI: Downloading webpage
[youtube+AGB+NSIG] XXYlFuWEuKI: Downloading android player API JSON
[info] XXYlFuWEuKI: Downloading 1 format(s): 251
[dashsegments] Total fragments: 1
[download] Destination: The Weeknd - Save Your Tears (Official Music Video) [XXYlFuWEuKI].webm
[download]  50.3% of ~   3.97MiB at    1.24MiB/s ETA 00:01 (frag 0/1)
ERROR: Interrupted by user

=> NOT throttled...

So it appears only the age-gated ones are being throttled in master... Yet, it's a recent change by Google...

superislam commented 1 year ago

Try with a Vevo video

yt-dlp -f ba "XXYlFuWEuKI" => 

[youtube+AGB+NSIG] Extracting URL: XXYlFuWEuKI
[youtube+AGB+NSIG] XXYlFuWEuKI: Downloading webpage
[youtube+AGB+NSIG] XXYlFuWEuKI: Downloading android player API JSON
[info] XXYlFuWEuKI: Downloading 1 format(s): 251
[dashsegments] Total fragments: 1
[download] Destination: The Weeknd - Save Your Tears (Official Music Video) [XXYlFuWEuKI].webm
[download]  50.3% of ~   3.97MiB at    1.24MiB/s ETA 00:01 (frag 0/1)
ERROR: Interrupted by user

=> NOT throttled...

So it appears only the age-gated ones are being throttled in master... Yet, it's a recent change by Google...

but this video not age-gated and its throttled [youtube] Extracting URL: https://www.youtube.com/watch?v=pJ0auP7dbcY [youtube] pJ0auP7dbcY: Downloading webpage [youtube] pJ0auP7dbcY: Downloading android player API JSON [info] pJ0auP7dbcY: Downloading 1 format(s): 251 [download] Resuming download at byte 302635 [download] Destination: كيف تنجح العلاقات مع ياسر الحزيمي ｜ بودكاست فنجان [pJ0auP7dbcY].webm [download] 0.4% of 137.24MiB at 25.53KiB/s ETA 01:31:21 ERROR: Interrupted by user 't' is not recognized as an internal or external command, operable program or batch file.

gamer191 commented 1 year ago

@superislam does it work with https://github.com/ytdl-patched/yt-dlp/releases? If not, please post a verbose log If yes, then it's issue #6369, which will be fixed in the next release

Vangelis66 commented 1 year ago

@superislam : As gamer191 pointed out, you're not on master or latest daily build; I am, thus:

yt-dlp -f ba "pJ0auP7dbcY" => 

[youtube+AGB+NSIG] Extracting URL: pJ0auP7dbcY
[youtube+AGB+NSIG] pJ0auP7dbcY: Downloading webpage
[youtube+AGB+NSIG] pJ0auP7dbcY: Downloading android player API JSON
[info] pJ0auP7dbcY: Downloading 1 format(s): 251
[dashsegments] Total fragments: 14
[download] Destination: كيف تنجح العلاقات مع ياسر الحزيمي ｜ بودكاست فنجان [pJ0auP7dbcY].webm
[download]   2.3% of ~ 140.00MiB at    1.14MiB/s ETA 01:59 (frag 0/14)
ERROR: Interrupted by user

superislam commented 1 year ago

@superislam does it work with https://github.com/ytdl-patched/yt-dlp/releases? If not, please post a verbose log If yes, then it's issue #6369, which will be fixed in the next release

yes resolved

gamer191 commented 1 year ago

Are age-gated non-music videos throttled?
I'd be interested what happens if you pass cookies, including for accounts that don't have access to age-gated videos. If you test this, please also include whether or not you have youtube premium

fren-archivist commented 1 year ago

I am also seeing this issue on a number of videos which are not age-restricted for both audio and video formats. I am using the patched version 2023.03.01.19419. It is a problem on most new videos for me, though somewhat inconsistent: sometimes only one format is throttled, sometimes both are, sometimes restarting the download solves the problem but sometimes it does not. Here is one example, I could give many more if it would help:

[debug] Command-line config: ['sJrDp5UeRIM', '-f', '243', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] ytdl-patched/yt-dlp version 2023.03.01.19419 [9b7cf00] (win_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19044-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg git-2020-07-29-cbb6ba2, ffprobe git-2020-07-29-cbb6ba2, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/ytdl-patched/yt-dlp/releases/latest
Latest version: 2023.03.01.19419, Current version: 2023.03.01.19419
yt-dlp is up to date (2023.03.01.19419)
[youtube] Extracting URL: sJrDp5UeRIM
[youtube] sJrDp5UeRIM: Downloading webpage
[youtube] sJrDp5UeRIM: Downloading android player API JSON
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube] Decrypted nsig XDya9AyvDaxN9kq => Lx_OTmr0Ont_h
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] sJrDp5UeRIM: Downloading 1 format(s): 243
[debug] Invoking dashsegments downloader on "https://rr2---sn-p5qlsn76.googlevideo.com/videoplayback?expire=1677833549&ei=7WABZLG3K83s8wSszp3ACQ&ip=xxxxxxxxxxxxxxxx&id=o-ALQR5Tc9LclKNRkCP0U14EMmqSq_bUZPUxJ-yvitqi16&itag=243&aitags=134%2C136%2C160%2C243%2C298%2C299&source=youtube&requiressl=yes&mh=pX&mm=31%2C29&mn=sn-p5qlsn76%2Csn-p5qddn76&ms=au%2Crdu&mv=m&mvi=2&pl=37&initcwndbps=1751250&vprv=1&mime=video%2Fwebm&ns=IDegC3IIl28mjUOvUa9WeV8L&gir=yes&clen=334865871&dur=8983.000&lmt=1677770505967956&mt=1677811254&fvip=4&keepalive=yes&fexp=24007246&c=WEB&txp=7219224&n=Lx_OTmr0Ont_h&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgbRIkjqkBn2S9pNFaCgs2uXm-qpjSITHAdexGc1zxM1gCIBx3TRg6yQt0lgafj52Ez4EbNwC-BREHgp6Do9w8am5i&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgRBn74XcIYWREnve1xcimboP7wYKYIBfEWFkZfUd-GYICICZk-ctk6yWr8swfA34qc3BWMMIvCxDtOpJNwQsy5G8l"
[dashsegments] Total fragments: 32
[download] Destination: 【零 ～月蝕の仮面～】ホラーゲームをそんなに怖がらないあの風見は、バケモンをスナップさせていただきます。【風見くく ⧸ あにまーれ】 [sJrDp5UeRIM].webm
[debug] File locking is not supported. Proceeding without locking
[download]   0.1% of ~ 320.00MiB at   39.87KiB/s ETA 02:16:49 (frag 0/32)
ERROR: Interrupted by user

Interestingly, I have not been able to get the throttling to occur with any videos that are a day old or older on the channels I am scanning, but many of the videos uploaded today are a problem.

ghost commented 1 year ago

Are age-gated non-music videos throttled?

yt-dlp 'https://www.youtube.com/watch?v=7t0SqerlBA0' =>
[youtube] Extracting URL: https://www.youtube.com/watch?v=7t0SqerlBA0
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] 7t0SqerlBA0: Downloading android player API JSON
[youtube] 7t0SqerlBA0: Downloading tv embedded player API JSON
[youtube] 7t0SqerlBA0: Downloading web embedded client config
[youtube] 7t0SqerlBA0: Downloading player 7862ca1f
[youtube] 7t0SqerlBA0: Downloading web embedded player API JSON
[SponsorBlock] Fetching SponsorBlock segments
[SponsorBlock] No matching segments were found in the SponsorBlock database
[info] 7t0SqerlBA0: Downloading 1 format(s): 299+140
[dashsegments] Total fragments: 5
[download] Destination: Test Age Restricted Video [7t0SqerlBA0].f299.mp4
[download]   0.4% of ~  50.00MiB at   78.91KiB/s ETA 10:46 (frag 0/5)^C
ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name=3 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=3>

Vangelis66 commented 1 year ago

Are age-gated non-music videos throttled?

yt-dlp -f bv "7t0SqerlBA0" =>
[youtube+AGB+NSIG] Extracting URL: 7t0SqerlBA0
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading webpage
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading android player API JSON
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading web embedded client config
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading player 7862ca1f
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading web embedded player API JSON
[info] 7t0SqerlBA0: Downloading 1 format(s): 315
[dashsegments] Total fragments: 19
[download] Destination: Test Age Restricted Video [7t0SqerlBA0].webm
[download]   5.3% of ~ 190.00MiB at  840.65KiB/s ETA 03:39 (frag 1/19)
ERROR: Interrupted by user

=> NOT throttled!

~~@0xcc07c9: You're probably NOT in master/latest daily build, so you're simply muddying the waters...~~

ghost commented 1 year ago

@0xcc07c9: You're probably NOT in master/latest daily build, so you're simply muddying the waters...

[debug] Command-line config: ['-v', '7t0SqerlBA0']
[debug] User config "<redacted>/.config/yt-dlp/config": ['--geo-bypass', '--sponsorblock-remove', 'all', '--no-playlist', '-f', 'bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c0154] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 9fddc12
[debug] Python 3.9.2 (CPython x86_64 64bit) - Linux-5.10.0-21-amd64-x86_64-with-glibc2.31 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.31)
[debug] exe versions: ffmpeg 4.3.5-0, ffprobe 4.3.5-0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.9.7, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, secretstorage-3.3.1, sqlite3-2.6.0, websockets-8.1
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[youtube] Extracting URL: 7t0SqerlBA0
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] 7t0SqerlBA0: Downloading android player API JSON
[youtube] 7t0SqerlBA0: Downloading tv embedded player API JSON
[youtube] 7t0SqerlBA0: Downloading web embedded client config
[youtube] 7t0SqerlBA0: Downloading player 7862ca1f
[youtube] 7t0SqerlBA0: Downloading web embedded player API JSON
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube] Decrypted nsig caLVGHxP5Y7eHKD => koXdPWELjuXjg
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube] Decrypted nsig C3A6Nkz6SrkL0z6 => -olVTNtYq7ICi
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[SponsorBlock] Fetching SponsorBlock segments
[debug] SponsorBlock query: https://sponsor.ajay.app/api/skipSegments/ecba?service=YouTube&categories=%5B%22intro%22%2C+%22interaction%22%2C+%22sponsor%22%2C+%22preview%22%2C+%22outro%22%2C+%22music_offtopic%22%2C+%22selfpromo%22%2C+%22filler%22%5D&actionTypes=%5B%22skip%22%2C+%22poi%22%2C+%22chapter%22%5D
[SponsorBlock] No matching segments were found in the SponsorBlock database
[info] 7t0SqerlBA0: Downloading 1 format(s): 299+140
[debug] Invoking dashsegments downloader on "https://rr4---sn-axq7sn7e.googlevideo.com/videoplayback?expire=1677834461&ei=fWQBZJaLCYy67QSEhxY&ip=<redacted>&id=o-AHteAAVvTlJEVrqmyJ3kDUKwHXSLRIpImOdJf7LUQy46&itag=299&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C299%2C302%2C303%2C308%2C315&source=youtube&requiressl=yes&mh=oD&mm=31%2C26&mn=sn-axq7sn7e%2Csn-f5f7kn7z&ms=au%2Conr&mv=m&mvi=4&pl=40&initcwndbps=1625000&vprv=1&mime=video%2Fmp4&ns=P_scZ2dQgG_rZFQqNYSmilUL&gir=yes&clen=43151645&dur=59.999&lmt=1580881166211915&mt=1677812459&fvip=5&keepalive=yes&fexp=24007246&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=1301222&n=-olVTNtYq7ICi&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAI5tKPLhRBNpsXFC21lse5gHnBethyqjoq0qVHohVJj9AiAwupqZ-qsZ9SHtMkoo1PN6DMBUvv_zop2VDcereiWr9w%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgCZcz8RPxAJmLOZvcm07Gxm075nsjGR4IAmDqO-viXk8CIAN9JUH6Tlt-FSMV5aWugwljGZlQYzaRKdKiszoi_8Q6"
[dashsegments] Total fragments: 5
[download] Destination: Test Age Restricted Video [7t0SqerlBA0].f299.mp4
[download]   0.1% of ~  50.00MiB at   81.33KiB/s ETA 10:28 (frag 0/5)^C
ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name=3 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=3>

look at "[debug] Git HEAD: 9fddc12" - git latest commit

fren-archivist commented 1 year ago

@Vangelis66 @0xcc07c9 The throttling may be inconsistent; I have seen the same format get both fast and slow download speeds just by retrying. I just tried 7t0SqerlBA0 several times. With format 315 I was able to get both throttled and unthrottled download speeds by just re-extracting and retrying. With 299 I only saw slow speeds though I only tried 5 times.

Vangelis66 commented 1 year ago

With 299 I only saw slow speeds

Can confirm:

[youtube+AGB+NSIG] Extracting URL: 7t0SqerlBA0
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading webpage
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading android player API JSON
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading web embedded client config
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading player 7862ca1f
[youtube+AGB+NSIG] 7t0SqerlBA0: Downloading web embedded player API JSON
[info] 7t0SqerlBA0: Downloading 1 format(s): 299
[dashsegments] Total fragments: 5
[download] Destination: Test Age Restricted Video [7t0SqerlBA0].mp4
[download]  16.5% of ~  50.00MiB at   70.21KiB/s ETA 10:08 (frag 0/5)
ERROR: Interrupted by user

Well, if the devs can't reproduce the issue themselves in a permanent/reliable fashion, debugging this will be really difficult...

ghost commented 1 year ago

With format 315 I was able to get both throttled and unthrottled download speeds by just re-extracting and retrying.

Tried 315 format 10 times, always throttled

[debug] Command-line config: ['-v', '-f', '315', '7t0SqerlBA0']
[debug] User config "<redacted>/.config/yt-dlp/config": ['--geo-bypass', '--sponsorblock-remove', 'all', '--no-playlist', '-f', 'bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c0154] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 9fddc12
[debug] Python 3.9.2 (CPython x86_64 64bit) - Linux-5.10.0-21-amd64-x86_64-with-glibc2.31 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.31)
[debug] exe versions: ffmpeg 4.3.5-0, ffprobe 4.3.5-0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.9.7, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, secretstorage-3.3.1, sqlite3-2.6.0, websockets-8.1
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[youtube] Extracting URL: 7t0SqerlBA0
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] 7t0SqerlBA0: Downloading android player API JSON
[youtube] 7t0SqerlBA0: Downloading tv embedded player API JSON
[youtube] 7t0SqerlBA0: Downloading web embedded client config
[youtube] 7t0SqerlBA0: Downloading player 7862ca1f
[youtube] 7t0SqerlBA0: Downloading web embedded player API JSON
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube] Decrypted nsig U8GW-1O2nWpNUI9 => nFa4vDzMDQdyx
[debug] Loading youtube-nsig.7862ca1f from cache
[debug] [youtube] Decrypted nsig CqIBbfdDzX613Uv => mHd1vuBrQ2p7Y
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[SponsorBlock] Fetching SponsorBlock segments
[debug] SponsorBlock query: https://sponsor.ajay.app/api/skipSegments/ecba?service=YouTube&categories=%5B%22filler%22%2C+%22interaction%22%2C+%22music_offtopic%22%2C+%22outro%22%2C+%22selfpromo%22%2C+%22sponsor%22%2C+%22preview%22%2C+%22intro%22%5D&actionTypes=%5B%22skip%22%2C+%22poi%22%2C+%22chapter%22%5D
[SponsorBlock] No matching segments were found in the SponsorBlock database
[info] 7t0SqerlBA0: Downloading 1 format(s): 315
[debug] Invoking dashsegments downloader on "https://rr4---sn-axq7sn7e.googlevideo.com/videoplayback?expire=1677835617&ei=AWkBZJvoAdfw7gPr84bYBg&ip=<redacted>&id=o-AGjagaSjNgaLPtn7Ccwbgry2objzWGZYrTKTqOgIwDtU&itag=315&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C299%2C302%2C303%2C308%2C315&source=youtube&requiressl=yes&mh=oD&mm=31%2C26&mn=sn-axq7sn7e%2Csn-f5f7lnld&ms=au%2Conr&mv=m&mvi=4&pl=40&initcwndbps=1603750&vprv=1&mime=video%2Fwebm&ns=bnG-8yc763ScRO85jocP6ZML&gir=yes&clen=198925456&dur=59.999&lmt=1629913446917715&mt=1677813661&fvip=1&keepalive=yes&fexp=24007246&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=1311222&n=mHd1vuBrQ2p7Y&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgXvPoy6hzkPqHkiIHzB3z8FSpaQYGhV8kLz-gB8gI8UkCIQC9UQ94bRQbjnZCsd_66ELQMvbVmIExFa3Gd2HzqvU2vQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAMiFqXna4SoYdl4sWQUsY3wTfK1_GhNlZyhx8khxPZbgAiBNIYPa51TaIwn-5Ppp1d6O3-MXGWRlBA91WAW5X0atIw%3D%3D"
[dashsegments] Total fragments: 19
[download] Destination: Test Age Restricted Video [7t0SqerlBA0].webm
[download]   0.0% of ~ 190.00MiB at   88.17KiB/s ETA 36:45 (frag 0/19)^C
ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name=3 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=3>

bashonly commented 1 year ago

I am also seeing this issue on a number of videos which are not age-restricted for both audio and video formats. I am using the patched version 2023.03.01.19419. It is a problem on most new videos for me, though somewhat inconsistent: sometimes only one format is throttled, sometimes both are, sometimes restarting the download solves the problem but sometimes it does not. [debug] Command-line config: ['sJrDp5UeRIM', '-f', '243', '-vU'] Interestingly, I have not been able to get the throttling to occur with any videos that are a day old or older on the channels I am scanning, but many of the videos uploaded today are a problem.

The pattern that's emerging is that throttling vs no throttling depends on the player client. Format 243 of your example video is only available via the web client. yt-dlp uses the tv_embedded client to circumvent the age-gate. Both of these clients seem to be throttled, with tv_embedded seemingly always throttled.

A workaround for now is to use --extractor-args "youtube:player_client=android", since android client downloads seem to be working fine for the time-being. Note that some formats may be missing when using this workaround, and yt-dlp's built-in age-gate bypass will override this arg for age-gated videos.

coletdjnz commented 1 year ago

It appears to be all JS-based clients that are being throttled. This looks very similar to the old n-param throttling issue we had.

android, ios clients are fine

edit: appears to be the n-param decoding wrong

rockxwre commented 1 year ago

Using the latest version (2023.03.03.19419) it does not support protocol https anymore for this video: https://www.youtube.com/watch?v=2fAalDg6sKE

I used the command

yt-dlp -vU -f "ba*[ext=m4a][protocol=https]" https://www.youtube.com/watch?v=2fAalDg6sKE but is results in the error

Latest version: 2023.03.03.19419, Current version: 2023.03.03.19419
yt-dlp is up to date (2023.03.03.19419)
[youtube] Extracting URL: https://www.youtube.com/watch?v=2fAalDg6sKE
[youtube] 2fAalDg6sKE: Downloading webpage
[youtube] 2fAalDg6sKE: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
ERROR: [youtube] 2fAalDg6sKE: Requested format is not available. Use --list-formats for a list of available formats

Omitting the protocol in the command does solve the slow data transfer issue though. Using the latest stable version (2023.02.17) does support the https protocol, but the download is very slow.

pukkandan commented 1 year ago

Using the latest version (2023.03.03.19419) it does not support protocol https anymore for this video: youtube.com/watch?v=2fAalDg6sKE

That is expected. If we don't convert the formats to DASH, youtube will throttle them

Vangelis66 commented 1 year ago

Immense thanks for the fix! 🥇 ... And I don't want to "blow my own horn" 😜 , but, consistent with:

edit: appears to be the n-param decoding wrong

I did hint in a previous comment (that I've since hidden) that

Perhaps new player 7862ca1f is somehow relevant?

Regards and thanks 😄 !

pukkandan commented 1 year ago

Tip for debugging in future:

When YT throttles, the first thing to check for is --extractor-args youtube:player-client=android. If it works, the problem is most likely that nsig broke. This can be verified by extracting relevant code manually[^2] from the player JS, running it in browser console and comparing with the value yt-dlp gives[^1].

Once we determine that the issue is with n-sig, the specific problem can be identified by stepping through the JS code with node/jsinterp in parallel and checking where they differed. Then all we are left with is actually fixing it[^3].

Debugging https://github.com/yt-dlp/yt-dlp/issues/6369 was more involved since it affected android formats as well and was unrelated to n-sig. None of the developers being unable to reproduce for the first day definitely didn't help either.

[^1]: player id, encrypted sig and result are all visible in verbose log [^2]: See https://github.com/yt-dlp/yt-dlp/blob/9acf1ee25f7ad3920ede574a9de95b8c18626af4/yt_dlp/extractor/youtube.py#L3074-L3078 [^3]: Which turned out to be easy this time, but not in #1437, #4635

ultrasound1372 commented 1 year ago

Slightly OT but a good comment for people wanting to test. Instead of telling pip to install directly from the git master, clone the git master, change into it and do

pip install -e .

Then when new updates come in, simply execute git pull within this directory, and wham latest patches are there! I personally use the awesome pipx for this, where the -e option still works.

skdursh commented 1 year ago

Haven't tested every option but chiming in to say that with the most recent release (https://github.com/yt-dlp/yt-dlp/commit/354d5fca7a533f60a6eee14e854b9b4c132a7314) I'm also getting throttling with -f 140 (avg of about ~350kb/s).

Without specifying a format it still seems to be throttled, but less severely (about 2mb/s)

bashonly commented 1 year ago

@skdursh show verbose log -vU

skdursh commented 1 year ago

@bashonly how does -vU work? Should it print a logfile to the destination folder or is the relevant info in the console? I see a debug comment in the console with a gibberish url, but nothing that looks relevant to the issue. If it makes any difference, the audio I was originally trying to download was for a "Live" video, from a stream that ended a few hours ago.

bashonly commented 1 year ago

@skdursh the debug information is used for debugging. If you want to get to the bottom of this issue, please post all of it

CoffeSiberian commented 1 year ago

hey, in my case I use the URL generated by the request directly to use my own download methods to which I assigned a header ('Range': 'bytes = 0-') to have more download speed but this way no longer It works since the last updates. Any suggestions?

bashonly commented 1 year ago

@CoffeSiberian If you absolutely must download with your own methods, you could try adding -f b to your command, however that will limit you to 720p. Otherwise the fastest download will be with yt-dlp. Youtube is intentionally throttling requests with range headers for most formats

chrcoluk commented 1 year ago

I had another 6 videos in my list to download, so I downloaded them using the fixed binary from the other issue.

5 of them were slow on the video, but fast on the audio download. (bits/second measured not time) 1 of them was fast on both.

However they were all old videos likely not in fast CDN's.

pukkandan commented 1 year ago

If more people keep reporting "issues" without showing logs, I'll just lock this thread too. Is it that difficult to actually provide some useful info when reporting something?

gatecrasher777 commented 1 year ago

Working on another YouTube project, today I managed to resolve all my recently experienced speed issues by adding &range=start-end to all (deciphered & n- transformed) stream urls.

Previously, you didn't need to have a range parameter in the urls, now you do otherwise the streams can be severely throttled. Having range headers in your download requests is not a substitute.

If you are using stream urls in html video or audio tags, as I do, just add &range=0-clen to the urls with clen being the content length parameter of the same stream urls.

Then everything is fast again.

NewUserHa commented 1 year ago

Tested with the url from https://github.com/yt-dlp/yt-dlp/issues/6400#issuecomment-1453335114: yt-dlp version 2023.02.17 [a0a7c01] w/ or w/o 'youtube:player-client=android' both has the same about 157KiB/s. yt-dlp version 2023.03.03 [934496428] w/ or w/o 'youtube:player-client=android' both has the same up to about 3MiB/s ( but from 10kb/s speed up to in per frag.). all tests with -vvvvvv, but none of them print any n-sig, so didn't include any in this result as well.

BTW, what is the difference between https and dash protocols for the same video/audio ID? Is that dash uses hls m3u8 of multi-parted webm/mp4 files, but https uses from a single webm/mp4 file? Because usually, encoded to a single video file has fewer keyframes (i.e. better quality) than encoded to multi-parted. (except cutting on stream and remuxing).

Edited: After checked, verified that the https uses class HttpFD(FileDownloader) and http_dash_segments uses class FragmentFD(FileDownloader), then both use urllib to download rather than using ffmpeg, and FragmentFD has decrypter() method and seems download frags are directly append to the file, than HttpFD.

pukkandan commented 1 year ago

BTW, what is the difference between https and dash protocols for the same video/audio ID?

See https://github.com/yt-dlp/yt-dlp/issues/6400#issuecomment-1454903800. Changing only Range headers to a single URL is the normal method of downloading files and is what ytdl/p calls "https" protocol. Youtube requires that we change the query parameter instead, effectively having to request multiple URLs per video. We call the result of each of the URL a "fragment" and the proces the "dash" protocol. This is similar to, but not the same as m3u8/hls. If you want to know detailed differences, I suggest you google it - these are both standard protocols used in video streaming.

NewUserHa commented 1 year ago

Right, it's a name that can easily get ppl confused with 'Dynamic Adaptive Streaming over HTTP (DASH)', but I already found those in the codes and updated the post, see the "Edited:".

BTW, if as the above one post said that &range=0-clen could work, then it can be better than http_frag because the single file only needs to speed up once, so higher downloading speed.

pukkandan commented 1 year ago

it's a name that can easily get ppl confused with 'Dynamic Adaptive Streaming over HTTP (DASH)'

It is the same. Just that we are constructing the playlist instead of youtube providing it directly

pukkandan commented 1 year ago

as the above one post said that &range=0-clen could work

We have tested that to not work for longer videos

gatecrasher777 commented 1 year ago

as the above one post said that &range=0-clen could work

We have tested that to not work for longer videos

Appending &range=0-clen to the stream url works in an html video source tag, regardless of video length. Without it, buffering on playback is horrible in all browsers. Only very low quality streams will play without buffering.

In a downloader, for a longer video, you would normally download it in chunks, so you would append &range=start-end to the url for each chunk. You would match these with the Range request header.

The point is that the YouTube player itself (base.js or embed.js) appends a &range parameter to the stream url of each video playback request, and the googlevideo.com servers (since recently) will delay and throttle requests if it is not included.

It might "not work" in every instance, but it is not going to break anything if it doesn't.

dozyx commented 1 year ago

as the above one post said that &range=0-clen could work

We have tested that to not work for longer videos

It seems work for longer videos now.

ForHelvete commented 1 year ago

Is there any quality loss from downloading the DASH-versions? It's always the same file size.

bashonly commented 1 year ago

@ForHelvete No, you end up with the same file, it's just been downloaded differently.

yt-dlp / yt-dlp