yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
72.18k stars 5.75k forks source link

YouTube: Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "BaW_jenozKc") #9554

Closed rredford6 closed 1 day ago

rredford6 commented 1 month ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Region

United States

Provide a description that is worded well enough to be understood

This is the crux of the issue:

WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "BaW_jenozKc")

This was previously solved in #4593. It was first rediscovered on https://github.com/yt-dlp/yt-dlp/pull/9546#issuecomment-2025347080

This can be reproduced on multiple YouTube videos. This warning is not seen on every download.

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch?v=BaW_jenozKc']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version master@2024.03.22.222728 from yt-dlp/yt-dlp-master-builds [e5d4f1110] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-163-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1i  8 Dec 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, requests-2.27.1, sqlite3-3.31.1, urllib3-1.26.9, websockets-10.4
[debug] Proxy map: {}
[debug] Request Handlers: urllib
[debug] Loaded 1807 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-master-builds/releases/latest
Latest version: master@2024.03.22.222728 from yt-dlp/yt-dlp-master-builds
yt-dlp is up to date (master@2024.03.22.222728 from yt-dlp/yt-dlp-master-builds)
[youtube] Extracting URL: https://www.youtube.com/watch?v=BaW_jenozKc
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading ios player API JSON
[youtube] BaW_jenozKc: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "BaW_jenozKc")
[debug] Loading youtube-nsig.3b96d06c from cache
[debug] [youtube] Decrypted nsig -Dr9JVw16PnpN3kB => IO0gtddH8cbvwQ
[debug] Loading youtube-nsig.3b96d06c from cache
[debug] [youtube] Decrypted nsig LbPw5yQFOwpSkCGf => Noiw4nN5a0OI9g
[youtube] BaW_jenozKc: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] BaW_jenozKc: Downloading 1 format(s): 248+251
[debug] Invoking http downloader on "https://"
[download] Destination: youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].f248.webm
[download] 100% of  965.31KiB in 00:00:00 at 3.34MiB/s
[debug] Invoking http downloader on "https://"
[download] Destination: youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].f251.webm
[download] 100% of  138.96KiB in 00:00:00 at 861.80KiB/s
[Merger] Merging formats into "youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:youtube-dl test video "'"'"'⧸⧹ä↭𝕐 [BaW_jenozKc].f248.webm' -i 'file:youtube-dl test video "'"'"'⧸⧹ä↭𝕐 [BaW_jenozKc].f251.webm' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:youtube-dl test video "'"'"'⧸⧹ä↭𝕐 [BaW_jenozKc].temp.webm'
Deleting original file youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].f251.webm (pass -k to keep)
Deleting original file youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].f248.webm (pass -k to keep)
bashonly commented 1 month ago

will be "fixed" by #9553 ( :cry: ). yt doesn't want us to have nice things

absidue commented 1 month ago

will be "fixed" by #9553 ( :cry: ). yt doesn't want us to have nice things

I'm pretty sure YouTube adding that has nothing to do with yt-dlp and is entirely because of ReVanced and yt-dlp is just collateral damage. As ReVanced modifies the official YouTube app (unlike NewPipe which is a separate app written from scratch), it's no longer enough for YouTube to just check if you are using the Android app, they now need to check if you are using a modded version of it.

pukkandan commented 1 month ago

To maintainers: Don't close the issue till release. From experience, keeping issue open seems to reduce number of duplicates

kiwib1rd1337 commented 1 month ago

I can confirm this happening in New Zealand also (no VPN, native IPv4/IPv6):

[youtube] Extracting URL: https://youtu.be/luXZz9AqBbQ [youtube] luXZz9AqBbQ: Downloading webpage [youtube] luXZz9AqBbQ: Downloading ios player API JSON [youtube] luXZz9AqBbQ: Downloading android player API JSON WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "luXZz9AqBbQ") [youtube] luXZz9AqBbQ: Downloading m3u8 information [info] luXZz9AqBbQ: Downloading 1 format(s): 18 [download] Destination: r⧸AskOuija where they talk one LETTER at a time [luXZz9AqBbQ].mp4 [download] 100% of 10.50MiB in 00:00:02 at 4.20MiB/s [Metadata] Adding metadata to "r⧸AskOuija where they talk one LETTER at a time [luXZz9AqBbQ].mp4" ` Downloading the "aQvGIIdgFDM" video gives a file titled "Video Not Available [aQvGIIdgFDM].mp4", length of 5 minutes, black background, circled exclamation point on left, "The following content is not available in this app. Watch this content on the latest version of YouTube" on the right.

Whatever kind of black magic is YT trying to pull on us now, and why it is happening to Android client APIs specifically is currently a mystery to me. Why is this not happening on the iOS one as well?

UPDATE: Tested on the 'https://youtu.be/BaW_jenozKc' video as well, failed to replicate on that video:

[youtube] Extracting URL: https://youtu.be/BaW_jenozKc [youtube] BaW_jenozKc: Downloading webpage [youtube] BaW_jenozKc: Downloading ios player API JSON [youtube] BaW_jenozKc: Downloading android player API JSON [youtube] BaW_jenozKc: Downloading m3u8 information [info] BaW_jenozKc: Downloading 1 format(s): 18 [download] Destination: youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].mp4 [download] 100% of 184.94KiB in 00:00:02 at 73.93KiB/s [Metadata] Adding metadata to "youtube-dl test video "'⧸⧹ä↭𝕐 [BaW_jenozKc].mp4"

Possible A/B testing perhaps? Knowing YT's shady testing behaviour, it's certainly could be...

xpalidotious commented 1 month ago

will be "fixed" by #9553

For an end-user is there any significant difference between that fix and simply using "--extractor-args youtube:player_client=ios,web" in your config?

The docs mention that

By default, ios,android,web is used, but tv_embedded and creator variants are added as required for age-gated videos.

and I'm not sure if explicitly specifying player clients overrides that behavior (and are those variants ever needed anyway if you're using cookies for a logged in youtube account?)

From experience, keeping issue open seems to reduce number of duplicates

That's probably because people like me tend to come here only when they have an issue themselves and don't realize that closed issues are hidden by default, and so think no one has posted about their issue yet.

pukkandan commented 1 month ago

From experience, keeping issue open seems to reduce number of duplicates

That's probably because people like me tend to come here only when they have an issue themselves and don't realize that closed issues are hidden by default, and so think no one has posted about their issue yet.

I know. We specifically ask to search closed issues, but many don't. It's what it is... Not trying to blame anyone. I was just passing the wisdom onto other maintainers since they are the ones handling most issues nowadays

  • [x] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates

For an end-user is there any significant difference between that fix and simply using "--extractor-args youtube:player_client=ios,web" in your config?

By default, ios,android,web is used, but tv_embedded and creator variants are added as required for age-gated videos.

and I'm not sure if explicitly specifying player clients overrides that behavior

No. The age-gate clients are still added when required. You could also simply ignore the warning.

(and are those variants ever needed anyway if you're using cookies for a logged in youtube account?)

It is not. If your account can directly access the video, we don't need any bypass


Whatever kind of black magic is YT trying to pull on us now, and why it is happening to Android client APIs specifically is currently a mystery to me.

Youtube is using Safetynet's AndroidGuard to validate that the requests are coming from their app; possibly in an effort to block revanced, though they have already implemented a bypass.


Possible A/B testing perhaps?

Yes, it happens randomly, though expect it to be completely blocked in few days

pukkandan commented 1 month ago

Fixed, but leaving open till release

SantiiRepair commented 1 month ago

This is happening in Venezuela also

gamer191 commented 1 month ago

@SantiiRepair try running yt-dlp --update-to master, and if it still keeps happening, add --verbose to your command and send the output

Javierchik commented 1 month ago

@SantiiRepair try running yt-dlp --update-to master, and if it still keeps happening, add --verbose to your command and send the output

Thanks for response, but I got this:

Current version: stable@2024.03.10 from yt-dlp/yt-dlp
Latest version: master@2024.04.08.232116 from yt-dlp/yt-dlp-master-builds

Looks like GitHub master branch needs to be updated? Thx

pukkandan commented 1 month ago

That is not a complete verbose log

pukkandan commented 1 month ago

Closing since release is out

gamer191 commented 3 weeks ago

The sequel: https://github.com/yt-dlp/yt-dlp/issues/9725

bashonly commented 3 weeks ago

Reopening this, since it's happening again. Note that as of today, it happens very infrequently. Presumably, this is due to some sort of A/B testing and the problem will become more prevalent eventually. It's likely that the new android player params' end-of-life date is near. This issue will be "fixed" by #9553

kolbdog323 commented 3 weeks ago

I have the same bug since it looks like the bug reappeared yesterday it keeps pausing on almost every download now it was fine 3 days ago. and it keeps saying [debug] Command-line config: ['--remux-video', 'mp4', '-vU', 'https://www.youtube.com/watch?v=yUDc78znrLM'] [debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version nightly@2024.04.21.232710 from yt-dlp/yt-dlp-nightly-builds [ff38a011d] (pip) [debug] Python 3.12.2 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 3.0.13 30 Jan 2024) [debug] exe versions: ffmpeg 7.0-full_build-www.gyan.dev (setts), ffprobe 7.0-full_build-www.gyan.dev [debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.43.1, urllib3-2.2.1, websockets-12.0 [debug] Proxy map: {} [debug] Request Handlers: urllib, requests, websockets [debug] Loaded 1810 extractors [debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest Latest version: nightly@2024.04.21.232710 from yt-dlp/yt-dlp-nightly-builds yt-dlp is up to date (nightly@2024.04.21.232710 from yt-dlp/yt-dlp-nightly-builds) [youtube] Extracting URL: https://www.youtube.com/watch?v=yUDc78znrLM [youtube] yUDc78znrLM: Downloading webpage [youtube] yUDc78znrLM: Downloading ios player API JSON [youtube] yUDc78znrLM: Downloading android player API JSON WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "yUDc78znrLM") [debug] Loading youtube-nsig.9135c2ab from cache [debug] [youtube] Decrypted nsig LAauB_HxR-jWq4Kf => E42fDI0hSq25Rg [debug] Loading youtube-nsig.9135c2ab from cache [debug] [youtube] Decrypted nsig TMahUCnN2xYyyrhq => gfGiU_As3Byuhg [youtube] yUDc78znrLM: Downloading m3u8 information [debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto [debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id [debug] Default format spec: bestvideo*+bestaudio/best [info] yUDc78znrLM: Downloading 1 format(s): 243+251 [debug] Invoking http downloader on "https://rr5---sn-8xgp1vo-p5qe.googlevideo.com/videoplayback?expire=1714043187&ei=0-QpZqlUu4SS5w-pqIzYBg&ip=71.171.34.118&id=o-ACBN7U_gWbgQwgm4V7JnhfuFglkYpmCa40ri08QltZ8P&itag=243&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=Tl&mm=31%2C26&mn=sn-8xgp1vo-p5qe%2Csn-vgqsrn6l&ms=au%2Conr&mv=m&mvi=5&pl=18&initcwndbps=1317500&vprv=1&svpuc=1&mime=video%2Fwebm&gir=yes&clen=17118776&dur=528.026&lmt=1527102558874107&mt=1714020757&fvip=4&keepalive=yes&c=IOS&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRAIgSb0QmkfNQpG50j5e9J0cAVNeddAKKriEloTWk9esutACIApTXKd_Tb3jxufHcYQ7pI6SqZZFZFamr9SkFpjI8RMf&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHWaYeowRQIgDasqd-6DN63YPcH770QJ_VLL0KbgLpuBAWUGdhhNAIgCIQCQ2N0zRjtCdSmKvuXPPuLqkNrTxx47YPVv5pvb916Jpw%3D%3D" [debug] File locking is not supported. Proceeding without locking [download] Destination: Thru The Mirror (1936) Mickey Mouse complete [yUDc78znrLM].f243.webm [download] 3.1% of 16.33MiB at 343.44KiB/s ETA 00:47

IT always says this for every download now: Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "yUDc78znrLM") and it always pasuses then retries

kclauhk commented 3 weeks ago

removing the android params seems work

bashonly commented 3 weeks ago

@kclauhk the stream URL expires after 1 minute resulting in a 403 error during download if the params are removed

kclauhk commented 3 weeks ago

ok

how is this "{1: {1: 1}}" converted to "CgIIAQ=="?

bashonly commented 3 weeks ago

how is this "{1: {1: 1}}" converted to "CgIIAQ=="?

It's protobuf. If you want to experiment with the params you could use https://github.com/Grub4K/qpb

randombyte-developer commented 3 weeks ago

For a temporary fix you can install the fix from this PR https://github.com/yt-dlp/yt-dlp/pull/9553:

python3 -m pip install -U pip hatchling wheel
python3 -m pip install --force-reinstall "yt-dlp[default] @ https://github.com/coletdjnz/yt-dlp-dev/archive/fix/youtube/remove-android-as-default.tar.gz"
Hrxn commented 2 weeks ago

FWIW, I'm not getting these intermittently, I've seen this warning now on all of the last ~ 30 videos or so from YT.

PS C:\> yt-dlp -U
Latest version: nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds)
PS C:\>

Region is EU (DE).

Oh, and there is also a second warning (at least for me) now, and I couldn't find anything about this with a search in Issues here.

PS C:\> yt-dlp --youtube 'https://www.youtube.com/watch?v=zKRYqI84ov0'
[youtube] Extracting URL: https://www.youtube.com/watch?v=zKRYqI84ov0
[youtube] zKRYqI84ov0: Downloading webpage
[youtube] zKRYqI84ov0: Downloading ios player API JSON
[youtube] zKRYqI84ov0: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "zKRYqI84ov0")
[youtube] zKRYqI84ov0: Downloading m3u8 information
WARNING: [youtube] Incomplete data received in embedded initial data; re-fetching using API.
[youtube] zKRYqI84ov0: Downloading initial data API JSON
[MetadataParser] Did not find '[.]' in uploader
[MetadataParser] Did not find '[.]' in title
[MetadataParser] Did not find '[.]' in extractor
[MetadataParser] Did not find '[.]' in id
[MetadataParser] Changed extractor to: YouTube
[MetadataParser] Did not find '[.]' in channel
[info] zKRYqI84ov0: Downloading 1 format(s): 313+251
[download] Destination:   [..]

I've just picked this video link from my front page, seems like it is the latest video upload from the WhistlinDiesel channel. Download (and video) still seem to work, btw.

meoyawn commented 2 weeks ago

also getting "android" warnings while running from Germany, Austria, Thailand

darklongbow commented 2 weeks ago

same warning from Hongkong

sseodate commented 1 week ago

any update guys ?

pukkandan commented 1 week ago

You can simply ignore the warning. We don't have any way to get the android formats now. But for most users, this will have no effect. The warning will be removed in the next stable release.

Pls avoid commenting unless you have something useful to add.

Hrxn commented 1 week ago

@pukkandan What about the second warning (from my log 5 comments above)?

WARNING: [youtube] Incomplete data received in embedded initial data; re-fetching using API.

Not sure if it's related at all. Maybe it belongs into its own issue, not sure.

pukkandan commented 1 week ago

@Hrxn Was that a one-off, or consistent issue? If is is happening consistently, pls open a new issue. Otherwise, don't worry about it. Youtube API sometimes gives incomplete results and so we have fallbacks for it. The warning is just informing that a fallback was triggered

Hrxn commented 1 week ago

Definitely not a one-off thing for me, happened for the last dozen or so videos I tried..

But it seems that I cannot reproduce with --ignore-config. I only get the android client response warning from this issue here then, as expected. Only with my normal cookies I get these two warnings again.

PS C:\Apps\Temp\2024> yt-dlp --freshcookies
Extracting cookies from chrome
Extracted 3264 cookies from chrome
Latest version: nightly@2024.05.05.232701 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.05.232701 from yt-dlp/yt-dlp-nightly-builds)
PS C:\Apps\Temp\2024> yt-dlp --ignore-config 'https://www.youtube.com/watch?v=NAwH6d1Gp9g'
[youtube] Extracting URL: https://www.youtube.com/watch?v=NAwH6d1Gp9g
[youtube] NAwH6d1Gp9g: Downloading webpage
[youtube] NAwH6d1Gp9g: Downloading ios player API JSON
[youtube] NAwH6d1Gp9g: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "NAwH6d1Gp9g")
[youtube] NAwH6d1Gp9g: Downloading m3u8 information
[info] NAwH6d1Gp9g: Downloading 1 format(s): 313+251
[download] Destination: Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f313.webm
[download] 100% of  989.63MiB in 00:00:41 at 23.74MiB/s
[download] Destination: Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f251.webm
[download] 100% of   10.54MiB in 00:00:00 at 23.26MiB/s
[Merger] Merging formats into "Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].webm"
Deleting original file Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f251.webm (pass -k to keep)
Deleting original file Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f313.webm (pass -k to keep)
PS C:\Apps\Temp\2024> ren '.\Audi Group B Rally Car vs THE STIG | TG Tunnel Run `[NAwH6d1Gp9g`].webm' '.\Audi Group B Rally Car vs THE STIG | TG Tunnel Run `[NAwH6d1Gp9g`]_NO_CONFIG_.webm'
PS C:\Apps\Temp\2024> yt-dlp --ignore-config --cookies .\yt-dlp.cookies.txt 'https://www.youtube.com/watch?v=NAwH6d1Gp9g'
[youtube] Extracting URL: https://www.youtube.com/watch?v=NAwH6d1Gp9g
[youtube] NAwH6d1Gp9g: Downloading webpage
[youtube] NAwH6d1Gp9g: Downloading ios player API JSON
[youtube] NAwH6d1Gp9g: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "NAwH6d1Gp9g")
[youtube] NAwH6d1Gp9g: Downloading m3u8 information
WARNING: [youtube] Incomplete data received in embedded initial data; re-fetching using API.
[youtube] NAwH6d1Gp9g: Downloading initial data API JSON
[info] NAwH6d1Gp9g: Downloading 1 format(s): 313+251
[download] Destination: Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f313.webm
[download] 100% of  989.63MiB in 00:00:49 at 19.93MiB/s
[download] Destination: Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f251.webm
[download] 100% of   10.54MiB in 00:00:00 at 18.72MiB/s
[Merger] Merging formats into "Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].webm"
Deleting original file Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f251.webm (pass -k to keep)
Deleting original file Audi Group B Rally Car vs THE STIG | TG Tunnel Run [NAwH6d1Gp9g].f313.webm (pass -k to keep)
PS C:\Apps\Temp\2024>

Does this even make any sense??

rredford6 commented 1 week ago

Creating a new issue for an unrelated problem makes sense

liaolingfen4 commented 1 week ago

the issue has any update?

liaolingfen4 commented 1 week ago

any update for this issue?

xopclabs commented 2 days ago

This is happening to me using python api while not happening calling yt-dlp through os.system or subprocess (or in plain bash) using the same arguments. Weird!

sseodate commented 2 days ago

This is happening to me using python api while not happening calling yt-dlp through os.system or subprocess (or in plain bash) using the same arguments. Weird!

does user agent affect this ?

bashonly commented 2 days ago

does user agent affect this ?

No. And claims like the one above have no meaning if verbose logs are not shared