Open saidjonUzz opened 1 month ago
that is not a verbose log
that is not a verbose log
this full log
Run your yt-dlp command with
-vU
flag added (yt-dlp -vU <your command line>
) Copy the WHOLE output (starting with[debug] Command-line config
)
yt-dlp -vU
yt-dlp
-vU https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U
[debug] Command-line config: ['-vU', 'https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.05.27 from yt-dlp/yt-dlp [12b248ce6] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.15.0-107-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1f 31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2019.11.28, mutagen-1.47.0, requests-2.31.0, sqlite3-3.31.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.05.27 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.05.27 from yt-dlp/yt-dlp)
[youtube] Extracting URL: https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U
[youtube] 3z0IT3bXm_E: Downloading webpage
[youtube] 3z0IT3bXm_E: Downloading ios player API JSON
ERROR: [youtube] 3z0IT3bXm_E: The following content is not available on this app.. Watch on the latest version of YouTube.
File "/home/user/.local/bin/yt-dlp/yt_dlp/extractor/common.py", line 734, in extract
ie_result = self._real_extract(url)
File "/home/user/.local/bin/yt-dlp/yt_dlp/extractor/youtube.py", line 4248, in _real_extract
self.raise_no_formats(reason, expected=True)
File "/home/user/.local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1257, in raise_no_formats
raise ExtractorError(msg, expected=expected, video_id=video_id)
Still not working, could it be a problem with the server?
same here
´´´ system@user:~/frey$ ./yt-dlp -U Latest version: stable@2024.05.27 from yt-dlp/yt-dlp yt-dlp is up to date (stable@2024.05.27 from yt-dlp/yt-dlp) system@user:~/frey$ ./yt-dlp Q1kmXoeeKAE [youtube] Extracting URL: Q1kmXoeeKAE [youtube] Q1kmXoeeKAE: Downloading webpage [youtube] Q1kmXoeeKAE: Downloading ios player API JSON ERROR: [youtube] Q1kmXoeeKAE: The following content is not available on this app.. Watch on the latest version of YouTube. system@user:~/frey$ ./yt-dlp -vU https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U [debug] Command-line config: ['-vU', 'https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U'] [debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version stable@2024.05.27 from yt-dlp/yt-dlp [12b248ce6] (zip) [debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-182-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1f 31 Mar 2020, glibc 2.31) [debug] exe versions: ffmpeg N-92517-gd0f594202a-cesarkrespo (fdk) [debug] Optional libraries: certifi-2019.11.28, requests-2.22.0, secretstorage-2.3.1, sqlite3-3.31.1, urllib3-1.25.8 [debug] Proxy map: {} [debug] Request Handlers: urllib [debug] Loaded 1820 extractors [debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest Latest version: stable@2024.05.27 from yt-dlp/yt-dlp yt-dlp is up to date (stable@2024.05.27 from yt-dlp/yt-dlp) [youtube] Extracting URL: https://youtu.be/3z0IT3bXm_E?si=sJlxgc-qRI8mxT3U [youtube] 3z0IT3bXm_E: Downloading webpage [youtube] 3z0IT3bXm_E: Downloading ios player API JSON ERROR: [youtube] 3z0IT3bXm_E: The following content is not available on this app.. Watch on the latest version of YouTube. File "./yt-dlp/yt_dlp/extractor/common.py", line 734, in extract ie_result = self._real_extract(url) File "./yt-dlp/yt_dlp/extractor/youtube.py", line 4248, in _real_extract self.raise_no_formats(reason, expected=True) File "./yt-dlp/yt_dlp/extractor/common.py", line 1257, in raise_no_formats raise ExtractorError(msg, expected=expected, video_id=video_id) ´´´
Same problem here:
DownloadError: ERROR: [youtube] ... : The following content is not available on this app.. Watch on the latest version of YouTube.
Works correctly for me on latest nightly version
Works correctly for me on latest nightly version
I don't think you're affected by this problem because there's been no change between the release and the nightly. I have the same error on the last release and the nightly as well.
same problem here
Works correctly for me on latest nightly version
I don't think you're affected by this problem because there's been no change between the release and the nightly. I have the same error on the last release and the nightly as well.
So might be just YouTube doing some A/B testing, with B version not compatible with yt-dlp...
It works for me on one computer, but not on another. I thought it might be the IP-addresses. Made many requests from the computer where it suddenly doesn't work anymore. Could it be I've been blocked?
Yes, this is how youtube's IP banning is manifesting itself via yt-dlp output now
Yes, this is how youtube's IP banning is manifesting itself via yt-dlp output now
YouTube is harming legitimate users and data preservation efforts by doing this yet failing to address scammers. They must REALLY hate their user base if they are going this far. People are just going to work around these blocks; it's a waste of time. How they are still doing this baffles me. Probably because it harms their bottom line.
Is this related? https://support.google.com/youtube/thread/269521462?hl=en
Is this related? https://support.google.com/youtube/thread/269521462?hl=en
I took one good look at it and it's a blatant "screw you" to their users. YouTube just decided to lock themselves into a battle that they cannot win to "protect the creators" even though they do next to nothing for the creators themselves and allow scammers to run wild and free. They must have been trying to aim at NewPipe and third-party YouTube clients and yt-dlp got caught in the crossfire or has been targeted directly. What's the point in trying to block third party clients when they can just patch their code to work around your blocks?
One workaround I can think of is creating a Firefox fork and stripping out Firefox until it's just the bare browser itself and bundle that with yt-dlp. The idea is to use a modified web browser to avoid tripping this system by making it look like a legitimate browser connection has been made. But that's complicated, and forging yt-dlps user agent to report as the user's main everyday browser might work.
EDIT: The fix I mentioned above will no longer work as YouTube is now actually flagging and banning IP addresses. There is sadly no fix.
The message remains the same: YouTube hates you, even as a legitimate user or a paying customer.
Any solutation?
New error message. This one appeared after a few downloads (with an ip that never downloaded + a google account that wasn't used for that either)
ERROR: [youtube] YVkUvmDQ3HY: You’ve hit the request limit.. Your limits will reset soon, usually within an hour.
File "/opt/yt-dlp/yt_dlp/extractor/common.py", line 734, in extract
ie_result = self._real_extract(url)
File "/opt/yt-dlp/yt_dlp/extractor/youtube.py", line 4248, in _real_extract
self.raise_no_formats(reason, expected=True)
File "/opt/yt-dlp/yt_dlp/extractor/common.py", line 1257, in raise_no_formats
raise ExtractorError(msg, expected=expected, video_id=video_id)
Same issue here. Would a proxy fix it ?
Same issue here. Would a proxy fix it ?
You can wait for your IP to change naturally or for YouTube to unblacklist your IP. Most people have dynamic IPs; it may change every week, or every month. A proxy or VPN will get around this.
Getting something similar on my end, not sure if related
[youtube] Extracting URL: https://www.youtube.com/watch?v=c0_lGFdm_Ak
[youtube] Sleeping 1.0 seconds ...
[youtube] c0_lGFdm_Ak: Downloading webpage
WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests
[youtube] Sleeping 1.0 seconds ...
[youtube] c0_lGFdm_Ak: Downloading ios player API JSON
[youtube] Sleeping 1.0 seconds ...
[youtube] c0_lGFdm_Ak: Downloading iframe API JS
[youtube] Sleeping 1.0 seconds ...
[youtube] c0_lGFdm_Ak: Downloading web player API JSON
ERROR: [youtube] c0_lGFdm_Ak: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 734, in extract
ie_result = self._real_extract(url)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/youtube.py", line 4248, in _real_extract
self.raise_no_formats(reason, expected=True)
File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1257, in raise_no_formats
raise ExtractorError(msg, expected=expected, video_id=video_id)
Getting something similar on my end, not sure if related
[youtube] Extracting URL: https://www.youtube.com/watch?v=c0_lGFdm_Ak [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading webpage WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading ios player API JSON [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading iframe API JS [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading web player API JSON ERROR: [youtube] c0_lGFdm_Ak: Sign in to confirm you’re not a bot. This helps protect our community. Learn more File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 734, in extract ie_result = self._real_extract(url) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/youtube.py", line 4248, in _real_extract self.raise_no_formats(reason, expected=True) File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1257, in raise_no_formats raise ExtractorError(msg, expected=expected, video_id=video_id)
YouTube appears to be banning IPs fully now; "The following content is not available on this app. Watch on the latest version of YouTube." was just a coverup of what they were really trying to do. The only real fix is for the devs to start their own proxy and make yt-dlp use that proxy by default, so IP addresses are unique and yours won't get blacklisted. But starting a proxy is complicated and I don't really understand how a proxy works
Sign in to confirm you’re not a bot. This helps protect our community. Learn more
It seems that this error is displayed randomly, I'd say 1 in 5 times, at least according to my tests.
Getting something similar on my end, not sure if related
[youtube] Extracting URL: https://www.youtube.com/watch?v=c0_lGFdm_Ak [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading webpage WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading ios player API JSON [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading iframe API JS [youtube] Sleeping 1.0 seconds ... [youtube] c0_lGFdm_Ak: Downloading web player API JSON ERROR: [youtube] c0_lGFdm_Ak: Sign in to confirm you’re not a bot. This helps protect our community. Learn more File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 734, in extract ie_result = self._real_extract(url) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/youtube.py", line 4248, in _real_extract self.raise_no_formats(reason, expected=True) File "/home/user/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1257, in raise_no_formats raise ExtractorError(msg, expected=expected, video_id=video_id)
YouTube appears to be banning IPs fully now; "The following content is not available on this app. Watch on the latest version of YouTube." was just a coverup of what they were really trying to do. The only real fix is for the devs to start their own proxy and make yt-dlp use that proxy by default, so IP addresses are unique and yours won't get blacklisted. But starting a proxy is complicated and I don't really understand how a proxy works
They also ban residential IPs, so this solution isn't viable in my opinion.
Sign in to confirm you’re not a bot. This helps protect our community. Learn more
It seems that this error is displayed randomly, I'd say 1 in 5 times, at least according to my tests.
If they suspect you are a machine, why don't they just toss HTTP 403 or HTTP 401?
If they suspect you are a machine, why don't they just toss HTTP 403 or HTTP 401?
They will throw a 403 error when you try to access the /videoplayback
endpoint returned by the innertube /player
endpoint.
They will throw a 403 error when you try to access the
/videoplayback
endpoint returned by the innertube/player
endpoint.
If that's the case, then yt-dlp should just throw FATAL: Your IP has been blocked by YouTube.
followed by HTTP 403 rather than Sign in to confirm you’re not a bot.
YouTube has launched a crusade against adblockers and failed. They launched a new crusade against YouTube downloaders. Lets hope history repeats itself, this generates controvery and YouTube looses.
I'm curious whether it judges your behavior or simply looks at the number of IP requests.
For example, the browser might also send requests to the /api/stats/watchtime
endpoint.
Probably kinda offtopic but I'm actually interested in where that "learn more" link goes...
I'm curious whether it judges your behavior or simply looks at the number of IP requests.
For example, the browser might also send requests to the
/api/stats/watchtime
endpoint.
Judging by behavior (users report they get this block on one computer, but other systems are unaffected), it's likely an IP blacklist. Download too many times and it will trip an automated system resulting in this block. We could slow downloads down to reduce the chances of tripping the system. And to Wack0, it likely leads you to some nonsense blog post filled with tons of corporate speak on how they are trying to "protect the creators" when they are doing NOTHING for them unless you are a major public figure, a celebrity or someone major on YouTube.
where that "learn more" link goes...
it goes here: https://support.google.com/youtube/answer/3037019#zippy=%2Ccheck-that-youre-signed-into-youtube
So does it go away if you're logged in on the browser and pass the browser cookies as a parameter? (Would test but am not in a position to do so right now — just want to throw the idea out there in case it helps someone.) EDIT: please don't, see reply here for why 👇
pass the browser cookies as a parameter?
Might wanna see this. Using browser cookies is probably a terrible idea.
Might wanna see this. Using browser cookies is probably a terrible idea.
Thank you - sorry, I hadn't seen that (or I wouldn't have suggested it). Yes it does sound like a terrible idea 🤦
Interested in how they're identifying that yt-dlp requests are not the same as regular requests from their app or the browser 🤔 (presuming the same thing doesn't happen if you watch too many videos in your browser)
Interested in how they're identifying that yt-dlp requests are not the same as regular requests from their app or the browser 🤔 (presuming the same thing doesn't happen if you watch too many videos in your browser)
You may have potentially found the key to working around this blockade that YouTube set up. yt-dlp may need to imitate and forge proper requests. But that may still not work; and the best solution is to avoid the block in the first place which sadly may mean slowing the connection or setting a counter for how many downloads you can do per day. It will cripple yt-dlp but at least work around the banning.
Thank you - sorry, I hadn't seen that (or I wouldn't have suggested it). Yes it does sound like a terrible idea 🤦
That was in a different repo, so you probably wouldn't have found it anyways. No need to apologise
Interested in how they're identifying that yt-dlp requests are not the same as regular requests from their app or the browser
My guess is either user agent, or what Kane suggested above
the browser might also send requests to the /api/stats/watchtime endpoint
Although ad blockers exist, so who knows
slowing the connection or setting a counter for how many downloads you can do per day.
Keep in mind Invidious, Piped, and other alternate frontends are experiencing this too. One of the people on the bug report in the Invidious one said it happened on their private instances as well, with barely anyone using it. A maximum download count is probably not the main issue here, or maybe it's a combination of issues.
it happened on their private instances as well, with barely anyone using it
that's because it's (seemingly) ip-based a/b testing. cloud provider's ips are probably more likely to be affected than residential ips.
client does not matter, even youtube's own web client will give you a sign in error if your ip (in this case of a vpn) isn't deemed good enough: https://github.com/TeamNewPipe/NewPipe/issues/11139#issuecomment-2154637732
Oh true, I forgot that small instances tend to be run on VPSs. Maybe it could just be a maximum download count then.
Oh true, I forgot that small instances tend to be run on VPSs. Maybe it could just be a maximum download count then.
If you don't download frequently or download with self-restraint then you should be unaffected. This is only affecting those who download frequently. Those who download in bulk or download playlists have the greatest risk of facing this block. YouTube is harming both personal non-commercial use of YouTube videos and data preservation efforts, and they will very likely continue these practices. This is anti-consumer and anti-user, and shouldn't be legal.
client does not matter, even youtube's own web client will give you a sign in error if your ip (in this case of a vpn) isn't deemed good enough: TeamNewPipe/NewPipe#11139 (comment)
An edit to that comment says the same video then plays again without demanding a login if the IP address changes (VPN is disabled). So it seems like what's critical here is not causing the IP address to get blocked in the first place.
I think there's a reasonable case for personal noncommercial downloads of YouTube videos and content from other streaming sites — not everyone is blessed with the kind of bleeding edge bandwidth so many sites demand to watch content at a good quality — and for data preservation as well.
If someone is downloading massive quantities of content quickly, though, I can see a reasonable case for whoever runs that infrastructure to want the requests to be throttled so it doesn't deny service to others. I know that's only the fig leaf YouTube would use here to justify what their motives really are, but the fig leaf in itself isn't entirely unreasonable.
Is it worth having some kind of throttling as the default setting (at least until this is figured out) and letting the user remove it with a command switch? Or have that as the default for whole playlist downloads, even if not for individual video downloads?
If it really was about stopping bots Google could just throw their "I'm not a robot" recaptcha.
So I agree with those thinking this is mostly to stop downloaders. This benefit both Google and those producing content worth throwing ads at, if they manage to do that.
And if logged into account at Google they can do more statistics if they get people to sign in more.
But incredible devs at yt-dlp manage to work around things before, so probably will again.
Going with Brave browser give the same "sign in....bot"-suggestion so it's not just yt-dlp, it's general problem and wanting people to sign in more, as well, as I see it.
And I do vpn all the time, and swapping servers and it might go through in the end. And all cookies and history is cleared in Brave every time I start it. So new ip all the time and still get it mostly on some files.
If it really was about stopping bots Google could just throw their "I'm not a robot" recaptcha.
- you can configure that to do really abusive tests as well as rather modest
So I agree with those thinking this is mostly to stop downloaders. This benefit both Google and those producing content worth throwing ads at, if they manage to do that.
And if logged into account at Google they can do more statistics if they get people to sign in more.
But incredible devs at yt-dlp manage to work around things before, so probably will again.
Going with Brave browser give the same "sign in....bot"-suggestion so it's not just yt-dlp, it's general problem and wanting people to sign in more, as well, as I see it.
- and just on some files, not all
- I went back and forth a couple of times, and only certain files give "bot"-message
And I do vpn all the time, and swapping servers and it might go through in the end. And all cookies and history is cleared in Brave every time I start it. So new ip all the time and still get it mostly on some files.
And the only reason why they are stopping downloaders is because it harms their bottom line. They went on a crusade against adblock and failed. This issue with them blocking downloaders is already sparking controversy as it broke music bots on Discord and it won't be long before this (hopefully) reaches the mainstream.
Hopefully Google find that side effects are worse and getting less views on content and thereby less ads played so they retract the idea.
That browsers don't get access either will create resentment from content producers as well.
The cure become worse than the disease.
Is this a clue, maybe?
Using an embedded youtube video it works like here with browser https://riks.se/riksronden-ska-eu-forskare-och-experter-styra-sverige/
but if I search that on youtube and access it need login.
To emulate being embedded might be a way to download it?
Just a thought....
To emulate being embedded might be a way to download it?
Note that the site you linked uses youtube.com and not youtube-nocookie.com, so it's using your account's cookies if you're signed in. In case you didn't already, you might want to try a private window.
To emulate being embedded might be a way to download it?
Note that the site you linked uses youtube.com and not youtube-nocookie.com, so it's using your account's cookies if you're signed in. In case you didn't already, you might want to try a private window.
A priori what he said seems correct.
My IP is currently blocked too, if I'm not logged in then I can't play videos.
I've tried going to the url mentioned without private browsing and with (without being logged into an account in both cases) and the video plays.
To emulate being embedded might be a way to download it?
Note that the site you linked uses youtube.com and not youtube-nocookie.com, so it's using your account's cookies if you're signed in. In case you didn't already, you might want to try a private window.
I have not been signed into youtube for last 5 years, so I am not now.
I tried the /embed/ url thingy from view source, but that was not enough.
So something how it's embedded make a difference it seems.
And again - it's even with browser it refuses, not only yt-dlp
Is this a clue, maybe?
Using an embedded youtube video it works like here with browser https://riks.se/riksronden-ska-eu-forskare-och-experter-styra-sverige/
but if I search that on youtube and access it need login.
- either just clicking to watch it directly on youtube or search separately on youtube
- as does yt-dlp also does needing sign in
To emulate being embedded might be a way to download it?
Just a thought....
You could try test this by setting innertube_host extractor arg (see readme) to the YouTube no cookie domain
And set the player_client extractor arg to web_embedded
There is also web_embedscreen client I believe could try test as well
i would be curious whether this change affects things like DuckDuckGo's youtube video embeds when you search for video results - if they somehow get an exception to this maybe that could lead to a workaround
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
Checklist
Please make sure the question is worded well enough to be understood
There is such an error, what can be done?
Provide verbose output that clearly demonstrates the problem
yt-dlp -vU <your command line>
)'verbose': True
toYoutubeDL
params instead[debug] Command-line config
) and insert it belowComplete Verbose Output