search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.28k stars 9.94k forks source link

Can´t download a YT video #30255

Closed doktor-trapu closed 1 year ago

doktor-trapu commented 2 years ago

Checklist

Verbose log

youtube-dl "https://www.youtube.com/watch?v=p7FCgw_GlWc" -v [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'https://www.youtube.com/watch?v=p7FCgw_GlWc', u'-v'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2021.06.06 [debug] Python version 2.7.18 (CPython) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-Ubuntu-20.04-focal [debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4 [debug] Proxy map: {} [youtube] p7FCgw_GlWc: Downloading webpage [youtube] p7FCgw_GlWc: Refetching age-gated info webpage WARNING: unable to download video info webpage: HTTP Error 410: Gone Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/local/bin/youtube-dl/main.py", line 19, in File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 475, in main File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 465, in _real_main File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2069, in download File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1503, in _real_extract File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2644, in compat_parse_qs File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2614, in _parse_qsl AttributeError: 'bool' object has no attribute 'split'



## Description

Hey, i am trying to download a youtube video (https://www.youtube.com/watch?v=p7FCgw_GlWc). It shows me an "WARNING: unable to download video info webpage: HTTP Error 410: Gone" error, i am not really sure what i am doing wrong. I am running youtube-dl on Windows 11 using WSL. AFAIK, the video is age-restricted, but is public and accessable with my browser - i dont see how that could be a problem.
jmf1988 commented 2 years ago

The video is not public it requires you to sign in to confirm your age, public means not sign in required.

ghost commented 2 years ago

Duplicate. Was once addressed in master 03ab02730f77da5b7ad05ca78ff1624d8226ec5f but it doesn't work now.

May be able to get around by passing cookies. Or try yt-dlp.

nicolaasjan commented 2 years ago

May be able to get around by passing cookies. Or try yt-dlp.

Yt-dlp works without passing cookies. :)

yt-dlp -v https://www.youtube.com/watch?v=p7FCgw_GlWc
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=p7FCgw_GlWc']
[debug] User config file: /home/nico/.config/yt-dlp/config
[debug] User config: ['-o', '/dev/shm/test-dlp/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4][vcodec!*=av01]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--add-metadata', '--embed-thumbnail', '--force-ipv4', '--sponsorblock-remove', 'all', '--downloader', 'aria2c', '--downloader-args', 'aria2c:-x 10 -s 10 -j 10 -k 1M --log-level=info --file-allocation=none']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2021.11.16 (zip)
[debug] Python version 3.6.9 (CPython 64bit) - Linux-5.4.0-90-generic-x86_64-with-LinuxMint-19.3-tricia
[debug] exe versions: ffmpeg N-104554-gdb932241ee-Nico-20211116-With-VP9-Timestamp-Fix (setts,fdk), ffprobe N-104554-gdb932241ee-Nico-20211116-With-VP9-Timestamp-Fix, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=p7FCgw_GlWc
[youtube] p7FCgw_GlWc: Downloading webpage
[youtube] p7FCgw_GlWc: Downloading android player API JSON
[youtube] p7FCgw_GlWc: Downloading android agegate player API JSON
[youtube] p7FCgw_GlWc: Downloading web agegate player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[SponsorBlock] Fetching SponsorBlock segments
[debug] SponsorBlock query: https://sponsor.ajay.app/api/skipSegments/dd49?service=YouTube&categories=%5B%22selfpromo%22%2C+%22intro%22%2C+%22music_offtopic%22%2C+%22preview%22%2C+%22interaction%22%2C+%22sponsor%22%2C+%22outro%22%5D
WARNING: <urlopen error timed out>. Retrying...
[SponsorBlock] Found 2 segments in the SponsorBlock database
[info] p7FCgw_GlWc: Downloading 1 format(s): 137+140
[info] Downloading video thumbnail 41 ...
WARNING: Unable to download video thumbnail 41: HTTP Error 404: Not Found
[info] Downloading video thumbnail 40 ...
[info] Writing video thumbnail 40 to: /dev/shm/test-dlp/Kanye West - Famous.jpg
[debug] Invoking downloader on "https://r1---sn-5hnedn7s.googlevideo.com/videoplayback?expire=1637508380&ei=vBCaYdLlFZuc8gPmm4uADQ&ip=<REDACTED>&id=o-AGiafiVMF3YjFkoYxMl7Hx0_Ll9dsN98eP7AWQY1u4pD&itag=137&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-5hnedn7s%2Csn-5hne6ns6&ms=au%2Crdu&mv=m&mvi=1&pl=19&gcr=nl&initcwndbps=1383750&vprv=1&mime=video%2Fmp4&gir=yes&clen=226969479&dur=636.902&lmt=1637456618561210&mt=1637486447&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhANWOQmttHZ6DK550xd9AWtvEa_8V30fjSd6Y5XpbCZeLAiEAibgzF5MNDaGRjrESxpoCQyfinvR3WJ_L65enTkJaAOA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIWlQqnX3gNsQ8bl-5naNSdClR3lIbtiZphKMF4etVNmAiEA0imXwQ1oJdik9b7uTAN234w84OiJNmx2SaSzs_CQbwA%3D"
[download] Destination: /dev/shm/test-dlp/Kanye West - Famous.f137.mp4
[debug] aria2c command line: aria2c -c --console-log-level=warn --summary-interval=0 --download-result=hide --file-allocation=none -x16 -j16 -s16 --min-split-size 1M --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.75 Safari/537.36' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Encoding: gzip, deflate' --header 'Accept-Language: en-us,en;q=0.5' --interface 0.0.0.0 --check-certificate=true --remote-time=false -x 10 -s 10 -j 10 -k 1M --log-level=info --file-allocation=none --dir /dev/shm/test-dlp/ --out './Kanye West - Famous.f137.mp4.part' --auto-file-renaming=false -- 'https://r1---sn-5hnedn7s.googlevideo.com/videoplayback?expire=1637508380&ei=vBCaYdLlFZuc8gPmm4uADQ&ip=<REDACTED>&id=o-AGiafiVMF3YjFkoYxMl7Hx0_Ll9dsN98eP7AWQY1u4pD&itag=137&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-5hnedn7s%2Csn-5hne6ns6&ms=au%2Crdu&mv=m&mvi=1&pl=19&gcr=nl&initcwndbps=1383750&vprv=1&mime=video%2Fmp4&gir=yes&clen=226969479&dur=636.902&lmt=1637456618561210&mt=1637486447&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhANWOQmttHZ6DK550xd9AWtvEa_8V30fjSd6Y5XpbCZeLAiEAibgzF5MNDaGRjrESxpoCQyfinvR3WJ_L65enTkJaAOA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIWlQqnX3gNsQ8bl-5naNSdClR3lIbtiZphKMF4etVNmAiEA0imXwQ1oJdik9b7uTAN234w84OiJNmx2SaSzs_CQbwA%3D'
[aria2c] Downloaded 226969479 bytes:8.7MiB]                                    
[download] 100% of 216.45MiB in 00:26
[debug] Invoking downloader on "https://r1---sn-5hnedn7s.googlevideo.com/videoplayback?expire=1637508380&ei=vBCaYdLlFZuc8gPmm4uADQ&ip=<REDACTED>&id=o-AGiafiVMF3YjFkoYxMl7Hx0_Ll9dsN98eP7AWQY1u4pD&itag=140&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-5hnedn7s%2Csn-5hne6ns6&ms=au%2Crdu&mv=m&mvi=1&pl=19&gcr=nl&initcwndbps=1383750&vprv=1&mime=audio%2Fmp4&gir=yes&clen=10309815&dur=636.992&lmt=1637451611628671&mt=1637486447&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgP-P25XKhehBpF_b9NxjIo3uXFh4n5pUmID_mf474ixcCIF50dp4AR4OS37nvQgmMMvQe75FIkAtk5uIx-RJ-vgbf&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIWlQqnX3gNsQ8bl-5naNSdClR3lIbtiZphKMF4etVNmAiEA0imXwQ1oJdik9b7uTAN234w84OiJNmx2SaSzs_CQbwA%3D"
[download] Destination: /dev/shm/test-dlp/Kanye West - Famous.f140.m4a
[debug] aria2c command line: aria2c -c --console-log-level=warn --summary-interval=0 --download-result=hide --file-allocation=none -x16 -j16 -s16 --min-split-size 1M --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.75 Safari/537.36' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Encoding: gzip, deflate' --header 'Accept-Language: en-us,en;q=0.5' --interface 0.0.0.0 --check-certificate=true --remote-time=false -x 10 -s 10 -j 10 -k 1M --log-level=info --file-allocation=none --dir /dev/shm/test-dlp/ --out './Kanye West - Famous.f140.m4a.part' --auto-file-renaming=false -- 'https://r1---sn-5hnedn7s.googlevideo.com/videoplayback?expire=1637508380&ei=vBCaYdLlFZuc8gPmm4uADQ&ip=<REDACTED>&id=o-AGiafiVMF3YjFkoYxMl7Hx0_Ll9dsN98eP7AWQY1u4pD&itag=140&source=youtube&requiressl=yes&mh=a3&mm=31%2C29&mn=sn-5hnedn7s%2Csn-5hne6ns6&ms=au%2Crdu&mv=m&mvi=1&pl=19&gcr=nl&initcwndbps=1383750&vprv=1&mime=audio%2Fmp4&gir=yes&clen=10309815&dur=636.992&lmt=1637451611628671&mt=1637486447&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgP-P25XKhehBpF_b9NxjIo3uXFh4n5pUmID_mf474ixcCIF50dp4AR4OS37nvQgmMMvQe75FIkAtk5uIx-RJ-vgbf&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIWlQqnX3gNsQ8bl-5naNSdClR3lIbtiZphKMF4etVNmAiEA0imXwQ1oJdik9b7uTAN234w84OiJNmx2SaSzs_CQbwA%3D'
[aria2c] Downloaded 10309815 bytesL:1.5MiB]                                    
[download] 100% of 9.83MiB in 00:07
[Merger] Merging formats into "/dev/shm/test-dlp/Kanye West - Famous.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/dev/shm/test-dlp/Kanye West - Famous.f137.mp4' -i 'file:/dev/shm/test-dlp/Kanye West - Famous.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:/dev/shm/test-dlp/Kanye West - Famous.temp.mp4'
Deleting original file /dev/shm/test-dlp/Kanye West - Famous.f140.m4a (pass -k to keep)
Deleting original file /dev/shm/test-dlp/Kanye West - Famous.f137.mp4 (pass -k to keep)
[debug] ffprobe command line: ffprobe -hide_banner -show_format -show_streams -print_format json 'file:/dev/shm/test-dlp/Kanye West - Famous.mp4'
[ModifyChapters] Removing chapters from /dev/shm/test-dlp/Kanye West - Famous.mp4
[debug] Writing concat spec to /dev/shm/test-dlp/Kanye West - Famous.temp.mp4.concat
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -hide_banner -nostdin -f concat -safe 0 -i 'file:/dev/shm/test-dlp/Kanye West - Famous.temp.mp4.concat' -c copy -c:s mov_text -movflags +faststart 'file:/dev/shm/test-dlp/Kanye West - Famous.temp.mp4'
Deleting original file /dev/shm/test-dlp/Kanye West - Famous.uncut.mp4 (pass -k to keep)
[Metadata] Adding metadata to "/dev/shm/test-dlp/Kanye West - Famous.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/dev/shm/test-dlp/Kanye West - Famous.mp4' -i 'file:/dev/shm/test-dlp/Kanye West - Famous.meta' -map 0 -dn -c copy -map_metadata 1 -metadata 'title=Kanye West - Famous' -metadata date=20160701 -metadata 'description=Kanye West - Famous

http://vevo.ly/cVDXmL' -metadata 'synopsis=Kanye West - Famous

http://vevo.ly/cVDXmL' -metadata 'purl=https://www.youtube.com/watch?v=p7FCgw_GlWc' -metadata 'comment=https://www.youtube.com/watch?v=p7FCgw_GlWc' -metadata artist=KanyeWestVEVO 'file:/dev/shm/test-dlp/Kanye West - Famous.temp.mp4'
[EmbedThumbnail] mutagen: Adding thumbnail to "/dev/shm/test-dlp/Kanye West - Famous.mp4"
Deleting original file /dev/shm/test-dlp/Kanye West - Famous.jpg (pass -k to keep)
dirkf commented 2 years ago

This patch back-ports the Android embed client work-around from yt-dlp to achieve the same result:

--- old/youtube-dl/youtube_dl/extractor/youtube.py
+++ new/youtube-dl/youtube_dl/extractor/youtube.py
@@ -283,15 +283,18 @@
     _YT_INITIAL_PLAYER_RESPONSE_RE = r'ytInitialPlayerResponse\s*=\s*({.+?})\s*;'
     _YT_INITIAL_BOUNDARY_RE = r'(?:var\s+meta|</script|\n)'

-    def _call_api(self, ep, query, video_id, fatal=True):
+    def _call_api(self, ep, query, video_id, fatal=True, headers=None):
         data = self._DEFAULT_API_DATA.copy()
         data.update(query)
+        real_headers = {'content-type': 'application/json'}
+        if headers:
+            real_headers.update(headers)

         return self._download_json(
             'https://www.youtube.com/youtubei/v1/%s' % ep, video_id=video_id,
             note='Downloading API JSON', errnote='Unable to download API page',
             data=json.dumps(data).encode('utf8'), fatal=fatal,
-            headers={'content-type': 'application/json'},
+            headers=real_headers,
             query={'key': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8'})

     def _extract_yt_initial_data(self, video_id, webpage):
@@ -1503,26 +1506,28 @@
                 'player', {'videoId': video_id}, video_id)

         playability_status = player_response.get('playabilityStatus') or {}
-        if playability_status.get('reason') == 'Sign in to confirm your age':
-            video_info = self._download_webpage(
-                base_url + 'get_video_info', video_id,
-                'Refetching age-gated info webpage',
-                'unable to download video info webpage', query={
-                    'video_id': video_id,
-                    'eurl': 'https://youtube.googleapis.com/v/' + video_id,
-                    'html5': 1,
-                    # See https://github.com/ytdl-org/youtube-dl/issues/29333#issuecomment-864049544
-                    'c': 'TVHTML5',
-                    'cver': '6.20180913',
-                }, fatal=False)
-            if video_info:
-                pr = self._parse_json(
-                    try_get(
-                        compat_parse_qs(video_info),
-                        lambda x: x['player_response'][0], compat_str) or '{}',
-                    video_id, fatal=False)
-                if pr and isinstance(pr, dict):
-                    player_response = pr
+        if (playability_status.get('reason') == 'Sign in to confirm your age'
+            and int_or_none(self._downloader.params.get('age_limit'), default=18) >= 18):
+            self.report_age_confirmation()
+            query = {
+                'playbackContext': {'contentPlaybackContext': {'html5Preference': 'HTML5_PREF_WANTS'}},
+                'contentCheckOk': True,
+                'racyCheckOk': True,
+                'context': {
+                    'client': {'clientName': 'ANDROID', 'clientVersion': '16.20', 'hl': 'en', 'clientScreen': 'EMBED'},
+                    'thirdParty': {'embedUrl': 'https://google.com'},
+                },
+                'videoId': video_id,
+            }
+            headers = {
+                'X-YouTube-Client-Name': '3',
+                'X-YouTube-Client-Version': '16.20',
+                'Origin': 'https://www.youtube.com'
+            }
+            video_info = self._call_api('player', query, video_id, fatal=False, headers=headers)
+            if video_info and isinstance(video_info, dict):
+                player_response = video_info
+                playability_status = player_response.get('playabilityStatus') or {}

         trailer_video_id = try_get(
             playability_status,

Then:

$ python -m youtube_dl -F 'https://www.youtube.com/watch?v=p7FCgw_GlWc'
[youtube] p7FCgw_GlWc: Downloading webpage
[youtube] Confirming age
[youtube] p7FCgw_GlWc: Downloading API JSON
[info] Available formats for p7FCgw_GlWc:
format code  extension  resolution note
139          m4a        audio only tiny   48k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 3.71MiB
249          webm       audio only tiny   51k , webm_dash container, opus @ 51k (48000Hz), 3.89MiB
250          webm       audio only tiny   66k , webm_dash container, opus @ 66k (48000Hz), 5.05MiB
251          webm       audio only tiny  127k , webm_dash container, opus @127k (48000Hz), 9.70MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 9.83MiB
160          mp4        256x144    144p   33k , mp4_dash container, avc1.4d400c@  33k, 30fps, video only, 2.53MiB
278          webm       256x144    144p   78k , webm_dash container, vp9@  78k, 30fps, video only, 5.98MiB
394          mp4        256x144    144p   84k , mp4_dash container, av01.0.00M.08@  84k, 30fps, video only, 6.40MiB
133          mp4        426x240    240p   55k , mp4_dash container, avc1.4d4015@  55k, 30fps, video only, 4.23MiB
242          webm       426x240    240p   82k , webm_dash container, vp9@  82k, 30fps, video only, 6.24MiB
395          mp4        426x240    240p  189k , mp4_dash container, av01.0.00M.08@ 189k, 30fps, video only, 14.41MiB
134          mp4        640x360    360p  114k , mp4_dash container, avc1.4d401e@ 114k, 30fps, video only, 8.67MiB
243          webm       640x360    360p  142k , webm_dash container, vp9@ 142k, 30fps, video only, 10.85MiB
396          mp4        640x360    360p  301k , mp4_dash container, av01.0.01M.08@ 301k, 30fps, video only, 22.87MiB
135          mp4        854x480    480p  193k , mp4_dash container, avc1.4d401f@ 193k, 30fps, video only, 14.71MiB
244          webm       854x480    480p  234k , webm_dash container, vp9@ 234k, 30fps, video only, 17.77MiB
397          mp4        854x480    480p  547k , mp4_dash container, av01.0.04M.08@ 547k, 30fps, video only, 41.59MiB
136          mp4        1280x720   720p  452k , mp4_dash container, avc1.4d401f@ 452k, 30fps, video only, 34.36MiB
247          webm       1280x720   720p  470k , webm_dash container, vp9@ 470k, 30fps, video only, 35.71MiB
398          mp4        1280x720   720p 1112k , mp4_dash container, av01.0.05M.08@1112k, 30fps, video only, 84.46MiB
399          mp4        1920x1080  1080p 1759k , mp4_dash container, av01.0.08M.08@1759k, 30fps, video only, 133.61MiB
248          webm       1920x1080  1080p 1767k , webm_dash container, vp9@1767k, 30fps, video only, 134.21MiB
137          mp4        1920x1080  1080p 2850k , mp4_dash container, avc1.640028@2850k, 30fps, video only, 216.45MiB
17           3gp        176x144    144p   68k , mp4v.20.3, 7fps, mp4a.40.2 (22050Hz), 5.22MiB
18           mp4        640x360    360p  509k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 38.68MiB
22           mp4        1280x720   720p 2260k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)
$

Videos like Cr381pDsSsA (Some Girls from Equestria have a ...) that reject embedding are not handled.

dirkf commented 2 years ago

That ID doesnt match that description.

What happens when you try to remember what you did yesterday instead of looking it up!

Reino17 commented 2 years ago

This patch back-ports the Android embed client work-around from yt-dlp to achieve the same result [...]

@dirkf Python is not really my cup of thee, so if you don't mind I was wondering if you could tell me what is happening exactly.

The initial call, sending {"context":{"client":{"clientName":"ANDROID","clientVersion":"17.31.35"}},"videoId":"p7FCgw_GlWc"} in a POST-request to https://www.youtube.com/youtubei/v1/player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 indeed returns:

{
  [...]
  "playabilityStatus": {
    "status": "LOGIN_REQUIRED",
    "reason": "This video may be inappropriate for some users.",
    [...]
  }
  [...]
}

But what needs to happen then? If I send {"playbackContext":{"contentPlaybackContext":{"html5Preference":"HTML5_PREF_WANTS"}},"contentCheckOk":True,"racyCheckOk":True,"context":{"client":{"clientName":"ANDROID","clientVersion":"16.20","hl":"en","clientScreen":"EMBED"},"thirdParty":{"embedUrl":"https://google.com"}},"videoId":"p7FCgw_GlWc"} in a POST-request, along with the headers Content-Type: application/json, X-YouTube-Client-Name: 3, X-YouTube-Client-Version: 16.20, Origin: https://www.youtube.com, to https://www.youtube.com/youtubei/v1/player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 the "playabilityStatus" is still "LOGIN_REQUIRED".

dirkf commented 2 years ago

Try the linked PR #31043 which worked when I last tested it.

There is a separate issue #31164 with YT today but it doesn't affect this.

Reino17 commented 2 years ago

I know about your PR. I have a working binary with your PR applied.
I also know about TVHTML5_SIMPLY_EMBEDDED_PLAYER. Very interesting stuff.

However, I'm quoting your post about a work-around for the Android approach. Am I right to say that this work-around might have worked when you posted it, but not anymore today?

dirkf commented 2 years ago

Well, check the dates for yourself (yes). The PR fixes the work-around as much as it can be. I am not a YT expert (indeed I rarely access YT content except using yt-dl, Invidious, etc) and I would be pleased to hear of any other approaches.

baptx commented 1 year ago

@dirkf "Videos like Cr381pDsSsA (Some Girls from Equestria have a ...) that reject embedding are not handled." Does it mean that this video still cannot be downloaded with youtube-dl without giving logged in cookies (even with yt-dlp)?

ghost commented 1 year ago

@baptx those type cannot be anonymously downloaded with ANY client, which includes the official YouTube web client, official YouTube Android client, as well as YouTube-DL, YT-DLP and any other client you can think of. its just simply not possible. for those videos you MUST provide some authentication, either Cookie header or OAuth2 Authorization header.

baptx commented 1 year ago

@1268 In this case, do you know why the video mentioned in this issue (https://www.youtube.com/watch?v=p7FCgw_GlWc) that also says "This video is age-restricted and only available on YouTube." on https://www.youtube.com/embed/p7FCgw_GlWc can currently be downloaded anonymously with youtube-dl? I don't see what is different compared to Cr381pDsSsA, does it mean the uploader can check an option to choose if the video can be watched anonymously or not?

ghost commented 1 year ago

@baptx I dont know what the difference is, but there is one. maybe someone has a method to determine what level of security each video has, but I am not aware of one. I just trial and error. but I know that the hacks that work on some videos (like you mentioned) dont work on all, and no method is publicly known (that I am aware of) that allows anonymous download of ALL free videos.

baptx commented 1 year ago

@1268 I intercepted the HTTP requests of youtube-dl with mitmproxy like that: proxychains youtube-dl --no-check-certificate https://www.youtube.com/watch?v=Cr381pDsSsA And exported as cURL command the HTTP request that allows to check if a video can be downloaded anonymously or not: curl -H 'Host: www.youtube.com' -H 'Cookie: CONSENT=YES+cb.20210328-17-p0.en+FX+477; GPS=1; YSC=ES6rBztzKdI; VISITOR_INFO1_LIVE=H-jN7BrqCxE' -H 'Content-Type: application/json' -H 'X-Youtube-Client-Name: 85' -H 'X-Youtube-Client-Version: 2.0' -H 'Origin: https://www.youtube.com' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.57 Safari/537.36' -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --compressed -H 'Accept-Language: en-us,en;q=0.5' -H 'Connection: close' -X POST 'https://www.youtube.com/youtubei/v1/player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -d '{"context": {"client": {"clientName": "TVHTML5_SIMPLY_EMBEDDED_PLAYER", "clientVersion": "2.0", "hl": "en", "clientScreen": "EMBED"}, "thirdParty": {"embedUrl": "https://google.com"}}, "playbackContext": {"contentPlaybackContext": {"html5Preference": "HTML5_PREF_WANTS", "signatureTimestamp": 19571}}, "contentCheckOk": true, "racyCheckOk": true, "videoId": "Cr381pDsSsA"}' This way we can replace the videoId in the cURL command to see if a video can be downloaded anonymously, without trying to download it with youtube-dl. The only useful information I can see in the response is the following one:

  "playabilityStatus": {
    "status": "UNPLAYABLE",
    "reason": "Playback on other websites has been disabled by the video owner"
  }

So I guess p7FCgw_GlWc works because it allows embed even if it is age restricted (but YouTube does not allow it directly using https://www.youtube.com/embed/p7FCgw_GlWc).

dirkf commented 1 year ago

Exactly. The YT extractor uses the TVHTML5 client to get the video details when a video is age-restricted (and no user authentication data that would pass the age test has been provided). This client automatically allows embeddable videos regardless of age restriction. The yt-dlp YT extractor supports many more client types, selectable by extractor-specific command-line options.

Vangelis66 commented 1 year ago

"Videos like Cr381pDsSsA (Some Girls from Equestria have a ...) that reject embedding are not handled." Does it mean that this video still cannot be downloaded with youtube-dl without giving logged in cookies (even with yt-dlp)?

... For those "very" difficult 😜 to fetch age-gated videos, the solution is to use yt-dlp with the YTAgeGateBypass plugin: 

yt-dlp -vF "Cr381pDsSsA" --extractor-args youtube:no-video-proxy => 

[debug] Command-line config: ['--ffmpeg-location', '..\\FFmpeg', '--downloader-args', 'ffmpeg:-v 8 -stats', '-vF', 'Cr381pDsSsA', '--extractor-args', 'youtube:no-video-proxy']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version nightly@2023.08.04.221808 [db7b054a6] (win_x86_exe)
[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168 (setts), ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Extractor Plugins: AGB+NSIG (YoutubeIE)
[debug] Plugin directories: ['<redacted>\\yt-dlp\\yt-dlp-plugins\\YTAgeGateBypass.zip\\yt_dlp_plugins', '<redacted>\\yt-dlp\\yt-dlp-plugins\\YTNSigProxy.zip\\yt_dlp_plugins']
[debug] Loaded 1860 extractors
[youtube+AGB+NSIG] Extracting URL: Cr381pDsSsA
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading webpage
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading ios player API JSON
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading tv embedded player API JSON
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading android player API JSON
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading android embedded player API JSON
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading web embedded client config
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading player 4eae42b1
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading web embedded player API JSON
[youtube+AGB+NSIG] Cr381pDsSsA: Downloading Zerody API JSON
[debug] [youtube+AGB+NSIG] Extracting signature function js_4eae42b1_106
[debug] Loading youtube-sigfuncs.js_4eae42b1_106 from cache
[debug] Loading youtube-nsig.4eae42b1 from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig mZgP2hZg1bNym4JUs => P-2PbBA3FH6d_w
[debug] Loading youtube-nsig.4eae42b1 from cache
[debug] [youtube+AGB+NSIG] Decrypted nsig V1xqE4xPitbZK6dML => s2Fyc2AXzf77iw
[debug] [youtube+AGB+NSIG] Extracting signature function js_4eae42b1_102
[debug] Loading youtube-sigfuncs.js_4eae42b1_102 from cache
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[info] Available formats for Cr381pDsSsA:
ID  EXT  RESOLUTION FPS CH |   FILESIZE   TBR PROTO | VCODEC        VBR ACODEC    ABR ASR MORE INFO
-------------------------------------------------------------------------------------------------------------
249 webm audio only      2 |    5.74MiB   50k https | audio only        opus    50k 48k low, webm_dash
250 webm audio only      2 |    7.44MiB   65k https | audio only        opus    65k 48k low, webm_dash
140 m4a  audio only      2 |   14.51MiB  127k https | audio only        mp4a.40.2  127k 44k medium, m4a_dash
251 webm audio only      2 |   14.32MiB  125k https | audio only        opus   125k 48k medium, webm_dash
160 mp4  256x144     30    |    8.64MiB   76k https | avc1.4d400c   76k video only          144p, mp4_dash
278 webm 256x144     30    |    9.45MiB   83k https | vp9           83k video only          144p, webm_dash
133 mp4  426x240     30    |   16.82MiB  147k https | avc1.4d4015  147k video only          240p, mp4_dash
242 webm 426x240     30    |   18.67MiB  163k https | vp9          163k video only          240p, webm_dash
134 mp4  640x360     30    |   39.35MiB  345k https | avc1.4d401e  345k video only          360p, mp4_dash
18  mp4  640x360     30  2 | ≈ 55.07MiB  471k https | avc1.42001E       mp4a.40.2       44k 360p
243 webm 640x360     30    |   33.57MiB  294k https | vp9          294k video only          360p, webm_dash
135 mp4  854x480     30    |   71.37MiB  625k https | avc1.4d401f  625k video only          480p, mp4_dash
244 webm 854x480     30    |   57.65MiB  505k https | vp9          505k video only          480p, webm_dash
136 mp4  1280x720    30    |  133.29MiB 1167k https | avc1.4d401f 1167k video only          720p, mp4_dash
247 webm 1280x720    30    |  114.86MiB 1006k https | vp9         1006k video only          720p, webm_dash

... All the difference is being made by this line: 

[youtube+AGB+NSIG] Cr381pDsSsA: Downloading Zerody API JSON

Credits to zerodytrash :heart: and pukkandan :heart: ...

ghost commented 1 year ago

I am not discouraging the above solution, but I think it should be more explicit about what its doing.

the above solution has not "solved" the problem of authentication. ultimately any requests even with that code are STILL authenticating. the only difference is the authentication is run through a proxy, so the end user themself doesn't have to authenticate. this is pretty cool, but it also could leak information about the user to the proxy itself, so I would caution its use, or at least understand what you are doing before blindly installing this plugin.

nicolaasjan commented 1 year ago

@Vangelis66

(offtopic)

[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)

yt-dlp soon dropping support for Python 3.7...