search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.38k stars 10.04k forks source link

ERROR: Unable to extract yt initial data on playlist #32499

Open Fabian200703 opened 1 year ago

Fabian200703 commented 1 year ago

Checklist

Verbose log

PASTE VERBOSE LOG HERE

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '-x', 'https://youtube.com/playlist?list=PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2021.12.17 [debug] Python version 3.8.10 (CPython) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 [debug] exe versions: none [debug] Proxy map: {} [youtube:tab] PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a: Downloading webpage ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper return func(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info ie_result = ie.extract(url) File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 534, in extract ie_result = self._real_extract(url) File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/youtube.py", line 2841, in _real_extract data = self._extract_yt_initial_data(item_id, webpage) File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/youtube.py", line 299, in _extract_yt_initial_data self._search_regex( File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex raise RegexNotFoundError('Unable to extract %s' % _name) youtube_dl.utils.RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

WRITE DESCRIPTION HERE

as you can see above the program fails to download even a single video from the playlist i was using the minimum of arguments in the command and trying the same command with single videos from the playlist produces a different error: "ERROR: Unable to extract uploader id;" this seems to be consistent across the entire playlist

Fabian200703 commented 1 year ago

this happens on the nightly build:

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '-x', 'https://youtube.com/playlist?list=PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a'] [debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252 [debug] youtube-dl version 2023.08.07 [86e3cf5e5] (single file build) [debug] This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] For support, visit the main site. [debug] Python 3.4.4 (CPython AMD64 32bit) - Windows-10-10.0.22621 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} [youtube:tab] PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a: Downloading webpage ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 863, in wrapper File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 959, in __extract_info File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 565, in extract File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 3335, in _real_extract File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 312, in _extract_yt_initial_data File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 1045, in _search_regex youtube_dl.utils.RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

dirkf commented 1 year ago

YT is now ignoring the ucbc=1 parameter that the extractor adds and redirecting via consent.youtube.com, for some pages. There is a fix but meanwhile try passing cookies from a browser session where you have already gone through the consent check, using the nightly build.

See also https://github.com/yt-dlp/yt-dlp/issues/7594.

facundobatista commented 1 year ago

Hello! Do you have instructions on how to pass the cookies from a browser session? Thanks!

dirkf commented 1 year ago

See manual and FAQ.

Konubinix commented 1 year ago

I came accross the issue with consent yesterday.

For people using qutebrowser, you first need to convert its cookies into netscape format.

I used https://gist.github.com/guidocella/a272b6e68f9c44532b011f6596e95c61#file-dump-cookies-sh to do so and then youtube-dl --cookies $XDG_RUNTIME_DIR/cookies.txt

Just posting this in case it saves some people some trouble.

mdavis-xyz commented 1 year ago

Using a VPN to move from the EU to outside the EU is also a workaround.

ghost commented 1 year ago 
youtube-dl --cookies www.youtube.com_cookies.txt -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU --verbose
[debug] System config: [u'--no-mtime', u'-o', u'~/Downloads/YouTube/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--cookies', u'www.youtube.com_cookies.txt', u'-f', u'bestvideo[height<=480]+bestaudio/best[height<=480]', u'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2023.08.31.810 (single file build)
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 2.7.18 (CPython x86_64 64bit) - Linux-5.15.0-79-generic-x86_64-with-LinuxMint-20.3-una - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 825, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 846, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 535, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 3328, in _real_extract
    data = self._extract_yt_initial_data(item_id, webpage)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 312, in _extract_yt_initial_data
    self._YT_INITIAL_DATA_RE), webpage, 'yt initial data'),
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1013, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

So I updated to latest version with -U, then I passed cookies from my browser after consenting, making sure they are in netscape format. Still doesn't work... Would it help if I provided my cookies txt file?

dirkf commented 1 year ago

You have to have consented in the browser and then export the cookies.

Although it may not make any difference, also review #30839 to update to a newer version: it will have an earlier date, but I think the ytdl-patched repo is not currently being updated even though builds are being run.

ghost commented 1 year ago

Thank you for your answer, unfortunately it does not solve the issue, I have now updated to the nightly build but I still get the error message. I made sure to export cookies AFTER consenting to youtube cookies.

greg@greg-OptiPlex-3060:~$ youtube-dl --cookies www.youtube.com_cookies.txt -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU --verbose
[debug] System config: [u'--no-mtime', u'-o', u'~/Downloads/YouTube/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--cookies', u'www.youtube.com_cookies.txt', u'-f', u'bestvideo[height<=480]+bestaudio/best[height<=480]', u'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2023.08.07 [86e3cf5e5] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.18 (CPython x86_64 64bit) - Linux-5.15.0-79-generic-x86_64-with-LinuxMint-20.3-una - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 863, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 959, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 3335, in _real_extract
    data = self._extract_yt_initial_data(item_id, webpage)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 312, in _extract_yt_initial_data
    self._YT_INITIAL_DATA_RE), webpage, 'yt initial data'),
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1045, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
dirkf commented 1 year ago

That's quite surprising. Perhaps YT also requires you to set the UA header to match the browser that acquired the cookies. At any rate, I'm able to get the playlist with a (slightly fixed - hence why it is) WIP version:

$ python -m youtube_dl --flat-playlist 'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU'
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
[download] Downloading playlist: Superwings S1
[youtube:tab] playlist Superwings S1: Downloading 23 videos
[download] Downloading video 1 of 23
[download] Downloading video 2 of 23
[download] Downloading video 3 of 23
[download] Downloading video 4 of 23
[download] Downloading video 5 of 23
[download] Downloading video 6 of 23
[download] Downloading video 7 of 23
[download] Downloading video 8 of 23
[download] Downloading video 9 of 23
[download] Downloading video 10 of 23
[download] Downloading video 11 of 23
[download] Downloading video 12 of 23
[download] Downloading video 13 of 23
[download] Downloading video 14 of 23
[download] Downloading video 15 of 23
[download] Downloading video 16 of 23
[download] Downloading video 17 of 23
[download] Downloading video 18 of 23
[download] Downloading video 19 of 23
[download] Downloading video 20 of 23
[download] Downloading video 21 of 23
[download] Downloading video 22 of 23
[download] Downloading video 23 of 23
[download] Finished downloading playlist: Superwings S1
$
ghost commented 1 year ago

This is the output after passing my browser's User Agent header to youtube-dl. BTW - I'm using Brave browser. Unfortunately it still does not work. Note that the cookies file is named cookies2.txt it's because I also tried various methods like signing in to google, to no avail.

greg@greg-OptiPlex-3060:~$ youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" --cookies www.youtube.com_cookies2.txt -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU --verbose
[debug] System config: [u'--no-mtime', u'-o', u'~/Downloads/YouTube/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--user-agent', u'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36', u'--cookies', u'www.youtube.com_cookies2.txt', u'-f', u'bestvideo[height<=480]+bestaudio/best[height<=480]', u'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2023.08.07 [86e3cf5e5] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.18 (CPython x86_64 64bit) - Linux-5.15.0-79-generic-x86_64-with-LinuxMint-20.3-una - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 863, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 959, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 3335, in _real_extract
    data = self._extract_yt_initial_data(item_id, webpage)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 312, in _extract_yt_initial_data
    self._YT_INITIAL_DATA_RE), webpage, 'yt initial data'),
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1045, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I also tried changing browser, thinking it might be an issue with Brave's agressive privacy behaviour. So here is the output of a fresh Firefox install, with cookies passed after consent and UA header:

greg@greg-OptiPlex-3060:~$ youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" --cookies FFcookies.txt -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' 'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU' --verbose
[debug] System config: [u'--no-mtime', u'-o', u'~/Downloads/YouTube/%(title)s.%(ext)s']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--user-agent', u'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0', u'--cookies', u'FFcookies.txt', u'-f', u'bestvideo[height<=480]+bestaudio/best[height<=480]', u'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2023.08.07 [86e3cf5e5] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.18 (CPython x86_64 64bit) - Linux-5.15.0-79-generic-x86_64-with-LinuxMint-20.3-una - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 863, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 959, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 3335, in _real_extract
    data = self._extract_yt_initial_data(item_id, webpage)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 312, in _extract_yt_initial_data
    self._YT_INITIAL_DATA_RE), webpage, 'yt initial data'),
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1045, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Interestingly enough it works with another video. I clicked one of the first things that appeared on my feed (https://www.youtube.com/watch?v=FxdbD-N7pHE) and was greeted with a successfull download with the same parameters. There were errors (unable to extract player URL), but the video was still downloaded succesfully. I posted it here since I think it might be more appropriate: #https://github.com/ytdl-org/youtube-dl/issues/32538#issuecomment-1702282364

dirkf commented 1 year ago

Sadly, I'm out of ideas, except to expedite the QA on the extractor update.

mdavis-xyz commented 1 year ago

When I made that comment, I found that it didn't work for my French IP (without a VPN) and did for an Australian IP (PIA VPN to Sydney).

But now I'm trying again today in France and it kind of works. I see some tracebacks, but it still seems to download a file. So I think those tracebacks are somehow just for warnings. And I suspect they're unrelated.

$ youtube-dl "https://www.youtube.com/watch?v=IX6McAMbMIU"
[youtube] IX6McAMbMIU: Downloading webpage
[youtube] IX6McAMbMIU: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or bytes-like object, got 'NoneType' Traceback (most recent call last):
  File "/home/matthew/venv/lib/python3.11/site-packages/youtube_dl/extractor/youtube.py", line 1679, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matthew/venv/lib/python3.11/site-packages/youtube_dl/extractor/youtube.py", line 1644, in _extract_n_function
    player_id = self._extract_player_info(player_url)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matthew/venv/lib/python3.11/site-packages/youtube_dl/extractor/youtube.py", line 1476, in _extract_player_info
    id_m = re.search(player_re, player_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/re/__init__.py", line 176, in search
    return _compile(pattern, flags).search(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
)
[youtube] IX6McAMbMIU: Downloading API JSON
[dashsegments] Total fragments: 3
[download] Destination: HANABI  - les règles du jeu !-IX6McAMbMIU.f137.mp4
[download]   0.4% of ~30.00MiB at 53.94KiB/s ETA 09:27^C
dirkf commented 1 year ago

Based on https://github.com/ytdl-org/youtube-dl/issues/32533#issuecomment-1698311971, the patch from yt-dlp solves the problem for EU users. PR here RSN

ghost commented 1 year ago

this is the likely culprit:

Using a VPN to move from the EU to outside the EU is also a workaround.

if someone can name a specific country that triggers the error, I can check into this.

If it helps I live in Italy. It doesn't seem to be ISP related either. I tested it on my home network, a mobile hotspot and at work, so 3 different ISPs. I don't know why it should change anything, but I thought it was worth a try. I don't have access to any VPN at the moment, so I couldn't test if moving to a country outside EU helps in this case.

BTW I can download single videos from the playlist I mentioned above, albeit at slower speed.

Based on #32533 (comment), the patch from yt-dlp solves the problem for EU users. PR here RSN

That's some moderately good news... Should we just move to yt-dlp until a fix is released?

ghost commented 1 year ago

using Italy with current master, I do not get an outright failure, but it does fail to remove the throttling, which might be a separate issue:

> youtube-dl IX6McAMbMIU
[youtube] IX6McAMbMIU: Downloading webpage
[youtube] IX6McAMbMIU: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or bytes-like object Traceback (most recent call last):
  File "C:\python\lib\site-packages\youtube_dl\extractor\youtube.py", line 1679, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "C:\python\lib\site-packages\youtube_dl\extractor\youtube.py", line 1644, in _extract_n_function
    player_id = self._extract_player_info(player_url)
  File "C:\python\lib\site-packages\youtube_dl\extractor\youtube.py", line 1476, in _extract_player_info
    id_m = re.search(player_re, player_url)
  File "re.py", line 200, in search
TypeError: expected string or bytes-like object
)
[youtube] IX6McAMbMIU: Downloading API JSON
[dashsegments] Total fragments: 3
[download] Destination: HANABI  - les règles du jeu !-IX6McAMbMIU.f137.mp4
[download]   6.0% of ~30.00MiB at 79.29KiB/s ETA 06:04
dirkf commented 1 year ago

If the initial player response JSON can't be extracted, the extractor tries to proceed by fetching equivalent JSON from /youtubei/v1/player?videoId={video_id}.

The media links can be extracted from the .streamingData member, but the extractor only knows how to pull the player JS URL from the full web page (containing the initial player response) and not from the JSON (if it's even sent), so the unscrambling logic can't be used.

nicolaasjan commented 1 year ago

@1268 > WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled I faced the same. See issue #32534.

Until the patch arrives, you can apply this diff:

diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
index 9c419c0..3bf483c 100644
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -260,16 +260,10 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
         cookies = self._get_cookies('https://www.youtube.com/')
         if cookies.get('__Secure-3PSID'):
             return
-        consent_id = None
-        consent = cookies.get('CONSENT')
-        if consent:
-            if 'YES' in consent.value:
-                return
-            consent_id = self._search_regex(
-                r'PENDING\+(\d+)', consent.value, 'consent', default=None)
-        if not consent_id:
-            consent_id = random.randint(100, 999)
-        self._set_cookie('.youtube.com', 'CONSENT', 'YES+cb.20210328-17-p0.en+FX+%s' % consent_id)
+        socs = cookies.get('SOCS')
+        if socs and not socs.value.startswith('CAA'):  # not consented
+            return
+        self._set_cookie('.youtube.com', 'SOCS', 'CAI', secure=True)  # accept all (required for mixes)

     def _real_initialize(self):
         self._initialize_consent()

Then it works fine again:

youtube-dl -v --ignore-config IX6McAMbMIU
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', 'IX6McAMbMIU']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.09.01.1
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-162-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111711-gd295b6b693-Nico-20230808, ffprobe N-111711-gd295b6b693-Nico-20230808, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] IX6McAMbMIU: Downloading webpage
[youtube] IX6McAMbMIU: Downloading player 0a835141
[debug] [youtube] Decrypted nsig eMLFU58o3Ty3bHwmpN => PP6OPqsJ8SF-SQ
[debug] [youtube] Decrypted nsig JGB6co7DtTFZlRKSTI => YxvEq0bReXPeUg
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://rr2---sn-5hnednsz.googlevideo.com/videoplayback?expire=1694039823&ei=r6r4ZKvYGP666dsP4peCWA&ip=<REDACTED>&id=o-AE2Owhk2eLmFpqf1G0FqTEgP1BwezlGU05dRLXbbL2o1&itag=137&aitags=134%2C136%2C137%2C160%2C243&source=youtube&requiressl=yes&mh=Um&mm=31%2C26&mn=sn-5hnednsz%2Csn-4g5e6ns7&ms=au%2Conr&mv=m&mvi=2&pl=14&initcwndbps=2287500&spc=UWF9f3LCXGIQrdCG4aCP0tADEGG8B6rbIbJyHBMF4A&vprv=1&svpuc=1&mime=video%2Fmp4&ns=jfIQASlWGQvPGTF0VTpiDbwP&gir=yes&clen=25280532&dur=401.767&lmt=1693365754427220&mt=1694017749&fvip=3&keepalive=yes&fexp=24007246%2C24362685&beids=24350018&c=WEB&txp=6219224&n=PP6OPqsJ8SF-SQ&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhANPoyNjGdMsqc_doYoBWsy34L_nEI6wn9EbWwn909atVAiBFqVyq-6cllVcGYPN2R6EsVXyoiD0EeU1llgWIYYoq1Q%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgFz9MBbhqbJ_Ev0EWGo2rRJYdQEZ-1Ck4qdyYCWsq9GYCIBj8HKenFbJkrEyhmp4NSj10w-GqcLqrj17bD8Y7CXQQ'
[dashsegments] Total fragments: 3
[download] Destination: HANABI  - les règles du jeu !-IX6McAMbMIU.f137.mp4
[download] 100% of 24.11MiB in 00:02
[debug] Invoking downloader on 'https://rr2---sn-5hnednsz.googlevideo.com/videoplayback?expire=1694039823&ei=r6r4ZKvYGP666dsP4peCWA&ip=<REDACTED>&id=o-AE2Owhk2eLmFpqf1G0FqTEgP1BwezlGU05dRLXbbL2o1&itag=140&source=youtube&requiressl=yes&mh=Um&mm=31%2C26&mn=sn-5hnednsz%2Csn-4g5e6ns7&ms=au%2Conr&mv=m&mvi=2&pl=14&initcwndbps=2287500&spc=UWF9f3LCXGIQrdCG4aCP0tADEGG8B6rbIbJyHBMF4A&vprv=1&svpuc=1&mime=audio%2Fmp4&ns=jfIQASlWGQvPGTF0VTpiDbwP&gir=yes&clen=6504156&dur=401.844&lmt=1693365762273497&mt=1694017749&fvip=3&keepalive=yes&fexp=24007246%2C24362685&beids=24350018&c=WEB&txp=6218224&n=PP6OPqsJ8SF-SQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAI0HlshLS04cTLtsvCEmR84feV0gb-jUCImXXCz0VjOoAiA5AFpNKO-xA_2-YXUgP0kxsQiqVwxVsaZI8xRAvjvRRQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgFz9MBbhqbJ_Ev0EWGo2rRJYdQEZ-1Ck4qdyYCWsq9GYCIBj8HKenFbJkrEyhmp4NSj10w-GqcLqrj17bD8Y7CXQQ'
[dashsegments] Total fragments: 1
[download] Destination: HANABI  - les règles du jeu !-IX6McAMbMIU.f140.m4a
[download] 100% of 6.20MiB in 00:00
[ffmpeg] Merging formats into "HANABI  - les règles du jeu !-IX6McAMbMIU.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:HANABI  - les règles du jeu !-IX6McAMbMIU.f137.mp4' -i 'file:HANABI  - les règles du jeu !-IX6McAMbMIU.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:HANABI  - les règles du jeu !-IX6McAMbMIU.temp.mp4'
Deleting original file HANABI  - les règles du jeu !-IX6McAMbMIU.f137.mp4 (pass -k to keep)
Deleting original file HANABI  - les règles du jeu !-IX6McAMbMIU.f140.m4a (pass -k to keep)
ghost commented 1 year ago

thanks but I don't actually use YouTube-DL, I use my own tool. I was just trying to help troubleshoot the potential geo block issue.

daald commented 1 year ago

maybe renaming this ticket to "WARNING: unable to extract player URL -> ERROR: No video formats found" would result in less duplicates

dirkf commented 1 year ago

Or even, "fewer duplicates".

The error message depends on how exactly Google wants to screw with you but now that you've mentioned this version people who follow search suggestions in #30839 will find this issue as the second hit.

daald commented 1 year ago

Yes, sure, I visited this page on 4 different days, because it was the first search result each time, and each time I thought "no, it's not this, they still didn't report the newest bug". then I left and tried a few days later again. it's about psychology.

And I'm still not sure if the reported issue is was I see at home. But if the newly marked duplicates really match, especially https://github.com/ytdl-org/youtube-dl/issues/32546, then it must be this one. But I never saw the error message from the title, not in 2023.09.07 and not in 2021.12.17,

rexxxx1948 commented 1 year ago

I have the same Problem with Playlists. I'm here in Germany. I found out, that It will restart Downloading for a few Videos when I update the Cookie file and then I get the Error again. Hope that will help.

This Patch has fixed the problem in my case. https://github.com/yt-dlp/yt-dlp/commit/378ae9f9fb8e8c86e6ac89c4c5b815b48ce93620

damgut commented 1 year ago

I'm in Germany and the patch also worked for me:

mshkrebtan commented 1 year ago

If you use Firefox and don't want to install extensions due to the permissions they require, I have found a very nice shell script that allows you to export the cookies from the sqlite database: https://kb.mozillazine.org/Cookies.sqlite.

It needs a little correction, though: removing the indentation in the heredoc block for the sqlite3 command.

dirkf commented 1 year ago

Just make the leading spaces tabs (this isn't clear in the forum posting). The <<- POSIX heredoc syntax (vs <<) means "strip leading tabs".

arachnosoft commented 1 year ago

I'm running youtube-dl on a Raspberry Pi. So, using any of the solutions given here did not work for me.

But I ultimately succeeded in extracting a cookies.txt file, perfectly working with version 2021.12.17, from another computer, using a standard Chrome browser on a standard Windows version with no additional extensions or whatsoever, just by using Google Chrome Developer Tools (F12) and pasting a script on the Console, to get the cookies in Netscape format.

See https://apple.stackexchange.com/a/425700

Go to YouTube with Chrome and connect to your account the usual way. Open the DevTools using F12 Go to the Applications tab, scroll down to the Cookies section, uncheck all the HttpOnly checkboxes to ensure that all cookies will be extracted Run the script given in the above-linked answer into Chrome's console. Copy and paste the output in a UTF-8 .txt file Put this file in your youtube-dl environment Add --cookies yourcookiefile.txt to your youtube-dl command line And voilà!

Kudos to Maksim for this neat, straightforward working solution! In case the link would break, I attached the script here, with credits, of course. Cookie Extraction Google Chrome Script.txt

daald commented 1 year ago

one side-note: use the --cookies parameter in front of the url, and not after. my first tries failed because of this.

dirkf commented 1 year ago

That's surprising.

It could be logical for options to be parsed as either global or per-URL, so that the options in effect for a given URL would be the final or cumulative values for any global options plus the most recent or cumulative values, up to the position of the current URL.

As far as I understand, this does not happen, Instead all options are parsed together, with their latest or cumulative values applied to all URLs.

Possibly your observation is a side-effect of #32450, though I can't immediately see how.

daald commented 1 year ago

sorry, I can't reproduce it myself today (2023.09.07 and 2021.12.17), so just forget about it. as far I remember, I had played around with user-agent and cookies and it simply never worked until I moved all these arguments to the front. and I remember one day where I had a -x afterwards and it downloaded a video instead of music. i know, it's not how parameters are supposed to work, that's why I wrote it.

ReenigneArcher commented 1 year ago

I have the same Problem with Playlists. I'm here in Germany. I found out, that It will restart Downloading for a few Videos when I update the Cookie file and then I get the Error again. Hope that will help.

This Patch has fixed the problem in my case. yt-dlp/yt-dlp@378ae9f

Is there a PR up for this?

Also, this affects individual videos, not only playlists. I think the title of this issue should be edited to unable to extract player URL

dirkf commented 1 year ago

Is there a PR up for this?

Any minute now.

a-pav commented 12 months ago

RE Cookie extraction. Please don't use any utility or install shady browser extensions just to extract the cookies, that's ridiculous!

Run 

(() => {
    // Paste value of cookie's header obtained from network tab between the quotes,
    // or leave it as empty string.
    let COOKIE = '' || document.cookie;

    copy(
        '# Netscape HTTP Cookie File\n' +
        COOKIE.split(/; /g)
            .map(e => e.replace('=', '\t'))
            .map(e => window.location.hostname.replace('www.', '') + '\tFALSE\t/\tTRUE\t0\t' + e)
            .join('\n')
    )
})()

inside your browser console and the Netscape-formatted cookies will be copied to your clipboard. Paste inside a cookie.txt file and pass it to youtube-dl like --cookies ./cookie.txt.

Note: document.cookie seems to have all the necessary cookie values which are needed for a successful youtube-dl operation. But if that ever failed, try copying Cookie: header's value from your browser's network tab and paste it between the quotes indicated in the snippet, and try again. So far I haven't had to do that in my usage.

@dirkf Please promote this method anywhere you can. It's by far the safest and easiest way to get the cookies.

dirkf commented 12 months ago

We previously identified an apparently reliable browser extension that generates the necessary cookie file, and that's what the online documentation recommends.

a-pav commented 12 months ago

I understand that for the online documentation you probably want to stick with recommending something that just works.

But since most people don't know how to extract their cookies, youtube-dl documentation currently makes it look like that an extra, special tool is needed to do that.

Perhaps if youtube-dl had a --cookies-raw FILE option to pass raw key=value; key=value; <...> cookie pairs, then obtaining cookies would be a matter of copying it from browser's network tab. No special tool needed.

Now I wonder if --add-header FIELD:VALUE option could be used to pass cookies like: --add-header 'Cookie: key=value; <...>'?

dirkf commented 12 months ago

It could, but see the pinned security issue #32450 for why --cookies ... is preferred. (tldr; cookies need to be associated with specific domains, and the cookie header does not contain that information: the cookie file is the pig while the header is a sausage).

dirkf commented 11 months ago

I'll keep this open until there's a fixed release.

janacm commented 10 months ago

RE Cookie extraction. Please don't use any utility or install shady browser extensions just to extract the cookies, that's ridiculous!

Run

(() => {
    // Paste value of cookie's header obtained from network tab between the quotes,
    // or leave it as empty string.
    let COOKIE = '' || document.cookie;

    copy(
        '# Netscape HTTP Cookie File\n' +
        COOKIE.split(/; /g)
            .map(e => e.replace('=', '\t'))
            .map(e => window.location.hostname.replace('www.', '') + '\tFALSE\t/\tTRUE\t0\t' + e)
            .join('\n')
    )
})()

inside your browser console and the Netscape-formatted cookies will be copied to your clipboard. Paste inside a cookie.txt file and pass it to youtube-dl like --cookies ./cookie.txt.

Note: document.cookie seems to have all the necessary cookie values which are needed for a successful youtube-dl operation. But if that ever failed, try copying Cookie: header's value from your browser's network tab and paste it between the quotes indicated in the snippet, and try again. So far I haven't had to do that in my usage.

@dirkf Please promote this method anywhere you can. It's by far the safest and easiest way to get the cookies.

Hey I get undefined when running this

dirkf commented 10 months ago

Perhaps the poster has this function from SO. Anyhow, s/copy/return/, but generally https://github.com/ytdl-org/youtube-dl/issues/32499#issuecomment-1817481094 applies.

a-pav commented 10 months ago

I get undefined when running this

undefined what exactly? You should mention this, since virtually everything is undefined inside a browser console except the things that are not. I just ran the snippet in Firefox console with success. There should be no problem in a chrome console as well. Make sure to use the snippet in its entirety. If copy function is undefined, try writing the resultant string somewhere in DOM (within an element) and copy it manually from there. clipboard.writeText might be another option, but seems unnecessary.

PerryRylance commented 10 months ago

Using yt-dlp and a VPN in the US worked for me.

DawidKrysiak commented 9 months ago

I came accross the issue with consent yesterday.

For people using qutebrowser, you first need to convert its cookies into netscape format.

I used https://gist.github.com/guidocella/a272b6e68f9c44532b011f6596e95c61#file-dump-cookies-sh to do so and then youtube-dl --cookies $XDG_RUNTIME_DIR/cookies.txt

Just posting this in case it saves some people some trouble.

For any new user googling for that. THAT is the correct interim solution. Let's be honest, google does what they can to block downloading. So the solution is: use a plugin (e.g. 'get cookies.txt clean') downdload your youtube cookies, then

youtube-dl --cookies all_cookies.txt http://youtube.com/<channel>

dirkf commented 9 months ago

The consent issue is fixed in the nightly build which you should be using.

8tm commented 8 months ago

RE Cookie extraction. Please don't use any utility or install shady browser extensions just to extract the cookies, that's ridiculous!

Run

(() => {
    // Paste value of cookie's header obtained from network tab between the quotes,
    // or leave it as empty string.
    let COOKIE = '' || document.cookie;

    copy(
        '# Netscape HTTP Cookie File\n' +
        COOKIE.split(/; /g)
            .map(e => e.replace('=', '\t'))
            .map(e => window.location.hostname.replace('www.', '') + '\tFALSE\t/\tTRUE\t0\t' + e)
            .join('\n')
    )
})()

inside your browser console and the Netscape-formatted cookies will be copied to your clipboard. Paste inside a cookie.txt file and pass it to youtube-dl like --cookies ./cookie.txt.

Note: document.cookie seems to have all the necessary cookie values which are needed for a successful youtube-dl operation. But if that ever failed, try copying Cookie: header's value from your browser's network tab and paste it between the quotes indicated in the snippet, and try again. So far I haven't had to do that in my usage.

@dirkf Please promote this method anywhere you can. It's by far the safest and easiest way to get the cookies.

Where this code should be added?

ReenigneArcher commented 8 months ago

Where this code should be added?

In the console of your web browser.