Youtube - Random "Unable to extract JS player URL"

triplesixman commented 3 years ago

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.09.20
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

(see my second post to find the result of the --dump-pages without using a proxy)

root@root:~/test# youtube-dl https://youtu.be/w5LZ8YErl2M --proxy https://***/ --verbose; \
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/w5LZ8YErl2M', u'--proxy', u'https://***/', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2020.09.20
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-11-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {u'http': u'https://***/', u'https': u'https://***/'}
[youtube] w5LZ8YErl2M: Downloading webpage
[youtube] w5LZ8YErl2M: Downloading embed webpage
ERROR: Unable to extract JS player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 2101, in _real_extract
    ASSETS_RE, embed_webpage, 'JS player URL')
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 1010, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract JS player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []```

Description

Since a few days, I randomly get this error when I download youtube videos (maybe 1 chance out of 15 or 20 to get the error). The error appears on random videos, with or without proxy (for this example, it's with proxy).

Immediately after the error, I just returned the command line and the download went through:

root@root:~/test# youtube-dl https://youtu.be/w5LZ8YErl2M --proxy https://***/ --verbose; \
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/w5LZ8YErl2M', u'--proxy', u'https://***/', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2020.09.20
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-11-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {u'http': u'https://***/', u'https': u'https://***/'}
[youtube] w5LZ8YErl2M: Downloading webpage
[youtube] {18} signature length 111, html5 player 4b1ba5ea
[youtube] {137} signature length 111, html5 player 4b1ba5ea
[youtube] {248} signature length 107, html5 player 4b1ba5ea
[youtube] {399} signature length 111, html5 player 4b1ba5ea
[youtube] {136} signature length 111, html5 player 4b1ba5ea
[youtube] {247} signature length 111, html5 player 4b1ba5ea
[youtube] {398} signature length 111, html5 player 4b1ba5ea
[youtube] {135} signature length 111, html5 player 4b1ba5ea
[youtube] {244} signature length 107, html5 player 4b1ba5ea
[youtube] {397} signature length 107, html5 player 4b1ba5ea
[youtube] {134} signature length 111, html5 player 4b1ba5ea
[youtube] {243} signature length 111, html5 player 4b1ba5ea
[youtube] {396} signature length 107, html5 player 4b1ba5ea
[youtube] {133} signature length 111, html5 player 4b1ba5ea
[youtube] {242} signature length 111, html5 player 4b1ba5ea
[youtube] {395} signature length 111, html5 player 4b1ba5ea
[youtube] {160} signature length 107, html5 player 4b1ba5ea
[youtube] {278} signature length 111, html5 player 4b1ba5ea
[youtube] {394} signature length 111, html5 player 4b1ba5ea
[youtube] {140} signature length 111, html5 player 4b1ba5ea
[youtube] {249} signature length 111, html5 player 4b1ba5ea
[youtube] {250} signature length 111, html5 player 4b1ba5ea
[youtube] {251} signature length 111, html5 player 4b1ba5ea
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r4---sn-n4v7sn7z.googlevideo.com/videoplayback?expire=1600700607&ei=XmxoX5eSOI-RkgbA_ZrIDQ&ip=***&id=o-AJp4PAhbd0fVj7naGu7m-4aIai_FLd_2AVsgdfBJc75V&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=1I&mm=31%2C29&mn=sn-n4v7sn7z%2Csn-n4v7knlz&ms=au%2Crdu&mv=m&mvi=4&pl=24&initcwndbps=7731250&vprv=1&mime=video%2Fmp4&gir=yes&clen=31495116&dur=250.750&lmt=1577917632979239&mt=1600678842&fvip=4&keepalive=yes&fexp=23915655&beids=23886205&c=WEB&txp=5535432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAP7eqaI4VVgAx0O-X0lJKGkOPQ4PaSIG-BBrpwDmn4TDAiAYxptBdHeFtIiRpMcPwvAbwIwREqutdxaiTh6sDNj2yg%3D%3D&sig=AOq0QJ8wRQIgRWWIopKlKMyKkGWdplBBe1OHpX-YAiRNczIzQMc_GOgCIQDfMFEjtPr4N9waa1dSyaM8AMQ65t8M1BiVkCgM1DQAnw==&ratebypass=yes'
[download] Destination: Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f137.mp4
[download] 100% of 30.04MiB in 00:03
[debug] Invoking downloader on u'https://r4---sn-n4v7sn7z.googlevideo.com/videoplayback?expire=1600700607&ei=XmxoX5eSOI-RkgbA_ZrIDQ&ip=***&id=o-AJp4PAhbd0fVj7naGu7m-4aIai_FLd_2AVsgdfBJc75V&itag=251&source=youtube&requiressl=yes&mh=1I&mm=31%2C29&mn=sn-n4v7sn7z%2Csn-n4v7knlz&ms=au%2Crdu&mv=m&mvi=4&pl=24&initcwndbps=7731250&vprv=1&mime=audio%2Fwebm&gir=yes&clen=3872013&dur=250.781&lmt=1577917415398203&mt=1600678842&fvip=4&keepalive=yes&fexp=23915655&beids=23886205&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgHdYrubvzw0uS33WqI-kK3v-No3G-CPcdYh9dNPxfMkICIQCpe0mhl25flnAbrejjurNkmhV8VWnFJKKzT4_oxRRqIA%3D%3D&sig=AOq0QJ8wRQIgJRp_KjGW3oTCCS6lCRz2iY_D51ICUws_ZUwuTNk7kT0CIQDK6hTyxgzbDHA7ShNQggn8JrgxPV7yNBZuGtNztuMwKg==&ratebypass=yes'
[download] Destination: Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f251.webm
[download] 100% of 3.69MiB in 00:00
[ffmpeg] Merging formats into "Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f137.mp4' -i 'file:Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f251.webm' -c copy -map '0:v:0' -map '1:a:0' 'file:Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.temp.mkv'
Deleting original file Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f137.mp4 (pass -k to keep)
Deleting original file Marilyn_Manson_-_Deep_Six_Explicit_Official_Music_Video-w5LZ8YErl2M.f251.webm (pass -k to keep)

The problem may be difficult to reproduce, so don't hesitate to ask me for details if needed, I will do my best to pass them on to you.

triplesixman commented 3 years ago

The result with a "--dump-pages" that I got without using a proxy. log18.txt

triplesixman commented 3 years ago

I also regularly have "This video is DRM protected" and "Unable to extract video data" but randomly too. I think it may be related to the problem mentioned.

wernerdweight commented 3 years ago

I guess this might be caused by the fact that, sometimes, youtube page doesn't contain the player_response key, but rather raw_player_response like this: ytplayer.config={args:{raw_player_response:ytInitialPlayerResponse}. The variable (ytInitialPlayerResponse) then contains the player config JSON.

wernerdweight commented 3 years ago

Btw. the issue I described is fixed already.

ytdl-org / youtube-dl