search

blackjack4494 / yt-dlc

media downloader and library for various sites.
The Unlicense
2.89k stars 364 forks source link

[Broken] blocked it from display on this website or application #91

Open g1ra opened 3 years ago

g1ra commented 3 years ago

Checklist

Verbose log

youtube-dlc -v https://www.youtube.com/watch?v=fh2cDKyFdyU
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=fh2cDKyFdyU']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.31
[debug] Python version 2.7.18 (CPython) - Linux-5.4.0-52-generic-x86_64-with-Ubuntu-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[youtube] fh2cDKyFdyU: Downloading webpage
[youtube] fh2cDKyFdyU: Downloading embed webpage
[youtube] fh2cDKyFdyU: Refetching age-gated info webpage
ERROR: This video contains content from Mosfilm, who has blocked it from display on this website or application. Watch on YouTube
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dlc/extractor/youtube.py", line 2004, in _real_extract
    yt_initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
KeyError: u'conversationBar'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dlc/YoutubeDL.py", line 830, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dlc/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dlc/extractor/youtube.py", line 2235, in _real_extract
    raise ExtractorError(error_message, expected=True)
ExtractorError: This video contains content from Mosfilm, who has blocked it from display on this website or application. Watch on YouTube

Description

I believe this is broken , because I can play on youtube but not by application .... Please check it .

blackjack4494 commented 3 years ago

Can you try with cookies?

nicolaasjan commented 3 years ago

Doesn't work here either, even with a fresh cookie.txt file...

Vangelis66 commented 3 years ago

@g1ra, @blackjack4494, @nicolaasjan:

The solution is to also use the --user-agent switch and provide as value EXACTLY the useragent string of the browser where both 1) the login was performed (to verify age) and 2) the cookies.txt file was exported from (probably via a dedicated extension):

youtube-dl -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "YTcookies_20201101.txt" --user-agent "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" => 

[youtube] fh2cDKyFdyU: Downloading webpage
[youtube] fh2cDKyFdyU: Downloading js player ec262be6
[youtube] fh2cDKyFdyU: Downloading js player ec262be6
[info] Available formats for fh2cDKyFdyU:
format code  extension  resolution note
249          webm       audio only tiny   66k , opus @ 50k (48000Hz), 60.06MiB
250          webm       audio only tiny   84k , opus @ 70k (48000Hz), 77.05MiB
140          m4a        audio only tiny  137k , m4a_dash container, mp4a.40.2@128k (44100Hz), 151.33MiB
251          webm       audio only tiny  156k , opus @160k (48000Hz), 147.79MiB
278          webm       256x144    144p  103k , webm container, vp9, 25fps, video only, 103.07MiB
160          mp4        256x144    144p  117k , avc1.4d400c, 25fps, video only,68.74MiB
242          webm       426x240    240p  232k , vp9, 25fps, video only, 168.45MiB
133          mp4        426x240    240p  251k , avc1.4d4015, 25fps, video only,123.50MiB
243          webm       640x360    360p  467k , vp9, 25fps, video only, 298.67MiB
134          mp4        640x360    360p  638k , avc1.4d401e, 25fps, video only,293.37MiB
18           mp4        640x360    360p  466k , avc1.42001E, 25fps, mp4a.40.2@ 96k (44100Hz), 555.04MiB (best)

If you're using a Chromium derivative as the browser, then you probably need the cookies.txt v1.14.1 extension, but evil Google :smiling_imp: , current owners of Youtube, have recently obliterated it from the Chrome Web Store (CWS):

https://chrome.google.com/webstore/detail/njabckikapfpffapmjgojcnbfjonfjfg

If not installed already (I have it installed since long ago...), search for it elsewhere... FWIW, the best solution would be one where a Google account/mandatory browser login aren't required, but here's hopin'...

Best regards :smile:

nicolaasjan commented 3 years ago

Thanks, But that unfortunately didn't work...

I tried with both the cookies.txt extension and with another one, which is still available (Get cookies.txt). I have a zip-file of the cookies.txt extension here.

youtube-dlc -v https://www.youtube.com/watch?v=fh2cDKyFdyU
[debug] System config: []
[debug] User config: ['-i', '-o', '/dev/shm/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--add-metadata', '--embed-thumbnail', '--cookies', '/home/nico/.config/youtube-dlc/yt-cookies.txt', '--user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4301.0 Safari/537.36']
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=fh2cDKyFdyU']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.31
[debug] Python version 3.6.9 (CPython) - Linux-4.15.0-122-generic-x86_64-with-LinuxMint-19.3-tricia
[debug] exe versions: ffmpeg N-99430-g9d8f9b2-Nico_20201001, ffprobe N-99430-g9d8f9b2-Nico_20201001
[debug] Proxy map: {}
[youtube] fh2cDKyFdyU: Downloading webpage
[youtube] fh2cDKyFdyU: Downloading embed webpage
[youtube] fh2cDKyFdyU: Refetching age-gated info webpage
ERROR: This video contains content from Mosfilm, who has blocked it from display on this website or application. Watch on YouTube
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dlc/youtube_dlc/YoutubeDL.py", line 830, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dlc/youtube_dlc/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dlc/youtube_dlc/extractor/youtube.py", line 2214, in _real_extract
    raise ExtractorError(error_message, expected=True)
youtube_dlc.utils.ExtractorError: This video contains content from Mosfilm, who has blocked it from display on this website or application. Watch on YouTube
Vangelis66 commented 3 years ago

@nicolaasjan wrote:

Another video from "Mosfilm": Same error...

youtube-dlc -F https://www.youtube.com/watch?v=fh2cDKyFdyU

... Actually, it's the same video all the time, as referenced in OP's verbose log, and the same I conducted (successful) tests with...

My particulars:

  1. I am on Windows Vista SP2 32-bit
  2. Using latest yt-dlc release (2020.10.31), specifically the 32-bit compiled executable, youtube-dlc_x86.exe, simply renamed to youtube-dlc.exe (embeds Python 3.5.4 x86, won't run under Windows XP SP3 x86).
  3. Browser used is 360 Extreme Explorer v12 (Chromium 78 fork, but XP/Vista compatible)
  4. Extension used cookies.txt v1.14.1
  5. Youtube log-in was performed mere minutes ago (BTW, you also have to click on the video player itself to "accept" and initiate playback, before you extract cookies!).
  6. A fresh cookies.txt file was exported (for ONLY the YT tab), renamed for sanity to YTcookies_20201102.txt and placed adjacent to youtube-dlc.exe
  7. The following command worked:

youtube-dlc -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "YTcookies_20201102.txt" --user-agent "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" -v => 

[debug] Loading archive file None
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dlc version 2020.10.31
[debug] Python version 3.5.4 (CPython) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] fh2cDKyFdyU: Downloading webpage
[youtube] {18} signature length 109, html5 player ec262be6
[youtube] {134} signature length 105, html5 player ec262be6
[youtube] {243} signature length 105, html5 player ec262be6
[youtube] {133} signature length 109, html5 player ec262be6
[youtube] {242} signature length 109, html5 player ec262be6
[youtube] {160} signature length 109, html5 player ec262be6
[youtube] {278} signature length 109, html5 player ec262be6
[youtube] {140} signature length 109, html5 player ec262be6
[youtube] {249} signature length 109, html5 player ec262be6
[youtube] {250} signature length 109, html5 player ec262be6
[youtube] {251} signature length 109, html5 player ec262be6
[info] Available formats for fh2cDKyFdyU:
format code  extension  resolution note
249          webm       audio only tiny   66k , opus @ 50k (48000Hz), 60.06MiB
250          webm       audio only tiny   84k , opus @ 70k (48000Hz), 77.05MiB
140          m4a        audio only tiny  137k , m4a_dash container, mp4a.40.2@128k (44100Hz), 151.33MiB
251          webm       audio only tiny  156k , opus @160k (48000Hz), 147.79MiB
278          webm       256x144    144p  103k , webm container, vp9, 25fps, video only, 103.07MiB
160          mp4        256x144    144p  117k , avc1.4d400c, 25fps, video only,68.74MiB
242          webm       426x240    240p  232k , vp9, 25fps, video only, 168.45MiB
133          mp4        426x240    240p  251k , avc1.4d4015, 25fps, video only,123.50MiB
243          webm       640x360    360p  467k , vp9, 25fps, video only, 298.67MiB
134          mp4        640x360    360p  638k , avc1.4d401e, 25fps, video only,293.37MiB
18           mp4        640x360    360p  466k , avc1.42001E, 25fps, mp4a.40.2@ 96k (44100Hz), 555.04MiB (best)

FWIW, latest original youtube-dl (recently released v2020.11.1.1, safely moved into a private repo, off of GitHub), also works under the above scenario...

Since I know zilch about Linux, am afraid I can't assist you further... WFM it is here!

nicolaasjan commented 3 years ago

... Actually, it's the same video all the time, as referenced in OP's verbose log, and the same I conducted (successful) tests with...

Sorry... My mistake. Deleted the post.

nicolaasjan commented 3 years ago

My particulars:

1. I am on **Windows Vista SP2 32-bit**

2. Using latest **yt-dlc** release (**2020.10.31**), specifically the **32-bit** compiled executable,  **youtube-dlc_x86.exe**, simply renamed to youtube-dlc.exe (embeds **Python 3.5.4** x86, **won't** run under Windows XP SP3 x86).

3. Browser used is **360 Extreme Explorer v12** (Chromium 78 fork, but XP/Vista compatible)

4. Extension used **cookies.txt v1.14.1**

5. Youtube log-in was performed **mere minutes ago** (BTW, you also have to click on the video player itself to "accept" and initiate playback, **before** you extract cookies!).

6. A **fresh** cookies.txt file was exported (for ONLY the YT tab), renamed for sanity to **YTcookies_20201102.txt** and placed **adjacent** to youtube-dlc.exe

7. The following command worked:

youtube-dlc -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "YTcookies_20201102.txt" --user-agent "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" -v =>

Followed your instructions, but no luck (same error)... Youtube-dlc compiled from the master branch today (but the release version doesn't work either).

youtube-dlc --ignore-config -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies /home/nico/.config/youtube-dlc/yt-cookies.txt --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0" -v

  1. Youtube log-in was performed mere minutes ago (BTW, you also have to click on the video player itself to "accept" and initiate playback, before you extract cookies!).

That's what I did in my previous attempt as well. yt-cookies.txt placed adjacent to youtube-dlc made no difference. Something else must be the culprit. I think I just give up...

Vangelis66 commented 3 years ago

Something else must be the culprit. I think I just give up...

Try this: After you have logged-in to YT and initiated the age-gated video playback, reload the yt tab and THEN export cookies for that tab; FWIW, my cookies.txt file looks like this (tab-delimited, crucial cookie values redacted): 

# HTTP Cookie File downloaded with cookies.txt by Genuinous @genuinous
# This file can be used by wget, curl, aria2c and other standard compliant tools.
# Usage Examples:
#   1) wget -x --load-cookies cookies.txt "https://www.youtube.com/watch?v=fh2cDKyFdyU&amp;has_verified=1"
#   2) curl --cookie cookies.txt "https://www.youtube.com/watch?v=fh2cDKyFdyU&amp;has_verified=1"
#   3) aria2c --load-cookies cookies.txt "https://www.youtube.com/watch?v=fh2cDKyFdyU&amp;has_verified=1"
#
.youtube.com    TRUE    /   TRUE    1613759850  VISITOR_INFO1_LIVE  <removed>
www.youtube.com FALSE   /   TRUE    0   SL_GWPT_Show_Hide_tmp   1
www.youtube.com FALSE   /   TRUE    0   SL_wptGlobTipTmp    1
m.youtube.com   FALSE   /   TRUE    0   SL_GWPT_Show_Hide_tmp   1
m.youtube.com   FALSE   /   TRUE    0   SL_wptGlobTipTmp    1
.youtube.com    TRUE    /   TRUE    2146723199  CONSENT <removed>
.youtube.com    TRUE    /   FALSE   1667406285  PREF    <removed>
.youtube.com    TRUE    /   FALSE   1604335820  GPS 1
.youtube.com    TRUE    /   TRUE    0   YSC <removed>
.youtube.com    TRUE    /   FALSE   1667406132  SID <removed>
.youtube.com    TRUE    /   TRUE    1667406132  __Secure-3PSID  <removed>
.youtube.com    TRUE    /   FALSE   1667406132  HSID    <removed>
.youtube.com    TRUE    /   TRUE    1667406132  SSID    <removed>
.youtube.com    TRUE    /   FALSE   1667406132  APISID  <removed>
.youtube.com    TRUE    /   TRUE    1667406132  SAPISID <removed>
.youtube.com    TRUE    /   TRUE    1667406132  __Secure-3PAPISID   <removed>
.youtube.com    TRUE    /   TRUE    1667406146  LOGIN_INFO  <removed>
.youtube.com    TRUE    /   FALSE   1635870288  SIDCC   <removed>
.youtube.com    TRUE    /   TRUE    1635870288  __Secure-3PSIDCC    <removed>

Also, do not log-out from youtube in the browser before you launch the yt-dlc command...

Lastly, if at all possible, try the procedure on a Windows box...

blackjack4494 commented 3 years ago

@Vangelis66 have you tried different user agents so far?

nicolaasjan commented 3 years ago

Lastly, if at all possible, try the procedure on a Windows box...

Nope... (both Windows and Linux version compiled today from the master branch)

Tried with a new Firefox profile in a Windows 7 VM, with only the cookies.txtextension and my HOSTS- file disabled...:

C:\Users\Nico>youtube-dlc --ignore-config -F "https://www.youtube.com/watch?v=fh
2cDKyFdyU" --cookies "C:\Users\Nico\AppData\Roaming\youtube-dlc\cookies.txt" --u
ser-agent "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:82.0) Gecko/20100101 Fire
fox/82.0" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-F', 'https://www.youtube.com/wa
tch?v=fh2cDKyFdyU', '--cookies', 'C:\\Users\\Nico\\AppData\\Roaming\\youtube-dlc
\\cookies.txt', '--user-agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:82.
0) Gecko/20100101 Firefox/82.0', '-v']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.03
[debug] Python version 3.8.6 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 2020-10-07-git-a086b73e1f-essentials_build-www.gyan
.dev, ffprobe 2020-10-07-git-a086b73e1f-essentials_build-www.gyan.dev
[debug] Proxy map: {}
[youtube] fh2cDKyFdyU: Downloading webpage
[youtube] fh2cDKyFdyU: Downloading embed webpage
[youtube] fh2cDKyFdyU: Refetching age-gated info webpage
ERROR: This video contains content from Mosfilm, who has blocked it from display
 on this website or application. Watch on YouTube
Traceback (most recent call last):
  File "youtube_dlc\YoutubeDL.py", line 830, in extract_info
  File "youtube_dlc\extractor\common.py", line 532, in extract
  File "youtube_dlc\extractor\youtube.py", line 2252, in _real_extract
youtube_dlc.utils.ExtractorError: This video contains content from Mosfilm, who
has blocked it from display on this website or application. Watch on YouTube

Try this: After you have logged-in to YT and initiated the age-gated video playback, reload the yt tab and THEN export cookies for that tab; FWIW, my cookies.txt file looks like this (tab-delimited, crucial cookie values redacted):

On Linux I tried with Chromium and Firefox:

youtube-dlc --ignore-config -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "/home/nico/.config/youtube-dlc/yt-cookies.txt" --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4314.0 Safari/537.36" -v

youtube-dlc --ignore-config -F "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "/home/nico/.config/youtube-dlc/yt-cookies.txt" --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0" -v

# Netscape HTTP Cookie File
# This file is generated by youtube-dlc.  Do not edit.

.youtube.com    TRUE    /   FALSE   1667468233  APISID  <removed>
.youtube.com    TRUE    /   TRUE    2146723200  CONSENT <removed>
.youtube.com    TRUE    /   FALSE   1604398015  GPS <removed>
.youtube.com    TRUE    /   FALSE   1667468233  HSID    <removed>
.youtube.com    TRUE    /   TRUE    1667468233  LOGIN_INFO <removed>
.youtube.com    TRUE    /   FALSE   1609581196  PREF    <removed>
.youtube.com    TRUE    /   TRUE    1667468233  SAPISID <removed>
.youtube.com    TRUE    /   FALSE   1667468233  SID <removed>
.youtube.com    TRUE    /   FALSE   1635933197  SIDCC   <removed>
.youtube.com    TRUE    /   TRUE    1667468233  SSID    <removed>
.youtube.com    TRUE    /   TRUE    1619856492  VISITOR_INFO1_LIVE  <removed>
.youtube.com    TRUE    /   TRUE    0   YSC <removed>
.youtube.com    TRUE    /   TRUE    1667468233  __Secure-3PAPISID   <removed>
.youtube.com    TRUE    /   TRUE    1667468233  __Secure-3PSID  <removed>
.youtube.com    TRUE    /   TRUE    1635933197  __Secure-3PSIDCC    <removed>
.youtube.com    TRUE    /   FALSE   0   s_gl    <removed>
Vangelis66 commented 3 years ago

@blackjack4494 asked:

@Vangelis66 have you tried different user agents so far?

Hello, thanks for your efforts! 👍 As (I hope) I have explained, it is not a matter of trying various useragent strings; instead, it is an attempt to emulate (in the youtube-dlc command) as closely as possible a youtube sanctioned browser, along with a working yt account (one that can successfully bypass the age restriction and initiate video playback in said browser); if, in that context, you're now asking if I have tried the "procedure" on a different browser (other than 360EE v12), please read below...

Being on Windows Vista SP2 32-bit, I don't have a wide selection of current/updated browsers to choose from 😢 ; @nicolaasjan, if you're the same person as the MSFN forums member with the same username, then you should be familiar with roytam1's Serpent 52.9.0 browser (a fork of Basilisk/UXP browser, itself a fork of Firefox ESR 52).

Since I also use Serpent in my Vista laptop, I tried the "procedure" with it; unlike official Basilisk, Serpent has retained some rudimentary Web Extensions support; as such, the cookie-extracting add-on I use there is ganbo; of the versions listed on AMO, 0.2 is the one compatible with Serpent 52; additional modification is needed to convince it to install though, because it's an id-less WE and Serpent doesn't support these; the modified & compatible XPI can be accessed here 😄 .

Though using a fresh browser profile would be ideal, I made sure in my "dirty" profile that I first disabled all youtube-related extensions+userscripts (I had several installed) prior to loading the offending URI:

yt0

Now, Serpent comes with a pre-installed youtube SSUAO (site-specific-UA-override) inside its about:config settings,

general.useragent.override.youtube.com;Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0 Basilisk/52.9.0

the value of which should be used with the --user-agent switch in the yt-dlc command later on...

Below is the page after a successful yt sign-in:

yt1

Then, after I click on the "I understand and wish to continue (rough translation from Greek)" embedded player button, playback (usually) starts:

yt3

While the video still plays (and being signed-in), I click on the ganbo toolbar button, which displays all the yt related cookies (values erased in the screengrab):

yt4

Then, I quickly export (download) "Cookies for this tab" to a file, renamed to YTcookies_Serpent_20201103-2.txt and placed adjacent to youtube-dlc.exe; it is imperative for one to act swiftly issuing the following command:

youtube-dlc -f best "https://www.youtube.com/watch?v=fh2cDKyFdyU" --cookies "YTcookies_Serpent_20201103-2.txt" --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0 Basilisk/52.9.0" -v

which WORKED for me:

yt-dl

For proof, I let the cmd window open to download completion:

yt-dl2

I must again stress that the extracted cookies.txt file gets invalidated pretty quickly; to repeat a download attempt, one has to

  1. Sign-out of the yt account,
  2. Delete all youtube related cookies in the browser,
  3. Delete browser cache (and, for good measure, perhaps even restart the browser),
  4. Repeat the "procedure", to generate a fresh cookies.txt file...

Other than the above details, I remain completely clueless as to the reason(s) things aren't working for the rest of the people on this thread; I feel gifted that at least two browsers on my system are "suitable" for the task at hand, and I won't be losing any more time on doing further tests; I wish the best to all of you trying to decipher the issue on your setups...

Thanks for this nice yt-dl fork, best wishes and stay safe (a month-long lockdown imposed here as of yesterday 😞 ) ...

nicolaasjan commented 3 years ago

if you're the same person as the MSFN forums member with the same username

Yes, that's me. ;)

Tried again in a Windows XP virtual machine with Basilisk and your extension, but again no luck... Could download though, via the URL's shown via F12 (delete the &range=xxxxx part). See my explanation here.

screenshot_video-download

screenshot_vlc