search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.41k stars 10.04k forks source link

Vrtnu running with username and password: age restriction not handled correctly #29044

Open tisik opened 3 years ago

tisik commented 3 years ago

Checklist

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://www.vrt.be/vrtnu/a-z/petra/1/petra-s1a2/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.8.5 (CPython) - Linux-5.8.0-53-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[VrtNU] Logging in
[VrtNU] Requesting a token
[VrtNU] petra-s1a2: Downloading webpage
[Canvas] pbs-pub-e3cd125e-b92d-4142-a860-dc819cdd9424$vid-c358d4e5-e805-4db1-8c92-ccc087ead5ca: Downloading token
[Canvas] pbs-pub-e3cd125e-b92d-4142-a860-dc819cdd9424$vid-c358d4e5-e805-4db1-8c92-ccc087ead5ca: Downloading video JSON
ERROR: Age category 0+ is not allowed to access 12+ content
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/canvas.py", line 80, in _real_extract
    raise ExtractorError(data.get('message') or code, expected=True)
youtube_dl.utils.ExtractorError: Age category 0+ is not allowed to access 12+ content

Description

Can not download age restricted video from https://www.vrt.be/vrtnu/ (e.g. https://www.vrt.be/vrtnu/a-z/petra/1/petra-s1a2/), although my account allows playing it. An account can be made from the site: https://www.vrt.be/vrtnu/

Nick4-1 commented 3 years ago

Same problem, even with v2021.05.16

MatthiasCoppens commented 3 years ago

It's got something to do with the vrtPlayerToken. When watching https://www.vrt.be/vrtnu/a-z/bloodlands/1/bloodlands-s1a1/ on Firefox, the vrtPlayerToken is different from youtube-dl's vrtPlayerToken. When resending the GET request in FF with youtube-dl's vrtPlayerToken, it results in the same error. When overwriting the token in youtube-dl with FF's token, it works perfectly.

MatthiasCoppens commented 3 years ago

To get a valid vrtPlayerToken to watch age-restricted videos, an identityToken needs to be passed with the POST request at https://github.com/ytdl-org/youtube-dl/blob/dfbbe2902fc67f0f93ee47a8077c148055c67a9b/youtube_dl/extractor/canvas.py#L65 This identityToken can be found in the X-VRT-Token cookie, however for me this cookie differs between youtube-dl and my browser.

Dre62 commented 3 years ago

Is it possible to replace the incorrect token by the right one in the cmd line request?

sharethewisdom commented 3 years ago

dear @MatthiasCoppens thanks for your clear description of the problem. I don't quite understand how to add an identityToken to the POST request. I naively tried to add a Firefox cookie to test in canvas.py, for example

headers.update({'Content-Type': 'application/json'})
headers.update({'X-VRT-Token': 'eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL3Rva2VuLnZydC5iZSIsInN1YiI6IjYyOWE0ZThjZWQ1MzQxNDdhNTE3ZGI1OGM1YWE3NWZiIiwibmV3c3ViIjoiYjllOWY1ZGEtMDUwYy00NWQ0LWI4NzAtZjE5ZDFhOWZiZWY2IiwiY291IjoiQkUiLCJhZ2UiOjMzLCJhdWQiOiIvdnJ0bnUiLCJpYXQiOjE2MjE3NzUxNzgsImV4cCI6MTYyMTc3ODc3OCwiYWMiOiIxNisifQ.VNZoO0ZwgOezUt60So-j0cDkS8mtcg_2Kxf4ZtCeVCmw-Z5tsQfR5Ji0HVEWhWKz8ejN4B20d5XmD59KZCYkZgdGmqvhwA1_8vedi9xbFakXKDNHOYmqEjBrPOYa3Og5D7VYJHqnas2HXg-MrhPfKu0ECKqO9aS64Df0QMDUxXN9Y1ulIvNfiA0lX3cbSeBMoWZ5HUSKA7POLLWLgmnhiFKic1Lm7w4Pi7RNDw0Ri8oNaWr9ZYtVWKdrz3Y1STZRSsDmemnvFIz1-gLQJek73jBnbei20iG_TNvkuHCfWtiXIKBcBFv_Sih0S426Bsgr4pb4Io4yw8O3dGLwjLa7zfS_BGce8qwTno3cevkg5aeJBQA-8q3Uc9LR2deM0AxjHOfiME6ds9r9zqS7D7DKo2DtcrA-xJmhPs7e7aLkG5h32dEqoSpFkQmU_bvWU43TRzZ7tvNEgHTSV-HaZvCe8y877ckoBkec2rBtIpeeuL-YMsEA68xPUqOaPN8tIS5xy1NUPXcVs7MNaJbXRCIb2peHL15Dl2rQimPTWSsSM4o1HjjXM-f_TDC4jlHsjnQYMB1OBrkYwP6sKBMwwkd-C6lyvAeBwYKngXkP8w6bBpY5-JQhARVN1hKXMNgSBcT1_QmdMB3Jw9qhMDY-uPTfz2FvIbzwGbB_cvAGUEAyNa8'})

but I can't get it to work.

MatthiasCoppens commented 3 years ago

@sharethewisdom: This enables me to download https://www.vrt.be/vrtnu/a-z/before-we-die/2/before-we-die-s2a1/:

headers.update({'Content-Type': 'application/json'})
identityToken = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL3Rva2VuLnZydC5iZSIsInN1YiI6IjY3NmJjMTk1MDAwNzQ1MWFiMTU2NGNjYjI5NTdmZjJjIiwibmV3c3ViIjoiMTQ1ODRjZmUtMjllNC00NzZhLWI1NDEtYjM5MWJhOTc3M2U4IiwiY291IjoiQkUiLCJhZ2UiOjI0LCJhdWQiOiIvdnJ0bnUiLCJpYXQiOjE2MjE3Nzg2MDgsImV4cCI6MTYyMTc4MjIwOCwiYWMiOiIxNisifQ.g8lM8aQX9tVacP9OUz8EAplGac76v5P9wBT8DlXGoDwQvVLncNcafnZiI5OYwVsClcMJBScAsRVKdgSXPXWT9DuPhlfyUs49zpgz-Xb05NHDAIFuFxv0DTNOB9FJEqcMGho4ehl5_524ZoIa9PavlzmPRee-Y9HKw1f-E3o3rkws73GFYAkOhTjXqo55Gg3_HCOqfJEqgKrYzvF-ILMmY9XRqVmn4c1Fw5tTcBspt77h6nKkpb6S231BQELTurJtF7-zwixuKSwZZbdhnruTFUUYlPMK4Y71ZP8PmIapappGXuf7r162FqHEVMgX1UmxWgVdHBPGzaCrNsTZtM3jWzgSlsaopN9u78cKs1N7MKxqXmD2har2a1FJBMDT0XK-oqlMRF8afTnaNEMlHixpz8M-Y4xOcDiUCPmhSCPCiLxJVmcApMM-h_jixwE04eHMYAEndseWp5cZAkz7R3p0oGmrTnYne5xTEJ5-myq3HLtzGfsAkq873e0PepEqv4mDvEEQOUWzmOL44TUQboPv8Ju267Jk3stTBhUcXvKLoFDHmvFJxj_a77QSrnLtaSvLpIqQKjvigC6JejVjwHfe3TQuBDMdeZoH3maW0s9dyM3cK6fMLTI6eS0lqQ3Oct2YM3kvV3XHGO70QD_pLpCY1YbE2YpteWZbfC-2PY-5dP0"
post_data = json.dumps({"identityToken": identityToken}).encode()
token = self._download_json(
    '%s/tokens' % self._REST_API_BASE, video_id,
    'Downloading token',
    data=post_data,
    headers=headers)['vrtPlayerToken']
MatthiasCoppens commented 3 years ago

I can get a working X-VRT-Token with 

curl 'https://token.vrt.be/refreshtoken' -H 'Cookie: vrtlogin-rt=...' -I
sharethewisdom commented 3 years ago

thanks, you're so clever, that works :partying_face: (but now, how to fix canvas.py...)

I initially tried data=urlencode_postdata({ 'identityToken': '...' }) from utils, but got a 400 bad request error. (FYI I get my Netscape HTTP Cookie File with https://github.com/lennonhill/cookies-txt but a curl command is much better, thanks)

MatthiasCoppens commented 3 years ago

When you don't have the vrtlogin-rt cookie, you can get the X-VRT-Token from the regular login sequence of requests: https://github.com/ytdl-org/youtube-dl/blob/dfbbe2902fc67f0f93ee47a8077c148055c67a9b/youtube_dl/extractor/canvas.py#L283

# Sometimes authentication fails for no good reason, retry
login_attempt = 1
while login_attempt <= 3:
    try:
        # This request should set the cookie OIDCXSRF
        self._request_webpage(
            'https://token.vrt.be/vrtnuinitlogin?provider=site&destination=https://www.vrt.be/vrtnu/',
            None,
            note='Initializing login',
            errnote='Could not initialize login',
        )
        # This request should set the cookie X-VRT-Token
        post_data = {
            'UID': auth_info['UID'],
            'UIDSignature': auth_info['UIDSignature'],
            'signatureTimestamp': auth_info['signatureTimestamp'],
            'client_id': 'vrtnu-site',
        }
        for cookie in self._downloader.cookiejar:
            if cookie.name == 'OIDCXSRF':
                post_data['_csrf'] = cookie.value
                break
        self._request_webpage(
            'https://login.vrt.be/perform_login?client_id=vrtnu-site',
            None,
            note='Requesting a token',
            errnote='Could not get a token',
            headers={
                'Content-Type': 'application/x-www-form-urlencoded',
            },
            data=json.dumps(post_data).encode('utf-8'),
        )
    except ExtractorError as e:
        if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:

The second request fails (403: the server understood the request, but is refusing to authorize it). In browser this request sends some extra cookies to the server, not sure where they come from or which ones are important.

chaff02 commented 3 years ago

Hi, I'm also having the problem with the age restriction on vrtnu.be. I tried to fix the canvas.py file with the above code of MatthiasCoppens, but no luck so far. Always have the error: "ERROR: Could not get a token: HTTP Error 403: "

Is there a fix available for the canvas.py file. Many thanks

ghen2 commented 3 years ago

See also Kodi's VRT NU add-on bug https://github.com/add-ons/plugin.video.vrt.nu/issues/882 and fix https://github.com/add-ons/plugin.video.vrt.nu/pull/885; apparantly age restricted content now requires a "user" X-VRT-Token instead of the legacy token (recent change on VRT side).

Matthias' https://github.com/ytdl-org/youtube-dl/issues/29044#issuecomment-846569567 works for me.

chaff02 commented 3 years ago

Works for me too now. Was a little bit of a search for the correct cookie (vrtnu-site_profile_vt). Thanks to ghen2 and Matthias

MatthiasCoppens commented 3 years ago

I'm in the middle of my exams, so I can't really do a lot of troubleshooting right now, but the code at https://github.com/ytdl-org/youtube-dl/issues/29044#issuecomment-847016269 seems very similar to https://github.com/add-ons/plugin.video.vrt.nu/pull/885 so this shouldn't take a lot of tweaking to get it to work.

chaff02 commented 3 years ago

No problem Matthias. Your code at #29044 (comment) works for me. I only have to replace the value of the identityToken by the actual value of the 'vrtnu-site_profile_vt' cookie from www.vrt.be. However this cookie has a short lifetime, as a result many 'make youtube-dl' commands. Success with your exams.

pgaig commented 3 years ago

I wrote a fix #29614 heavily inspired from add-ons/plugin.video.vrt.nu#885 and the comments above.

Seems to work for me + fixes the authentication issue #29534 .

BelgianNoise commented 3 years ago

Until the linked PR is merged (and released) you can always download from vrtNU this way: https://github.com/ytdl-org/youtube-dl/issues/29534#issuecomment-904812729