search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.41k stars 10.04k forks source link

Comcast_SSO on ABC.com within the Comcast Network is unable to extract post url #30896

Closed TheJayMann closed 2 years ago

TheJayMann commented 2 years ago

Checklist

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://abc.com/shows/the-rookie/episode-guide/season-04/04-red-hot', '-o', '\\The.Rookie.S04E04.%(title)s.ts', '--ffmpeg-location', 'C:\\tools\\ytdl\\ffmpeg.exe', '--hls-prefer-ffmpeg', '--ap-mso', 'Comcast_SSO', '--ap-username', 'PRIVATE', '--verbose', '--proxy', '127.0.0.1:8888', '--no-check-certificate']
Type TV provider account password and press [Return]:
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg n4.3.1-222-g93061bc90c, ffprobe n4.3.1-222-g93061bc90c
[debug] Proxy map: {'https': '127.0.0.1:8888', 'http': '127.0.0.1:8888'}
[Go] 04-red-hot: Downloading webpage
[Go] VDKA24991970: Downloading JSON metadata
[Go] VDKA24991970: Downloading Provider Redirect Page
ERROR: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\go.py", line 234, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\adobepass.py", line 1453, in _extract_mvpd_auth
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\adobepass.py", line 1365, in post_form
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1021, in _html_search_regex
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1012, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I am attempting to download videos from ABC.com making use of my Comcast account, in which the post url is unable to be extracted. In an attempt to get even more debugging data, I passed the entire process through Fiddler proxy to see what was happening. What appears to be happening, when attempting to load sp.auth.adobe.com, a series of 302 redirects occur until the original abc.com url is loaded again. Below are the HTTP requests and responses with potentially sensitive values replaced with XXXX.

GET https://sp.auth.adobe.com/adobe-services/authenticate/saml?no_iframe=false&redirect_url=https%3A%2F%2Fabc.com%2Fshows%2Fthe-rookie%2Fepisode-guide%2Fseason-04%2F04-red-hot&mso_id=Comcast_SSO&domain_name=adobe.com&noflash=true&requestor_id=ABC HTTP/1.1
Connection: close
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3724.8 Safari/537.36
Host: sp.auth.adobe.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP/1.1 302 Found
Date: Mon, 25 Apr 2022 00:50:17 GMT
Content-Type: text/plain
Content-Length: 0
Connection: close
Set-Cookie: passgw=gw-ap-prod-uw2; Path=/; HttpOnly
Set-Cookie: AWSALB=XXXX; Expires=Mon, 02 May 2022 00:50:17 GMT; Path=/
Set-Cookie: AWSALBCORS=XXXX; Expires=Mon, 02 May 2022 00:50:17 GMT; Path=/; SameSite=None
Set-Cookie: JSESSIONID=XXXX; Path=/; Secure; HttpOnly
Set-Cookie: redirect_url=https%3A%2F%2Fabc.com%2Fshows%2Fthe-rookie%2Fepisode-guide%2Fseason-04%2F04-red-hot; path=/; Max-Age=2147483647; Secure; HttpOnly
Location: https://oauth.xfinity.com/oauth/authorize?response_type=code&redirect_uri=https://sp.auth.adobe.com/adobe-services/oauth2&state=XXXX&scope=openid profile https://login.comcast.net/pdp/tve&client_id=adobepass-abc&acr_values=urn:oasis:names:tc:SAML:2.0:ac:classes:InternetProtocol
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST,GET,OPTIONS,DELETE
Access-Control-Allow-Headers: ap_11,ap_42,ap_z,ap_19,ap_21,ap_23,authorization,content-type,pass_sfp,AP-Session-Identifier,AP-Device-Identifier,AP-SDK-Identifier,X-Device-Info
Access-Control-Expose-Headers: pass_sfp,Authzf-Error-Code,Authzf-Sub-Error-Code,Authzf-Error-Details
P3P: CP="NOI DSP COR CURa ADMa DEVa OUR BUS IND UNI COM NAV STA"
Cache-Control: No-Cache
Server: api-gateway/1.9.3.1
strict-transport-security: 60
X-Request-Id: XXXX

GET https://oauth.xfinity.com/oauth/authorize?response_type=code&redirect_uri=https://sp.auth.adobe.com/adobe-services/oauth2&state=XXXX&scope=openid%20profile%20https://login.comcast.net/pdp/tve&client_id=adobepass-abc&acr_values=urn:oasis:names:tc:SAML:2.0:ac:classes:InternetProtocol HTTP/1.1
Connection: close
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3724.8 Safari/537.36
Host: oauth.xfinity.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP/1.1 302 Found
Date: Mon, 25 Apr 2022 00:50:18 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubDomains;
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: https://login.xfinity.com/login?amr=urn%3Aoasis%3Anames%3Atc%3ASAML%3A2.0%3Aac%3Aclasses%3AInternetProtocol&c_ds_na=amr%2Cc_ds_na%2Cc_ds_no%2Cc_ds_ts%2Cclient_id%2Ccontinue%2CipAddrAuthn%2Cr%2CreqId%2Cs&c_ds_no=XXXX&c_ds_ts=XXXX&c_ds_val=XXXX&client_id=adobepass-abc&continue=https%3A%2F%2Foauth.xfinity.com%2Foauth%2Fauthorize%3Fresponse_type%3Dcode%26redirect_uri%3Dhttps%3A%2F%2Fsp.auth.adobe.com%2Fadobe-services%2Foauth2%26state%3DXXXX%26scope%3Dopenid%2520profile%2520https%3A%2F%2Flogin.comcast.net%2Fpdp%2Ftve%26client_id%3Dadobepass-abc%26acr_values%3Durn%3Aoasis%3Anames%3Atc%3ASAML%3A2.0%3Aac%3Aclasses%3AInternetProtocol%26response%3D1&ipAddrAuthn=1&r=comcast.net&reqId=XXXX&s=oauth
Content-Length: 0
Set-Cookie: SESSION=XXXX;Path=/oauth/;Secure;HttpOnly
Via: 1.1 oauth.xfinity.com
Connection: close
Set-Cookie: BIGipServerp_loginxf-wcdc-ipv6_443=XXXX; path=/; Httponly; Secure

GET https://login.xfinity.com/login?amr=urn%3Aoasis%3Anames%3Atc%3ASAML%3A2.0%3Aac%3Aclasses%3AInternetProtocol&c_ds_na=amr%2Cc_ds_na%2Cc_ds_no%2Cc_ds_ts%2Cclient_id%2Ccontinue%2CipAddrAuthn%2Cr%2CreqId%2Cs&c_ds_no=XXXX&c_ds_ts=XXXX&c_ds_val=XXXX&client_id=adobepass-abc&continue=https%3A%2F%2Foauth.xfinity.com%2Foauth%2Fauthorize%3Fresponse_type%3Dcode%26redirect_uri%3Dhttps%3A%2F%2Fsp.auth.adobe.com%2Fadobe-services%2Foauth2%26state%3DXXXX%26scope%3Dopenid%2520profile%2520https%3A%2F%2Flogin.comcast.net%2Fpdp%2Ftve%26client_id%3Dadobepass-abc%26acr_values%3Durn%3Aoasis%3Anames%3Atc%3ASAML%3A2.0%3Aac%3Aclasses%3AInternetProtocol%26response%3D1&ipAddrAuthn=1&r=comcast.net&reqId=XXXX&s=oauth HTTP/1.1
Connection: close
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3724.8 Safari/537.36
Host: login.xfinity.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP/1.1 302 Moved Temporarily
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubDomains;
Location: https://oauth.xfinity.com/oauth/authorize?response_type=code&redirect_uri=https://sp.auth.adobe.com/adobe-services/oauth2&state=XXXX&scope=openid%20profile%20https://login.comcast.net/pdp/tve&client_id=adobepass-abc&acr_values=urn:oasis:names:tc:SAML:2.0:ac:classes:InternetProtocol&response=1&cima.ticket=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2A
Content-Length: 0
Expires: Mon, 25 Apr 2022 00:50:20 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 25 Apr 2022 00:50:20 GMT
Connection: close
Set-Cookie: soc=1; Expires=Tue, 25-Apr-2023 00:50:19 GMT; Max-Age=31536000; Secure
Set-Cookie: tg_ticket=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; Path=/; SameSite=None; Secure; HttpOnly
Set-Cookie: bid=XXXX; Expires=Wed, 24 Apr 2024 00:50:20 UTC; SameSite=None; Secure; HttpOnly
Set-Cookie: SESSION=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Secure; HttpOnly

GET https://oauth.xfinity.com/oauth/authorize?response_type=code&redirect_uri=https://sp.auth.adobe.com/adobe-services/oauth2&state=XXXX&scope=openid%20profile%20https://login.comcast.net/pdp/tve&client_id=adobepass-abc&acr_values=urn:oasis:names:tc:SAML:2.0:ac:classes:InternetProtocol&response=1&cima.ticket=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2A HTTP/1.1
Cookie: SESSION=XXXX; BIGipServerp_loginxf-wcdc-ipv6_443=XXXX
Connection: close
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3724.8 Safari/537.36
Host: oauth.xfinity.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP/1.1 302 Found
Date: Mon, 25 Apr 2022 00:50:20 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubDomains;
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Cache-Control: no-store
Content-Language: en-US
Location: https://sp.auth.adobe.com/adobe-services/oauth2?code=XXXX&state=XXXX
Content-Length: 0
Set-Cookie: OAUTHCLIENTS=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Set-Cookie: SESSION=XXXX;Path=/oauth/;Secure;HttpOnly
Via: 1.1 oauth.xfinity.com
Connection: close

GET https://sp.auth.adobe.com/adobe-services/oauth2?code=XXXX&state=XXXX HTTP/1.1
Cookie: AWSALB=XXXX; AWSALBCORS=XXXX; redirect_url=https%3A%2F%2Fabc.com%2Fshows%2Fthe-rookie%2Fepisode-guide%2Fseason-04%2F04-red-hot; passgw=gw-ap-prod-uw2; JSESSIONID=XXXX
Connection: close
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3724.8 Safari/537.36
Host: sp.auth.adobe.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP/1.1 302 Found
Date: Mon, 25 Apr 2022 00:50:23 GMT
Content-Length: 0
Connection: close
Set-Cookie: AWSALB=XXXX; Expires=Mon, 02 May 2022 00:50:21 GMT; Path=/
Set-Cookie: AWSALBCORS=XXXX; Expires=Mon, 02 May 2022 00:50:21 GMT; Path=/; SameSite=None
Set-Cookie: redirect_url=https%3A%2F%2Fabc.com%2Fshows%2Fthe-rookie%2Fepisode-guide%2Fseason-04%2F04-red-hot; path=/; Max-Age=2147483647; Secure; HttpOnly
Set-Cookie: JSESSIONID=XXXX; Path=/; Secure; HttpOnly
Location: https://abc.com/shows/the-rookie/episode-guide/season-04/04-red-hot
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST,GET,OPTIONS,DELETE
Access-Control-Allow-Headers: ap_11,ap_42,ap_z,ap_19,ap_21,ap_23,authorization,content-type,pass_sfp,AP-Session-Identifier,AP-Device-Identifier,AP-SDK-Identifier,X-Device-Info
Access-Control-Expose-Headers: pass_sfp,Authzf-Error-Code,Authzf-Sub-Error-Code,Authzf-Error-Details
P3P: CP="NOI DSP COR CURa ADMa DEVa OUR BUS IND UNI COM NAV STA"
Cache-Control: No-Cache
Server: api-gateway/1.9.3.1
strict-transport-security: 60
X-Request-Id: XXXX
TheJayMann commented 2 years ago

Am I to assume that this is why the browser states that I need to enable DRM in order to play the video? I also attempted to load a variety of content from abc.com, both episodes as well as clips, and all of them have the enable DRM notification, which would suggest abc.com is not usable with youtube-dl anymore. Either that, or I'm missing something here.

TheJayMann commented 2 years ago

Well, interestingly enough, I try that Grey's Anatomy episode you mention, and, without specifying any --cookies or --ap-mso options, it downloads just fine (or, at least the first two minutes worth of data before I cancelled the download). So, without any further evidence, I am currently left to assume that the DRM in the show is preventing the download option.

TheJayMann commented 2 years ago

Apparently my previous communications lacked clarity. I did not mean to suggest the presence of the DRM is causing the issue, but that the lack of a DRM free option, and, thus I should have stated "DRM only" rather than just "DRM," though I assumed this would have been understood.

TheJayMann commented 2 years ago

I don't care anymore.

dirkf commented 2 years ago

The top-level metadata for a show might say it's DRMed (yt-dl might believe that and skip the download; the browser might also believe it and show a diagnostic that DRM/EME needs to be enabled). But, the actual stream might not be restricted as claimed.

dirkf commented 9 months ago

In any case, the original log never saw the media but failed at the MSO fence; duplicate of #11257 etc.