ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.35k stars 10.04k forks source link

Issue with syfy when using comcast sso #20749

Closed joshtish closed 5 years ago

joshtish commented 5 years ago

Please follow the guide below


Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.04.17. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?


The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.syfy.com/themagicians/videos/413-no-better-to-be-safe-than-sorry', u'--ap-mso=Comcast_SSO', u'--ap-username=PRIVATE', u'--verbose']
Type TV provider account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.17
[debug] Python version 2.7.10 (CPython) - Darwin-18.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[Syfy] 413-no-better-to-be-safe-than-sorry: Downloading webpage
[Syfy] 3940836: Downloading Provider Redirect Page
ERROR: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/syfy.py", line 47, in _real_extract
    url, video_id, 'syfy', resource)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/adobepass.py", line 1448, in _extract_mvpd_auth
    self._DOWNLOADING_LOGIN_PAGE)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/adobepass.py", line 1360, in post_form
    post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_page, 'post url', group='url')
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1013, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1004, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description of your issue, suggested solution and other information

Attempting to use youtube-dl on syfy using my comcast credentials results in a "Unable to extract post url" bug. It asked me to file a bug here, so I'm filing a bug here.

ealgase commented 5 years ago

As far as I can tell the Comcast_SSO doesn't work at all (didn't work for me on Simpsons World).

ealgase commented 5 years ago

This also is related/duplicate of #11257, #11538, and #16630.

joshtish commented 5 years ago

Could well be a dupe of that last one for sure - the "Automatically signing you in" is the page I get when browsing from my home network - and it might be hitting that and failing out in the same way as #16630.

joshtish commented 5 years ago

Just to follow-up for people finding this in the future. I tested on an external connection where I don't get automatically signed-in, and things work as one would expect. So, oddly enough... if you want to use your comcast credentials, you can't be in your own home, or if you are home, you must use VPN.

tv21 commented 5 years ago

For those running from source code or compiling your own, the quick fix for Comcast Xfinity is to use this version of adobepass.py:

https://raw.githubusercontent.com/bhamboss/youtube-dl/patch-1/youtube_dl/extractor/adobepass.py

That is from https://github.com/bhamboss/youtube-dl/blob/patch-1/youtube_dl/extractor/adobepass.py and I can confirm this works with Xfinity's new login if you are coming in from an Xfinity IP address.

Just to be clear, Xfinity authenticates differently if you are coming in from one of their IP addresses vs. if you are coming in from the IP of some other provider. I tested the above patch coming from an Xfinity IP. I did NOT test it from another provider's IP.

If I run diff -u on the original and patched files I get this:

$ diff -u adobepass.py.original adobepass.py.patched
--- adobepass.py.original   2018-10-17 01:22:56.000000000 -0400
+++ adobepass.py.patched    2019-04-23 08:53:36.000000000 -0400
@@ -1433,6 +1433,14 @@
                             provider_redirect_page, 'oauth redirect')
                         self._download_webpage(
                             oauth_redirect_url, video_id, 'Confirming auto login')
+                    elif 'automatically signed in with' in provider_redirect_page:
+                        # Seems like comcast is rolling up new way of automatically signing customers
+                        oauth_redirect_url = self._html_search_regex(
+                            r'continue:\s*"(https://oauth.xfinity.com/oauth/authorize\?.+)"',
+                            provider_redirect_page, 'oauth redirect (signed)')
+                        # Just need to process the request. No useful data comes back
+                        self._download_webpage(
+                            oauth_redirect_url, video_id, 'Confirming auto login')
                     else:
                         if '<form name="signin"' in provider_redirect_page:
                             provider_login_page_res = provider_redirect_page_res