search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.62k stars 9.97k forks source link

Add support for Adobe Pass Auth TV Provider Cox Communications #11396

Open elfurbe opened 7 years ago

elfurbe commented 7 years ago

What is the purpose of your issue?

Description of your issue, suggested solution and other information

Would like to request support for Cox Communications as an Adobe Pass MSO option. I can provide a Cox login as needed.

elfurbe commented 7 years ago

I'd rather not post my login details to a public issue. Is there an email address I can send them to?

An aside, adding the label doesn't appear to send a notification so I didn't realize you were waiting on a response.

yan12125 commented 7 years ago

Well, the label might be misleading. Actually it means "if a developer wants to fix it, an account is necessary". That is, there's no developer working on this yet. Those who are interested will post his/her email address here and ask for credentials.

remitamine commented 7 years ago

duplicate of #10871.

elfurbe commented 7 years ago

I don't think Charter and Cox are the same thing? I'm not sure how this ended up titled with "Charter" references, I didn't realize I'd titled it that. I have no interest in Charter, just Cox.

backcountrymountains commented 7 years ago

Hello.

I would also like to request support for Cox Communications as an Adobe Pass MSO option. It is listed on Adobe's list of supported MSO's

<id visible="true">Cox</id>
<displayName visible="true">Cox</displayName>
<logoUrl visible="true">
http://ww2.cox.com/wcm/en/common/image/cox/COX_logo_112x33.png
</logoUrl>
<foregroundLogout>true</foregroundLogout>
<authPerAggregator>false</authPerAggregator>
<iFrameRequired visible="true">false</iFrameRequired>
</mvpd>

I'm running windows but I could try linux so that I can run the uncompiled python.

I read the post about Charter referenced above.

Can I just update the adobepass.py file with the mso_id of Cox? 

mso_id = Cox
username_field = IDToken1
password_field = IDToken2

I'd really appreciate a pointer in the right direction here.

Thanks.

backcountrymountains commented 7 years ago

Is it really possible to just do a wget https://github.com/rg3/youtube-dl/archive/master.zip unzip it, edit the youtube-dl-master/youtube_dl/extractor/adobepass.py to include:

MSO_INFO = {
    'Cox': {
        'name': 'Cox',
        'username_field': 'IDToken1',
        'password_field': 'IDToken2',
    },
}

and then run youtube-dl-master/youtube_dl/__main__.py --ap-mso Cox --ap-username username@cox.net --ap-password ?

Any chance some super cool python developer/linux kung-fu master can compile a windows binary with the above changes for me to try?

Thanks!

backcountrymountains commented 7 years ago

Okay, so I got python for windows and and changed the adobepass.py but the Cox login page uses "username" and "password"

<input type="text" name="username" value="" maxlength="128" style="width:130px;">
<input type="password" name="password" maxlength="128" style="width:130px;">

so I put in the following:

'Cox': {
        'name': 'Cox',
        'username_field': 'username',
        'password_field': 'password',
    },

I found a locked video on Comedy Central, that had failed to work without credentials before. (Don't judge my video selection)

And then I ran: python.exe ~\youtube-dl-master\youtube_dl\__main__.py --ap-mso Cox --ap-username *username --ap-password *password http://www.cc.com/episodes/sveupz/adam-devine-s-house-party-neighbor-party-season-1-ep-107

and--holy crap!--stuff started to happen!

[ComedyCentral] adam-devine-s-house-party-neighbor-party-season-1-ep-107: Downloading webpage
[ComedyCentral] Downloading JSON metadata
[ComedyCentral] 80182651-fc4d-4037-adee-91ce361278e5: Downloading info
[ComedyCentral] 0bfa2f25-e954-484e-bf95-2ced3991b609: Extracting information
[ComedyCentral] 0bfa2f25-e954-484e-bf95-2ced3991b609: Downloading video urls
[ComedyCentral] 0bfa2f25-e954-484e-bf95-2ced3991b609: Downloading m3u8 information
[ComedyCentral] 26e72a25-5d14-4d84-b5d9-50d23437a5b7: Extracting information
[ComedyCentral] 26e72a25-5d14-4d84-b5d9-50d23437a5b7: Downloading video urls
[ComedyCentral] 26e72a25-5d14-4d84-b5d9-50d23437a5b7: Downloading m3u8 information
[ComedyCentral] 13dedffa-42bc-449b-9953-db2f05e68148: Extracting information
[ComedyCentral] 13dedffa-42bc-449b-9953-db2f05e68148: Downloading video urls
[ComedyCentral] 13dedffa-42bc-449b-9953-db2f05e68148: Downloading m3u8 information
[ComedyCentral] 2683954d-ca3e-4ffe-861a-0d2f431938b7: Extracting information
[ComedyCentral] 2683954d-ca3e-4ffe-861a-0d2f431938b7: Downloading video urls
[ComedyCentral] 2683954d-ca3e-4ffe-861a-0d2f431938b7: Downloading m3u8 information
[ComedyCentral] f718274c-7210-41fc-ad91-4ef35b4b824d: Extracting information
[ComedyCentral] f718274c-7210-41fc-ad91-4ef35b4b824d: Downloading video urls
[ComedyCentral] f718274c-7210-41fc-ad91-4ef35b4b824d: Downloading m3u8 information
[download] Downloading playlist: Neighbor Party
[ComedyCentral] playlist Neighbor Party: Collected 5 video ids (downloading 5 of them)
[download] Downloading video 1 of 5
[hlsnative] Downloading m3u8 manifest

but then it all fell apart

[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: Adam Devine's House Party_October 4, 2013_1_107_Neighbor Party_Act 1-0bfa2f25-e954-484e-bf95-2ced3991b609.mp4
ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Now I have to try and work ffmpeg into windows/python. Hrm.

Anyway, it looks like that adobepass.py MSO edit works for Cox so hopefully someone out there will notice this if they need it.

remitamine commented 7 years ago

Anyway, it looks like that adobepass.py MSO edit works for Cox so hopefully someone out there will notice this if they need it.

no, testing on Comedy Central does not prove that the change is working, Comedy Central extractor does not support Adobe Pass authentication.

backcountrymountains commented 7 years ago

So, installed ffmpeg and used --ffmpeg-location and it all worked! This thing is amaze-balls!

Thanks to @tv21 in the other thread for pointing me in the right direction.

backcountrymountains commented 7 years ago

Aaaaaand I broke it.

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '--ap-mso', 'Cox', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--ffmpeg-location', 'ffmpeg.exe', 'http://www.tbs.com/videos/full-frontal-with-samantha-bee/season-2/episode-11/june-14-2017.html'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] youtube-dl version 2017.06.18 [debug] Python version 3.6.1 - Windows-10-10.0.14393-SP0 [debug] exe versions: ffmpeg N-86537-gae6f6d4, ffprobe N-86537-gae6f6d4 [debug] Proxy map: {} [TBS] june-14-2017: Downloading webpage [TBS] june-14-2017: Downloading XML [TBS] 2097653: Downloading Provider Redirect Page [TBS] 2097653: Downloading Provider Login Page [TBS] 2097653: Logging in ERROR: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\YoutubeDL.py", line 762, in extract_info ie_result = ie.extract(url) File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\common.py", line 433, in extract ie_result = self._real_extract(url) File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\tbs.py", line 55, in _real_extract 'auth_required': video_params.get('isAuthRequired') != 'false', File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\turner.py", line 79, in _extract_cvp_info query['accessToken'] = self._extract_mvpd_auth(ap_data['url'], video_id, ap_data['site_name'], ap_data['site_name']) File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\adobepass.py", line 1485, in _extract_mvpd_auth post_form(mvpd_confirm_page_res, 'Confirming Login') File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\adobepass.py", line 1357, in post_form post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P.+?)\1', form_page, 'post url', group='url') File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\common.py", line 791, in _html_search_regex res = self._search_regex(pattern, string, name, default, fatal, flags, group) File "C:\Users\O\Downloads\python-3.6.1-embed-amd64\youtube-dl-master\youtube_dl\extractor\common.py", line 782, in _search_regex raise RegexNotFoundError('Unable to extract %s' % _name) youtube_dl.utils.RegexNotFoundError: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

backcountrymountains commented 7 years ago

I don't understand because if I inspect the login page using chrome I find:


<form method="POST" action="https://idm.east.cox.net/idm/coxnetlogin" enctype="application/x-www-form-urlencoded" name="LoginPage" onsubmit="return preProcess();" style="font-size: 13px; font-family: Arial, Helvetica, sans-serif; color: 151515; margin-left: 3em; margin-top: 1em; padding: 0px;">
<span style="font-size: 12px; font-weight: bold; font-family: Arial, Helvetica, sans-serif; color: 151515; line-height: 20pt;">User ID</span><br>
<input type="text" name="username" value="" maxlength="128" style="width:130px;"><br>
<span style="font-size: 12px; font-weight: bold; font-family: Arial, Helvetica, sans-serif; color: 151515; line-height: 20pt;">Password</span><br>
<input type="password" name="password" maxlength="128" style="width:130px;"><br>
<input id="rememberme" type="checkbox" name="rememberme" value="true" style="vertical-align: middle;"><span style="font-size: 11px; font-weight: normal; font-family: Arial, Helvetica, sans-serif; color: 151515; line-height: 20pt; vertical-align: middle;">Remember me</span>
<input type="hidden" name="target" value="https%3A%2F%2Fidm.east.cox.net%2Faffwebservices%2Ftbs%2Fredirect.jsp%3FSAMLRequest%3DnVPLbttADLz3K4S96%252BlH24WlwIkb1ECaqrbSQ28riYrXkHbVJZXYf9%252BVbDdugxpFrgQ5nOEMZ1e7pnaewKDUKmahFzAHVKFLqR5j9pDduh%252FYVfJuhqKpo5bPO9qoFfzsAMmZI4IhO3ejFXYNmDWYJ1nAw%252BouZhuiFrnvY%252BsJO%252BSJUufgFbqxFb9H89fzL3evIJizsNBSCRr4nFBk2XggkCzAzlNAfDwe%252BaKqniHHw1L02y6vZTFgR4iaObfaFDAwjlklagTmLBcxE1E4DeW22uSjqpiW23AqpzKaBNtxGeSTR9uEqUCUT%252FAyhtjBUiEJRTGLgvC9G0zdKMiCMZ%252BEfPzRG40mP5iTGk260PW1VIcDdkZxLVAiV6IB5FTwXjaPvIDnhybkn7MsddOv64w5309GRL0R1hqF%252FHD6y1jtcTFLDk7xgbE5R7gMIE5GsOS3c3bKe22fVm4OG1FXrq787Ho98883JqekfNoRqF4LJjPamJb2fAXYalVm%252BkjrUP43r5MoDjuyzdKUbisM7d9A8W8GyZH1nzyPpXvLYrlItU3T3pnXtX6%252BMSDIxoFMB0OuGkGX79lXZOlWQysnI%252BwOUMScddrDf%252BtELSsJ5uxR%252Fk%252FLm1Lhv%252Bg9%252F%252BDkFw%253D%253D%26SAMLTRANSACTIONID%3D2541fa8c-232aaf5c-fd343918-aebce25d-6e7d8925-2f0%26SMPORTALURL%3Dhttps%253A%252F%252Fidm.east.cox.net%253A443%252Faffwebservices%252Fpublic%252Fsaml2sso%26SigAlg%3Dhttp%253A%252F%252Fwww.w3.org%252F2000%252F09%252Fxmldsig%2523rsa-sha1%26Signature%3DBQ%252BdrZ6Vz4shbMcE%252FV9fgecUNTGDp3pkiULicabYdoUpwoiO5Mkv26bgxbE2VlRrzbNhbkNbu5NQbnzO4oczcf%252FN5kGQYqCMEQXMlJuAyEGk6vbZY5p3Va%252F%252BEKYY%252BHyLiGPfZMDH%252F5BdFLP1yKlVLVmysrj4VMticYKzKQpnj4XJlmkntTxb7QG%252BI%252FNybjK9Wwyf7XBc8v1CZnGnv9vajfxcT2Pnh5y%252Bb38TiYJ3IyJ7tNB3NHL3rHWjdMOmSbU1afeeCQd8hM2Nu6su06FHZtEJWQ6r05YjjbREwRgmlzk62nTm6DgfHumAOj7UmO1S09gqn8VDkUcaTm%252Br5yYnNA%253D%253D%26">
<input type="hidden" name="onsuccess" value="https%3A%2F%2Fidm.east.cox.net%2Faffwebservices%2Ftbs%2Fredirect.jsp%3FSAMLRequest%3DnVPLbttADLz3K4S96%252BlH24WlwIkb1ECaqrbSQ28riYrXkHbVJZXYf9%252BVbDdugxpFrgQ5nOEMZ1e7pnaewKDUKmahFzAHVKFLqR5j9pDduh%252FYVfJuhqKpo5bPO9qoFfzsAMmZI4IhO3ejFXYNmDWYJ1nAw%252BouZhuiFrnvY%252BsJO%252BSJUufgFbqxFb9H89fzL3evIJizsNBSCRr4nFBk2XggkCzAzlNAfDwe%252BaKqniHHw1L02y6vZTFgR4iaObfaFDAwjlklagTmLBcxE1E4DeW22uSjqpiW23AqpzKaBNtxGeSTR9uEqUCUT%252FAyhtjBUiEJRTGLgvC9G0zdKMiCMZ%252BEfPzRG40mP5iTGk260PW1VIcDdkZxLVAiV6IB5FTwXjaPvIDnhybkn7MsddOv64w5309GRL0R1hqF%252FHD6y1jtcTFLDk7xgbE5R7gMIE5GsOS3c3bKe22fVm4OG1FXrq787Ho98883JqekfNoRqF4LJjPamJb2fAXYalVm%252BkjrUP43r5MoDjuyzdKUbisM7d9A8W8GyZH1nzyPpXvLYrlItU3T3pnXtX6%252BMSDIxoFMB0OuGkGX79lXZOlWQysnI%252BwOUMScddrDf%252BtELSsJ5uxR%252Fk%252FLm1Lhv%252Bg9%252F%252BDkFw%253D%253D%26SAMLTRANSACTIONID%3D2541fa8c-232aaf5c-fd343918-aebce25d-6e7d8925-2f0%26SMPORTALURL%3Dhttps%253A%252F%252Fidm.east.cox.net%253A443%252Faffwebservices%252Fpublic%252Fsaml2sso%26SigAlg%3Dhttp%253A%252F%252Fwww.w3.org%252F2000%252F09%252Fxmldsig%2523rsa-sha1%26Signature%3DBQ%252BdrZ6Vz4shbMcE%252FV9fgecUNTGDp3pkiULicabYdoUpwoiO5Mkv26bgxbE2VlRrzbNhbkNbu5NQbnzO4oczcf%252FN5kGQYqCMEQXMlJuAyEGk6vbZY5p3Va%252F%252BEKYY%252BHyLiGPfZMDH%252F5BdFLP1yKlVLVmysrj4VMticYKzKQpnj4XJlmkntTxb7QG%252BI%252FNybjK9Wwyf7XBc8v1CZnGnv9vajfxcT2Pnh5y%252Bb38TiYJ3IyJ7tNB3NHL3rHWjdMOmSbU1afeeCQd8hM2Nu6su06FHZtEJWQ6r05YjjbREwRgmlzk62nTm6DgfHumAOj7UmO1S09gqn8VDkUcaTm%252Br5yYnNA%253D%253D%26">
<input type="hidden" name="onfailure" value="http://idm.east.cox.net/coxlogin/ui/tvonline">
<br>
<input type="image" src="https://idm.east.cox.net/cci/coxmain/coxlogin/images/signin.png" alt="Sign In" style="margin-top: 10px;"><p></p><input type="hidden" name="MFP" value=""><input type="hidden" name="CallerID" value=""><input type="hidden" name="DeviceID" value=""></form>

Which seems to have the POST url.

I don't get Regex.

backcountrymountains commented 7 years ago

In response to @remitamine, I was able to download a comedy central episode that was locked and required me to login to Cox when using chrome. If Comedy Central doesn't use Adobe Pass, then how was I able to use youtube-dl to download the video?

Of course, @remitamine is correct, and I can download the video using youtube-dl without any adobe pass options. Sigh. I have accomplished nothing.

gkoelln commented 7 years ago

I would suggest getting HTTP Headers add-on/extension for Firefox or Chrome and see where the traffic is going. I haven't had much success getting my pull requests merged, but I'm willing to try adding support if anyone wants to e-mail me their Cox credentials at gkoelln7 [at] yahoo [dot] com.

WRootX commented 6 years ago

Are there unit tests to validate changes to the adobepass.py file? I scanned the test folder and didn't see anything directly testing the AdobePassIE class.

For Cox, I found two issues: the login page has multiple html form elements, so the regex in post_form(), which only matches the first form element, finds the wrong POST url; the correct form is the second or third form on the page.

Second, the flow from the login page to the posting of the SAML response is slightly different.

I coded a modified adobepass.py file and was able to pull from Cox, including a currently locked version of the URL in the linked issue #13929 (which, itself, is no longer locked).

But the fixes require a change to the post_form() method, adding an optional form_id parameter and changing the regex to include lookahead since the order of the attributes on the form tag can be volatile. I put the second change, for the flow into the SAML response, in an "if 'Cox'" block so I'm less worried about breaking working code with that change, but since the post_form() method is shared by all extractors I don't want to submit the change without doing my best to validate that I'm not damaging anything.

RLovelett commented 5 years ago

@WRootX can you elaborate on this part?

Second, the flow from the login page to the posting of the SAML response is slightly different.

I've been trying to work through this myself and I've basically to the point where it is POSTing to https://idm.east.cox.net/idm/coxnetlogin. But while performing "Retrieving Session" I get:

ERROR: Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I think what your referring to is that there are hidden fields in the <form>. Or is that a red herring?

HTML Form for Cox.net (expand to see full HTML) ```html
Sign In
No Account? Register Now! Forgot User ID / Password?
```
jmb12686 commented 5 years ago

This issue is over 2 years old and still unresolved. I to have Cox cable credentials and hope to leverage this utility.

Is anyone currently looking into this still? If not, I may try to take a crack at it.

RLovelett commented 5 years ago

@jmb12686 I continued to look into it. I made a little bit of progress. I got it to where it was properly authenticating (or at least I think it was properly authenticating) but I could never get the session information to be stored/handed off such that the rest of the code worked.

I think the challenge I was/am facing is that I just don't know Python (I happen to dislike it) and so I'm fighting the language and it's idioms as well as trying to reverse engineer the login mechanisms.

peppy6582 commented 5 years ago

@RLovelett I appreciate what you are doing. Python can get a little weird sometimes. Anything I can help you with, let me know. I would love for this to work.

TheQuiz03 commented 1 year ago

still seems that that support hasn't been added. I am willing to work with a developer, but i wont be able to share credentials