Open phinneus1 opened 7 years ago
I've been trying to add Dish MSO as well. However, the error I'm getting is "ERROR: Unable to extract post url;" and I'm getting it on this page too. I have a Dish login and when I try to view this video in my browser, it tries to load for a few seconds, then stops as if the video has finished playing.
if your using firefox that happened to me today also. Tried it in chrome and worked perfectly though.
Edit. tried it just now and working fine in firefox
I have crappy Satelite internet.......that why I bought vps........I download stuff there without dropping packets and then ftp home during my freezone 12am-5am
Can you try your command with the --dump-pages option? I want to see what you're getting vs. what I'm getting.
as of right now it hanging at [Syfy] 2877726: Downloading Provider Login Page and I dont see any additional info......is there a syntax I should be using? ill update this when it times out.
my bad........forgot to put command in but it looks like page is ncoded or something.........ill toss it in pastebin cause its huge.
I'm getting further, but when I get to the login confirmation, and I submit the form, I think it presses the Cancel button.
<fieldset class="buttons">
<legend>Sign in to DISH Online ID using your account?</legend>
<input type="submit" value="Sign In" class="submit button selected" id="allow">
<input class="submit button" id="cancel" name="cancel" type="submit" value="Cancel">
</fieldset>
Anyone have any ideas how to submit the form by pressing the Sign In button instead of the Cancel button?
Here's what I have so far:
'Dish': {
'name': 'Dish',
'username_field': 'username',
'password_field': 'password',
},
...
elif mso_id == 'Dish':
authstate_page_res = post_form(
provider_redirect_page_res, 'Downloading AuthState Page')
authstate_page, urlh = authstate_page_res
form_data = self._hidden_inputs(authstate_page)
provider_login_page_res = self._download_webpage_handle(
urlh.geturl(), video_id, 'Downloading Provider Login Page', query=form_data)
mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', {
mso_info.get('username_field', 'username'): username,
mso_info.get('password_field', 'password'): password,
})
post_form(mvpd_confirm_page_res, 'Confirming Login')
...or maybe it's during the login process that it's choosing the wrong form?
<div class="cobrandlogin">
<div class="social_login"><div class="logo_bar"><img class="logo_left" src="/images/dish/dish_logo_small.png" alt="DISH Network Logo"></div><div class="left_side"><form id="syn_login_form" method="POST" action="https://identity1.dishnetwork.com/saml/module.php/sociallogin/login.php?AuthState=_2bf2d3dc2902bead4ad63ea766c978a964a46b2f5c%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fsaml.sp.auth.adobe.com%26cookieTime%3D1486004317%26RequesterID%3D%255B%2522syfy%2522%255D%26NameIDFormat%3Durn%253Aoasis%253Anames%253Atc%253ASAML%253A2.0%253Anameid-format%253Atransient"><div id="login_form" class="form"><span class="login_message">Enter your DISH Online ID and Password</span><label for="username">Online ID</label><input placeholder="Online ID" class="input" id="username" type="text" name="username" value="" tabindex="1" autocapitalize="off" autocorrect="off" required/><div class="field-help"></div><label for="password">Password</label><input placeholder="Password" class="input" id="password" type="password" name="password" value="" tabindex="2" autocapitalize="off" autocorrect="off" required/><div class="field-help"></div><input type="hidden" name="login_type" value="username,password" /><input type="hidden" name="source" value="authsynacor_identity1.dishnetwork.com" /><button id="login" type="submit" name="source_button" value="authsynacor_identity1.dishnetwork.com" class="login" tabindex="3">
Log In
</button><span class="remember"><input type="checkbox" name="remember_me" value="yes" tabindex="4" checked="checked">
Remember Me
</span><!-- CUSTOM SELFCARE AREA --><ul id="selfcare_list"><li><a target="_blank" id="forgot_online_id" href="https://my.dish.com/customercare/usermanagement/prepForgotOnlineId.do">Forgot Online ID?</a></li><li><a target="_blank" id="forgot_password" href="https://my.dish.com/customercare/usermanagement/prepForgotPassword.do">Forgot Password?</a></li><li><a target="_blank" id="self_help" href="https://my.dish.com/customercare/usermanagement/verify.do">Create Online ID</a></li></ul><!-- CUSTOM SELFCARE AREA --></div></form></div><div class="right_side pull-right"><div class="social_buttons pull-right"><span class="login_message">Or log in using:</span><div class="social_buttons_wrapper"><form method="POST" action="https://identity1.dishnetwork.com/saml/module.php/sociallogin/login.php?AuthState=_2bf2d3dc2902bead4ad63ea766c978a964a46b2f5c%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fsaml.sp.auth.adobe.com%26cookieTime%3D1486004317%26RequesterID%3D%255B%2522syfy%2522%255D%26NameIDFormat%3Durn%253Aoasis%253Anames%253Atc%253ASAML%253A2.0%253Anameid-format%253Atransient"><input type="hidden" name="source" value="identity1.dishnetwork.com_facebook" /><input type="hidden" name="switch_as" value="1" /><button id="as_button_identity1.dishnetwork.com_facebook" type="submit" name="source_button" value="identity1.dishnetwork.com_facebook" class="facebook" tabindex="5"><span class="facebook logo pull-left"></span><span class="button_label pull-right">Facebook</span></button></form><form method="POST" action="https://identity1.dishnetwork.com/saml/module.php/sociallogin/login.php?AuthState=_2bf2d3dc2902bead4ad63ea766c978a964a46b2f5c%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fsaml.sp.auth.adobe.com%26cookieTime%3D1486004317%26RequesterID%3D%255B%2522syfy%2522%255D%26NameIDFormat%3Durn%253Aoasis%253Anames%253Atc%253ASAML%253A2.0%253Anameid-format%253Atransient"><input type="hidden" name="source" value="identity1.dishnetwork.com_googleplus" /><input type="hidden" name="switch_as" value="1" /><button id="as_button_identity1.dishnetwork.com_googleplus" type="submit" name="source_button" value="identity1.dishnetwork.com_googleplus" class="googleplus" tabindex="5"><span class="googleplus logo pull-left"></span><span class="button_label pull-right">Google</span></button></form><form method="POST" action="https://identity1.dishnetwork.com/saml/module.php/sociallogin/login.php?AuthState=_2bf2d3dc2902bead4ad63ea766c978a964a46b2f5c%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fsaml.sp.auth.adobe.com%26cookieTime%3D1486004317%26RequesterID%3D%255B%2522syfy%2522%255D%26NameIDFormat%3Durn%253Aoasis%253Anames%253Atc%253ASAML%253A2.0%253Anameid-format%253Atransient"><input type="hidden" name="source" value="identity1.dishnetwork.com_twitter" /><input type="hidden" name="switch_as" value="1" /><button id="as_button_identity1.dishnetwork.com_twitter" type="submit" name="source_button" value="identity1.dishnetwork.com_twitter" class="twitter" tabindex="5"><span class="twitter logo pull-left"></span><span class="button_label pull-right">Twitter</span></button></form></div></div></div>
</div>
So many hoops to jump through! I've made it this far:
elif mso_id == 'Dish':
authstate_page_res = post_form(
provider_redirect_page_res, 'Downloading AuthState Page')
authstate_page, urlh = authstate_page_res
form_data = self._hidden_inputs(authstate_page)
provider_login_page_res = self._download_webpage_handle(
urlh.geturl(), video_id, 'Downloading Provider Login Page', query=form_data)
provider_login_page, urlh = provider_login_page_res
post_url = self._html_search_regex(
r'<form id="syn_login_form"[^>]+action="(?P<url>.+?)">',
provider_login_page, 'post url', group='url')
mvpd_confirm_page_res = self._download_webpage_handle(
post_url, video_id, 'Logging in', data=urlencode_postdata(login_info),
headers={
'Content-Type': 'application/x-www-form-urlencoded',
})
mvpd_confirm_page, urhl = mvpd_confirm_page_res
confirm_redirect_url = self._html_search_regex(
r'content="0;\s*url=([^\'"]+)',
mvpd_confirm_page, 'meta refresh redirect')
confirmation_page_res = self._download_webpage_handle(
confirm_redirect_url,
video_id, 'Confirming Login')
confirmation_page, urlh = authstate_page_res
form_data = self._hidden_inputs(confirmation_page)
provider_login_page_res = self._download_webpage_handle(
urlh.geturl(), video_id, 'Redirecting', query=form_data)
But this last page is just this:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Redirecting...</title>
</head>
<body onload="window.history.go(-3);">
</body>
</html>
Now where do we go from here? I'm still getting ERROR: Unable to download webpage: HTTP Error 401: Unauthorized (caused by <HTTPError 401: 'Unauthorized'>); after it starts to retrieve session.
[Syfy] 110-escape-velocity: Downloading webpage
[Syfy] 2877726: Downloading Provider Redirect Page
[Syfy] 2877726: Downloading AuthState Page
[Syfy] 2877726: Downloading Provider Login Page
[Syfy] 2877726: Logging in
[Syfy] 2877726: Confirming Login
[Syfy] 2877726: Redirecting
[Syfy] 2877726: Retrieving Session
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ap-mso', 'Dish', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', 'http://www.syfy.com/killjoys/videos/110-escape-velocity', '-s', '-v']
[debug] Encodings: locale cp1252, fs utf-8, out cp1252, pref cp1252
[debug] youtube-dl version 2017.01.10
[debug] Git HEAD: 3a407e707
[debug] Python version 3.6.0 - Windows-10-10.0.14393-SP0
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2
[debug] Proxy map: {}
ERROR: Unable to download webpage: HTTP Error 401: Unauthorized (caused by <HTTPError 401: 'Unauthorized'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\python36\lib\youtube-dl\youtube_dl\extractor\common.py", line 406, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "C:\python36\lib\youtube-dl\youtube_dl\YoutubeDL.py", line 2001, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "C:\Python36\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Python36\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python36\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\Python36\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Python36\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
FYI: In many cases 401 is caused by insufficient information. --print-traffic and --write-pages can help for debugging login problems. Note that --print-traffic works better with Python 2 than Python 3.
So here's the route it takes: http://www.syfy.com/killjoys/videos/110-escape-velocity
https://sp.auth.adobe.com/adobe-services/authenticate/saml
https://identity1.dishnetwork.com/saml/saml2/idp/SSOService.php
And finally, <body onload="window.history.go(-3);">
(which I assume is back to the main video)
So, looking at the traffic, I don't see any insufficient information, but is it because it's not redirecting to the original page?
Dunno. Maybe Cookies or User-Agent that's causing the failure.
First, could your reproduce all requests necessary for logging in?
I think I'm there, though I'm not entirely sure it's the cleanest method. I got it work, anyway.
'Dish': {
'name': 'Dish',
'username_field': 'username',
'password_field': 'password',
},
...
elif mso_id == 'Dish':
authstate_page_res = post_form(
provider_redirect_page_res, 'Downloading AuthState Page')
authstate_page, urlh = authstate_page_res
form_data = self._hidden_inputs(authstate_page)
provider_login_page_res = self._download_webpage_handle(
urlh.geturl(), video_id, 'Downloading Provider Login Page', query=form_data)
provider_login_page, urlh = provider_login_page_res
post_url = self._html_search_regex(r'<form id="syn_login_form"[^>]+action=(["\'])(?P<url>.+?)\1', provider_login_page, 'post url', group='url')
mvpd_confirm_page_res = self._download_webpage_handle(
post_url, video_id, 'Logging in', data=urlencode_postdata({
mso_info.get('username_field', 'username'): username,
mso_info.get('password_field', 'password'): password,
'login_type': 'username&password',
'source': 'authsynacor_identity1.dishnetwork.com',
'source_button': 'authsynacor_identity1.dishnetwork.com',
'remember_me': 'no',
}), headers={
'Content-Type': 'application/x-www-form-urlencoded',
})
mvpd_confirm_page, urlh = mvpd_confirm_page_res
redirect_url = self._html_search_regex(
r'content="0;\s*url=([^\'"]+)',
mvpd_confirm_page, 'meta refresh redirect')
confirm_page_res = self._download_webpage_handle(
redirect_url,
video_id, 'Redirecting')
authstate_page, urlh = authstate_page_res
form_data = self._hidden_inputs(authstate_page)
redirect_page_res = self._download_webpage_handle(
urlh.geturl(), video_id, 'Reloading AuthState Page', query=form_data)
redirect_page, urlh = redirect_page_res
confirm_url = urlh.geturl()
confirm_url = confirm_url.replace('firstbookend', 'lastbookend')
confirm_page_res = self._download_webpage_handle(
confirm_url,
video_id, 'Finishing Login')
post_form(confirm_page_res, 'Confirming Login')
just tried it out......assuming your putting conditions code after comcost code but before non comcast code........and its hanging at [Syfy] 2877726: Downloading AuthState Page
I wish I could tell you why yours is hanging. Have you tried using the --print-traffic option?
yup.......I think problem is that is causing you to see something usefull and all I see is the encoded crap after the url like I posted in pastebin above...........your welcome to an account on my vps and see if you can get it working lol
@gkoelln Basically that's the correct workflow. If you have made it working, it's ready for a pull request.
would love for this to get merged but unfortunately I cant give out my sisters login :(
So is this working? How do I get it to work on my end ('Dish' is not on the list of MSO's right now)
There is a commit listed but flag tv provider login needed. So its stalled till they can verify it works. As i said before i cant provide my sisters login credentials........hell i even asked her if i could and she said no.
If you can so they can verify its working it would help me and others be able to use dish.
So all you need is a login credential?
sigh.
I've had a few site support requests get tagged with account needed. When I've asked about supplying one, I've been told that if someone gets to the point of writing code and needing it, I'd be contacted.
For some reason, I'm not even seeing email when a reply comes here -- that one from 9 days ago, I just saw now.
I'd have no problem giving you access. Just let me know when, I'll set up a dummy password for you, and when you've had a chance to test and verify, I'll reset my password back.
My pull request works as far as I can tell. Feel free to test it and let me know if there's a site on which it doesn't work. #12000
Ok, can you tell me how to test it?
Step 1: Clone youtube-dl:
keybounceMBP:youtube-dl michael$ git remote -v origin https://github.com/rg3/youtube-dl.git (fetch) origin https://github.com/rg3/youtube-dl.git (push) keybounceMBP:youtube-dl michael$
Step 2: ??? I have no idea how to bring your pull request in for testing.
Ahh...
{{{ 517 git fetch https://github.com/gkoelln/youtube-dl.git 8a0d6d29cdb85df642c6d5f38cf297a5ba6cfbf2 518 git fetch https://github.com/gkoelln/youtube-dl.git 8a0d6d29 519 git fetch 8a0d6d29 520 git fetch origin 8a0d6d29 521 git fetch https://github.com/gkoelln/youtube-dl.git Dish 522 git checkout FETCH_HEAD }}} Fetching from you, with the commit SHA, does not work. Seeing that the pull request is on rg3, and fetching that SHA from there, does not work. Finding out that it is named "Dish" on your repository, and fetching that, does work.
Now, I have a new repository, with your code merged; how do I run it?
EDIT: Silly me, there's a Makefile.
Run it like you would run the youtube-dl command line, but using "python main.py" instead of "youtube-dl."
Merged the code indicated by the comment "gkoelln commented on Feb 3" into a git clone pulled 2017-07-11 on the master branch. Compiled and executed successfully.
Yep. I've been commenting directly on the pull request #12000
It's been working fine for me for a while. I had a little problem with one site, briefly, but those went away.
Will Dish be added to the main code soon?
Would one of the contributors or developers please volunteer to complete this feature request? I have a dish.com login if needed for testing and debugging. If no one wants to do it, then I request that it be closed and tagged as "won't fix".
Please follow the guide below
x
into all the boxes [ ] relevant to your issue (like that [x])Make sure you are using the latest version: run
youtube-dl --version
and ensure your version is 2017.01.29. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
Description of your issue, suggested solution and other information
Ive been trying to add dish support for past few days on and off. I cant give login creditials out as there my sisters. So im forced to try to get it working on my own :(
Anyway here is as much info as I have.
ive added to adobepadd.py
url of redirect is similar to xfinity as its a redirect not a popup like most of rest
https://sp.auth.adobe.com/adobe-services/authenticate?requestor_id=syfy&no_iframe=false&mso_id=Dish&domain_name=syfy%2Ecom&redirect_url=http%3A%2F%2Fwww%2Esyfy%2Ecom%2Fkilljoys%2Fvideos%2F1%2F2616
which redirect too:
https://identity1.dishnetwork.com/saml/module.php/sociallogin/login.php?AuthState=_e1a33b890e60fbe4ae431ca47b642708b1f18a02c7%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fsaml.sp.auth.adobe.com%26cookieTime%3D1485809534%26RequesterID%3D%255B%2522syfy%2522%255D%26NameIDFormat%3Durn%253Aoasis%253Anames%253Atc%253ASAML%253A2.0%253Anameid-format%253Atransient
on this page there are 3 form elements.....1 of which is remember me button.
Online ID:
<input placeholder="Online ID" class="input" id="username" type="text" name="username" value="" tabindex="1" autocapitalize="off" autocorrect="off"
required="">`Password:
<input
placeholder="Password" class="input" id="password" type="password" name="password" value="" tabindex="2" autocapitalize="off" autocorrect="off"required="">
Remember me check box:
<input
type="checkbox" name="remember_me" value="yes" tabindex="4"checked="checked">
and login button:
<button
id="login" type="submit" name="source_button" value="authsynacor_identity1.dishnetwork.com" class="login" tabindex="3"> Log In</button>
after that I ran make......
issued command
./youtube-dl -v --ap-mso Dish --ap-username PRIVATE --ap-password PRIVATE --list-formats http://www.syfy.com/killjoys/videos/110-escape-velocity
and it hangs at:
[Syfy] 2877726: Downloading Provider Login Page
If I let it sit for like 5 minutes......it eventually gives ssl error time......
Now im running this on a remote vps so dont think its an internet issue but just to be sure.....I tried my login credentials on Direct tv and it errors out invalid credentials.
So Im thinking it just geting to page and timing out......but I cant figure out why.
Any help or steps I can do to resolve this would be greatly appreciated.
here is output up till hang:
[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'--ap-mso', u'Dish', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'--list-formats', u'http://www.syfy.com/killjoys/videos/110-escape-velocity'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2017.01.28 [debug] Python version 2.7.12 - Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial [debug] exe versions: avconv 2.8.10-0ubuntu0.16.04.1, avprobe 2.8.10-0ubuntu0.16.04.1, ffmpeg 2.8.10-0ubuntu0.16.04.1, ffprobe 2.8.10-0ubuntu0.16.04.1, rtmpdump 2.4 [debug] Proxy map: {} [Syfy] 110-escape-velocity: Downloading webpage [Syfy] 2877726: Downloading Provider Redirect Page [Syfy] 2877726: Downloading Provider Login Page ^C ERROR: Interrupted by user
Thanks for your time