search

yoonwaiyan / stackskills-dl

Simple Ruby script to download all StackSkills tutorials.
https://yoonwaiyan.github.io/stackskills-dl/
MIT License
59 stars 19 forks source link

Not working #51

Closed IcarianOM closed 3 years ago

IcarianOM commented 4 years ago

Looks like something broke, exit xxxxxxx@yyyyyy:~/stackskills-dl-master$ ruby stackskills_dl.rb -c https://stackskills.com/courses/enrolled/36717 Login Email: xxxxxxxxx@gmail.com Login password: ***** Traceback (most recent call last): 10: from stackskills_dl.rb:11:in <main>' 9: from /home/icarian/stackskills-dl-master/lib/course_finder.rb:11:inrun' 8: from /home/icarian/stackskills-dl-master/lib/utilities.rb:12:in mkchdir' 7: from /home/icarian/stackskills-dl-master/lib/utilities.rb:12:inchdir' 6: from /home/icarian/stackskills-dl-master/lib/utilities.rb:13:in block in mkchdir' 5: from /home/icarian/stackskills-dl-master/lib/course_finder.rb:12:inblock in run' 4: from /home/icarian/stackskills-dl-master/lib/course_finder.rb:28:in execute' 3: from /var/lib/gems/2.5.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:inget' 2: from /var/lib/gems/2.5.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:321:in fetch' 1: from /var/lib/gems/2.5.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:998:inresponse_redirect' /var/lib/gems/2.5.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:329:in `fetch': 503 => Net::HTTPServiceUnavailable for https://sso.teachable.com/secure/1453/users/sign_in?clean_login=true&reset_purchase_session=1 -- unhandled response (Mechanize::ResponseCodeError)

yoonwaiyan commented 4 years ago

Seems like StackSkills is having problem with their server. Please retry after the server is back online and let me know if you're still having issue downloading courses.

andrewiassur commented 4 years ago

/var/lib/gems/2.5.0/gems/mechanize-2.7.4/lib/mechanize/http/agent.rb:316:in `fetch': 503 => Net::HTTPServiceUnavailable for https://sso.teachable.com/secure/1453/users/sign_in?clean_login=true&reset_purchase_session=1 -- unhandled response (Mechanize::ResponseCodeError)

i am having the same issue i just wanted to make sure it was their end. i tried on several machines and the same problem.

yoonwaiyan commented 4 years ago

Seems like they have changed the site structure and the login page is affected. I'll evaluate and fix the login issue. Thanks for reporting!

andrewiassur commented 4 years ago

i dont know if this is relevant but i have the same issue when i tried using youtube-dl

andrewiassur commented 4 years ago

would cloudflare-scrape be able to help someone mentioned it on youtube-dl git hub.

yoonwaiyan commented 4 years ago

@andrewiassur Thanks, I'll take a look at it.

marens101 commented 4 years ago

Any updates? I've got the same bug

yoonwaiyan commented 4 years ago

as @andrewiassur mentioned it's related to Cloudflare's DDOS protection that the login page is not accessible. I've tried on my side by changing user agents and did some research but yields no result.

Due to my limitation of knowledge on this issue I hope someone could help on it. Please feel free to submit a PR if any of you managed to make it work.

andrewiassur commented 4 years ago

@yoonwaiyan i know nothing of ruby. from looking at the cloudflare-scrape would have to call with with the website. i not exactly sure how to do i tried playing around with the code to see if i could get it to do any thing. i was able to find some kind of way to call python in ruby thats about it like i said i'm not really a programmer. i only really write bash scripts autohotkey and a little java and very little c++ and python.

andrewiassur commented 4 years ago

this is working flawlessly

andrewiassur commented 4 years ago

had to leave the location i was at and went to home to start the download again. still same issue. i have no idea why it worked. i was connect from a vpn. so i tried again from a vpn still the 503 error. when it did work from the other location, i noticed that it had a long lag time from when i typed in the credentials til when it started to download the files.

andrewiassur commented 4 years ago

this github project does work but not what is hosted on git hub you have to donate $1.36 for it and download it from the second link.

merberich commented 3 years ago

I think I found a different way to handle sign-on... the dl_coursera project ended up avoiding sign-on altogether by having the user supply a cookies file that has the active session cookies. Loading the cookies file into a session object and then using that object to make the requests works. My equivalent project for teachable does the same thing (to great success), but isn't set up to handle StackSkills' web format. Using teachable-dl I was able to get through login on StackSkills though, so I think this method would work for stackskills-dl as well.

yoonwaiyan commented 3 years ago

@merberich thanks for the suggestion, I'll take a look at it.

flatlinebb commented 3 years ago

I am able to download from stackskills.com using just plain youtube-dl with a cookes.txt file. The limitation is that I have to copy all of the video URLs into a text file and then direct youtube-dl to that text file in the options, as youtube-dl is not able to download the whole course from the enrollment URL. If we could get the cookies.txt method to work for authentication, hopefully the rest will work. I appreciate your time working on these issues. Let me know how I can help further.

yoonwaiyan commented 3 years ago

Sorry to keep you all waiting. I just picked up this again lately and tried to use cookies but I've failed to access the page. It's fair that they tightened up the security of their pages to prevent scraping or DDOS in general. I'm archiving this script.