search

jkeesh / scpd-scraper

download and convert SCPD lectures easily
17 stars 17 forks source link

Update for new SCPD site #17

Closed jlkravitz closed 9 years ago

jlkravitz commented 9 years ago

Now uses selenium, which is needed since the site is made through angular.

jkeesh commented 9 years ago

Already ditching all the lectures?

— Jeremy

On Sat, Nov 29, 2014 at 2:13 PM, Joshua Kravitz notifications@github.com wrote:

Now uses selenium, which is needed since the site is made through angular. You can merge this Pull Request by running: git pull https://github.com/jlkravitz/scpd-scraper master Or you can view, comment on it, or merge it online at: https://github.com/jkeesh/scpd-scraper/pull/17 -- Commit Summary --

  • Add full access script
  • Use clearer url
  • Refactor browser out of course download so we don't need 2-step auth for every course
  • Lots of cleanup (not in working condition)
  • Add phantom and selenium so we can use js
  • Add requirements.txt
  • Start incorporating selenium
  • Better format for specifiying class names
  • Clean up scpd access script
  • Make try/catch have less code
  • Working with selenium
  • Remove js script

  • Update README -- File Changes -- M .gitignore (1) M README.md (71) A package.json (5) A requirements.txt (3) M scrape.py (452) -- Patch Links -- https://github.com/jkeesh/scpd-scraper/pull/17.patch https://github.com/jkeesh/scpd-scraper/pull/17.diff

    Reply to this email directly or view it on GitHub: https://github.com/jkeesh/scpd-scraper/pull/17

jlkravitz commented 9 years ago

It happens

jkeesh commented 9 years ago

cool that this is still being used (or at least forked). ill merge this but dont even have the credentials anymore to test it

jlkravitz commented 9 years ago

yeah...it's kind of a pain to test since you have to log in every time, enter two-step code, etc. But I know the basic usage definitely works.

siilats commented 9 years ago

gotta remove "," in the package.json - correct line is

"chromedriver": "^2.10.0-1"

looks like the server is html5 now. Does anyone have it working? What do you do with the two factor. RIght now it times out for me once I manually type in the 2factor code.

jkeesh commented 9 years ago

I'm not a student any more so I don't use or maintain it. However I do merge in patches people make since it still seems to be used 

— Jeremy

On Sat, Mar 7, 2015 at 7:35 PM, keith siilats notifications@github.com wrote:

gotta remove "," in the package.json - correct line is "chromedriver": "^2.10.0-1"

looks like the server is html5 now. Does anyone have it working? What do you do with the two factor. RIght now it times out for me once I manually type in the 2factor code.

Reply to this email directly or view it on GitHub: https://github.com/jkeesh/scpd-scraper/pull/17#issuecomment-77730933