Queens-Hacks / qcumber-scraper

Scrapes SOLUS and generates structured data
3 stars 6 forks source link

FIxed scraper for 2016 SOLUS changes #32

Closed MaxBittker closed 8 years ago

MaxBittker commented 8 years ago

-One of the link ID regexes changed -The format of course components changed, it is not always the last element of the list -SOLUS has a new disambiguation page between courses and the course itself for cases that 1 course is offered in multiple "careers" aka distance vs Bader vs main campus. My solution attempts to select the main campus option and ignore the others to avoid front-end changes for now.

Let me know if there are any stylistic or code review objections and I'll happily fix up the PR :) Completed a full shallow scrape in 3 hrs with this.

mystor commented 8 years ago

:+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1:

Awesome!

Graham42 commented 8 years ago

Thanks for working on this @MaxBittker :+1:

mystor commented 8 years ago

LGTM :+1: