search

VIAplanner / via-timetable

Our mission is to use cutting-edge technology to build the best timetable planner for the University of Toronto
https://timetable.viaplanner.ca/about
Other
44 stars 12 forks source link

Update Scraper Courses Source #290

Closed TheFatPanda97 closed 1 year ago

TheFatPanda97 commented 3 years ago

Describe the solution you'd like Previously, we relied on UofT Course Finder as our primary source for scraping courses. However, UofT is no longer updating the UTSG courses on this website (or at least very delayed updates).

UofT Course Finder also had other issues such as displaying incorrect UTM sections (eg. Sections that's closed until further notice). It would be ideal to move away from this website.

Describe Possible Solutions

  1. For UTSG courses, we could use this website instead. It provides many benefits such as a publicly accessible API, saving us the trouble of writing a scrapper.
  2. For UTM courses, we unfortunately need to rewrite our current scraper. I wrote a scraper before for the UTM Timetable Planner, but it has some drawbacks such as slow scraping time. We can improve on the scraping time significantly by using the puppeteer-cluster library (similar to the scraper for UofT Course Finder).
  3. UTSC also have their own timetable planner, but I'm not sure what it's called.
jcserv commented 3 years ago

For UTSG, maybe we could leverage this: https://timetable.iit.artsci.utoronto.ca/api/20219/courses?code=eco

jcserv commented 3 years ago

I can get someone on this but it'd have to be starting in August - so if anyone from the OG viaplanner team can tackle this before then that'd be great!

TheFatPanda97 commented 3 years ago

I can get someone on this but it'd have to be starting in August - so if anyone from the OG viaplanner team can tackle this before then that'd be great!

Yep sounds good, but it will probably take me some time 😅

TheFatPanda97 commented 3 years ago

Update: The UTSG Course scraper has been complete.

logonoff commented 1 year ago

no longer relevant as we scrape ttb for tri-campus course data