search

cobalt-uoft / uoft-scrapers

Public web scraping scripts for the University of Toronto.
https://pypi.python.org/pypi/uoftscrapers
MIT License
48 stars 14 forks source link

Make textbooks scraper utilize threading #27

Closed qasim closed 8 years ago

qasim commented 8 years ago

Check it out! Still need to see if it can be cleaned up more, but this is my implementation of threading for textbooks scraper. Before threading, textbooks scraper took ~50min. After threading, textbooks scraper takes ~5min. Results will probably vary when it's on the server though.

qasim commented 8 years ago

The progress logging: cool

(It only prints to stdout, not the logger).