search

cobalt-uoft / uoft-scrapers

Public web scraping scripts for the University of Toronto.
https://pypi.python.org/pypi/uoftscrapers
MIT License
48 stars 14 forks source link

Patch hang on book list retrieval #75

Closed kashav closed 8 years ago

kashav commented 8 years ago

Scraper hangs when request response is None.

Stack trace:

Exception in thread Thread-89:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/Users/kashavmadan/Desktop/cobalt/uoft-scrapers/uoftscrapers/scrapers/textbooks/__init__.py", line 387, in run
    books = Textbooks.retrieve_books(section)
  File "/Users/kashavmadan/Desktop/cobalt/uoft-scrapers/uoftscrapers/scrapers/textbooks/__init__.py", line 229, in retrieve_books
    soup = BeautifulSoup(xml, "html.parser")
  File "/usr/local/lib/python3.5/site-packages/bs4/__init__.py", line 176, in __init__
    elif len(markup) <= 256:
TypeError: object of type 'NoneType' has no len()