search

18F / fbopen

[DEPRECATED] An open API server, data import tools, and sample apps to help small businesses search for opportunities to work with the U.S. government.
Other
101 stars 45 forks source link

dodsbir scraper is breaking #167

Open arowla opened 9 years ago

arowla commented 9 years ago

It gets a long way through the scrape and then fails like so:

Starting dodsibr-nightly Fri Jan  2 11:18:15 EST 2015
FBOPEN_URI = http://localhost:9200
FBOPEN_INDEX = fbopen
Scraping topics JSON...
Traceback (most recent call last):
  File "scrape.py", line 7, in <module>
    s.get_all_topics() #takes one second per topic (roughly 90 seconds total)
  File "/Users/alisonrowland/.virtualenvs/fbopen/lib/python3.4/site-packages/dodsbir/scrape.py", line 130, in get_all_topics
    topic = self.get_topic(key)
  File "/Users/alisonrowland/.virtualenvs/fbopen/lib/python3.4/site-packages/dodsbir/scrape.py", line 114, in get_topic
    topic = self.html_to_topic(resp.text, topic_id)
  File "/Users/alisonrowland/.virtualenvs/fbopen/lib/python3.4/site-packages/dodsbir/scrape.py", line 78, in html_to_topic
    topic.acquisition_program = rows[0].findAll('td')[1].contents[0].string
IndexError: list index out of range
DavidEBest commented 9 years ago

This is corrected in PR 18F/dodsbir-scrape#9.