search

unitedstates / inspectors-general

Collecting reports from Inspectors General across the US federal government.
https://sunlightfoundation.com/blog/2014/11/07/opengov-voices-opening-up-government-reports-through-teamwork-and-open-data/
Creative Commons Zero v1.0 Universal
106 stars 21 forks source link

SBA redesigned their OIG site #187

Closed konklone closed 9 years ago

konklone commented 9 years ago

And the AJAX report feed no longer functions.

Before:

before

After:

after

The sba scraper is basically inert until this is addressed.

spulec commented 9 years ago

I just ran sba locally and it seemed to work. You can still see the ajax calls being used here. From what I can tell, this feed is still being updated.

konklone commented 9 years ago

Interesting. My sba script hangs every time on my laptop, from different IP addresses:

$ ./inspectors/sba.py --debug
POST - https://www.sba.gov/views/ajax
Starting new HTTPS connection (1): www.sba.gov
sleeping for 5 seconds before retry
Starting new HTTPS connection (2): www.sba.gov

But on the server, it works fine. Not sure what's going on. Closing until I have more information to suggest there's a problem outside my local environment.

spulec commented 9 years ago

I forgot to mention that I was having some SSL issues, but they were resolved after upgrading Requests. Maybe that is contributing to the problem.