Library of Congress scraper

unitedstates / inspectors-general

Collecting reports from Inspectors General across the US federal government.

https://sunlightfoundation.com/blog/2014/11/07/opengov-voices-opening-up-government-reports-through-teamwork-and-open-data/

Creative Commons Zero v1.0 Universal

107 stars 21 forks source link

Library of Congress scraper #145

Closed audiodude closed 10 years ago

audiodude commented 10 years ago

Just claimed for now.

audiodude commented 10 years ago

I don't know what went wrong with this. Probably when I did git rebase master on the loc branch after a git pull origin master on my master branch. You'd think I'd know how to use git by now.

Should I just nuke this and start a new branch and a new PR?

Also, how do you deal with 404s?

@konklone @spulec

spulec commented 10 years ago

My normal strategy for 404s:

Try to see if it is just a typo and you can map to the real url.
If there are just a few, hardcode a blacklist and add a note under the 'Notes for IG's web team' section. Then we can reach out to the IG and ask them to fix the links.
If there are a lot of 404s, you can use the missing param. We've only done this for a couple of reports.

konklone commented 10 years ago

Works great! I fixed the branch (a git merge master did the trick), fixed the pyflakes error that broke Travis (the os module was imported but not used) and made inspectors/loc.py an executable file.

Thanks for doing one of our final scrapers, @audiodude - and nice work!