Closed divergentdave closed 9 years ago
I'm going to look into this more, but it looks like the proper report can be found here.
I have emailed the IG about the 404 and added a temporary, hacky fix with 7c6892138bd63c9c33b938c5254d577e8c244a6c.
Let's leave this open. If the IG responds within a week or so and fixes it, then we can remove the hack. Otherwise, let's discuss some better possibilities.
Here's a list of all the IG websites that don't return 404 response codes when appropriate. The script for this is on my fork at https://github.com/divergentdave/inspectors-general/blob/scripts/check_site_404s.sh.
http://www.cftc.gov/About/Offi... HTTP/1.1 301 Moved Permanently
http://www.cpb.org/oig/doesyou... HTTP/1.1 200 OK
http://www.dodig.mil/doesyour4... HTTP/1.1 302 Redirect
http://www.eac.gov/inspector_g... HTTP/1.1 302 Moved Temporarily
http://www.exim.gov/oig/doesyo... HTTP/1.1 302 Moved Temporarily
http://fcc.gov/oig/doesyour404... HTTP/1.1 301 Moved Permanently
http://www.fec.gov/fecig/doesy... HTTP/1.1 302 Found
http://www.fmc.gov/bureaus_off... HTTP/1.1 302 Object moved
http://www.gpo.gov/oig/doesyou... HTTP/1.1 302 Moved Temporarily
http://gsaig.gov/doesyour404wo... HTTP/1.1 302 Moved Temporarily
http://www.ncua.gov/about/Lead... HTTP/1.1 200 OK
http://www.nrc.gov/doesyour404... HTTP/1.1 302 Moved Temporarily
http://www.peacecorps.gov/abou... HTTP/1.1 301 MOVED PERMANENTLY
http://www.si.edu/OIG/doesyour... HTTP/1.1 200 OK
http://oig.state.gov/doesyour4... HTTP/1.1 302 Moved Temporarily
Oh hey, great idea, @divergentdave, and maybe that's even worth a field on the eventual dashboard.
@spulec I assume the FEC OIG didn't get back to you about the hack?
Correct, no response :/
I guess we'll close it for now then. :/
The first link here points to here, which returns a 302 Found, pointing to here, which is in turn served with 200 OK. Perhaps we should have the fec.py scraper check reports after they have been saved to look for "Page Not Found." More generally, we may need to make errors from the text conversion routines more visible, because we get a CalledProcessError from both pdfinfo and pdftotext with this file.