search

unitedstates / inspectors-general

Collecting reports from Inspectors General across the US federal government.
https://sunlightfoundation.com/blog/2014/11/07/opengov-voices-opening-up-government-reports-through-teamwork-and-open-data/
Creative Commons Zero v1.0 Universal
107 stars 21 forks source link

New State website is missing old reports #185

Closed konklone closed 9 years ago

konklone commented 9 years ago

State dropped a ton of old reports when migrating to a new scraper, as detailed in #180. This can be closed when we've determined how to keep the old reports. Ideally, State will finish importing their reports, as they've said they will.

spulec commented 9 years ago

I reran state this morning and it appears that one of the two missing reports is available again.

Report #228987 is now available as #211870 (AUD-SI-13-32).

We are still missing #162347 ("Audit of Department of State Controls Over Bureau of Diplomatic Security Domestic Firearms and Optics (AUD/SI-11-25)"). From what I can tell, the full report was never released and what we had in the past was just a summary. I still have a copy of the summary. I'll ping state again and see if there is any update on the migration, but I'm not optimistic. We may be better off just adding this as a out-of-band report.

konklone commented 9 years ago

We may be better off just adding this as a out-of-band report.

Let's do that. Got any ideas of the best way to track and store out-of-band reports for the project? Internet Archive is handy, but we still would have to track which items in the bucket are out-of-band.

spulec commented 9 years ago

I think we want the reports and metadata to be tracked by git(/hub) to allow contributors to easily add and edit reports. Tracking data in a VCS isn't ideal, but I worry that anything else will increase the barrier to contribution.

It would probably be too messy to add them to this repo so it may be best to create a separate repository and have it mimic the structure of data/. Maybe add a separate config option to admin.yaml (out_of_band_directory) that will scoop them all up and put them in data.

I don't love it, but it is the best idea I have right now.

konklone commented 9 years ago

Another repository sounds just fine. I made a https://github.com/unitedstates/reports repo, and gave you permissions on it.

Want to create an inspectors-general/ directory in there, with the data structure mirrored as you described?

Also, there's https://github.com/unitedstates/documents, which has a gem that's seen barely any use. That repository could be renamed without causing any breakage to clients pulling the gem down from rubygems. Maybe down the line, if we want to.

spulec commented 9 years ago

Cool, I'll do some work.

I was actually just reading your comment on #192. If I get some time, I may pull those and put them in another directory next to inspectors-general.

spulec commented 9 years ago

Great, I have added the remaining report to that repo with https://github.com/unitedstates/reports/commit/1b034c3210880635c3c7e18703a7ff4c3b2540a5

I'm going to close this out in favor of any work on #204