search

MuckRock / muckrock

MuckRock's source code - Please report bugs, issues and feature requests to info@muckrock.com
https://www.muckrock.com
GNU Affero General Public License v3.0
114 stars 22 forks source link

Data Pulls for Hackathon #873

Closed morisy closed 8 years ago

morisy commented 8 years ago

For the BuzzFeed hackathon, we need two types of data pulled. All data pulled should be based on unembargoed requests ONLY.

— A CSV of all request meta data: What was requested, when, from what agency, what jurisdiction, what the outcome was, how many pages the request produced, and any other metadata we can pull easily (# of communications?). This script should get us most of the way there, if not all of the way there. — An export of all public request data, each request stored to a difference folder. Folder names should include request number, request title. Metadata on each request (agency, status) needs to be handled somehow: Maybe organize all those folders into "Completed", "Awaiting Response", etc folders?

It'd also be great if we could have two other CSVs — Stats by agency (average response time, average # of communications, pages released, number of requests, response breakdown by type, anything else? and the same except stats by jurisdiction.

It'd also be super, super great if we could have an example FOIA Filing API script. Theoretically this exists in the API, but I don't think it's ever been used or even had an example script coded.

Would be great if we could take on FOIA the Dead as a demo.

mitchelljkotler commented 8 years ago

For the folders, is the output format of this script https://github.com/MuckRock/API-examples/blob/master/export_as_files.py acceptable?

morisy commented 8 years ago

That should be fine. Forwarded them this issue so if they have other ideas, they can let us know but definitely good starting point. On Mon, Apr 4, 2016 at 9:41 PM mitchelljkotler notifications@github.com wrote:

For the folders, is the output format of this script https://github.com/MuckRock/API-examples/blob/master/export_as_files.py acceptable?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/MuckRock/muckrock/issues/873#issuecomment-205586017

morisy commented 8 years ago

Maybe the CSV should have a column that corresponds to the folder names for easy cross reference? On Mon, Apr 4, 2016 at 9:48 PM Michael Morisy m@morisy.com wrote:

That should be fine. Forwarded them this issue so if they have other ideas, they can let us know but definitely good starting point. On Mon, Apr 4, 2016 at 9:41 PM mitchelljkotler notifications@github.com wrote:

For the folders, is the output format of this script https://github.com/MuckRock/API-examples/blob/master/export_as_files.py acceptable?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/MuckRock/muckrock/issues/873#issuecomment-205586017

allanlasser commented 8 years ago

Do we know API filing works if we've never used it?

morisy commented 8 years ago

We don't, one of the reasons we want a script.