Make data downloadable as CSV, JSON

themarshallproject / hall-of-justice

Working with criminal justice data.

http://hallofjustice.sunlightfoundation.com/

BSD 3-Clause "New" or "Revised" License

14 stars 7 forks source link

Make data downloadable as CSV, JSON #20

Closed dcloud closed 9 years ago

dcloud commented 9 years ago

Probably best to think of in terms of user downloads (over API). So this should probably include:

search results that can be downloaded as CSV or JSON
category and location-based pages where data can be downloaded as CSV or JSON

Having search-based CSV creation could be transactionally expensive, but perhaps we could optimize PostgreSQL for generating CSV data efficiently.

dcloud commented 9 years ago

When generating a CSV in a celery task, the file pointer location is on the task queue machine, which is a problem.

dcloud commented 9 years ago

Switch search views to new-style view classes as part of this.

dcloud commented 9 years ago

Progress in 3e99ec2...9d37ffe for exporting to CSV from search. Waiting for user research to determine if JSON API is needed/desirabled.

Also, Haystack's new-style view classes are broken in 2.4.0, so no dice there.

dcloud commented 9 years ago

1b817de...19dd056 improve the CSV export using postgreSQL intarray extension to query and order Datasets by list of ids. Better than iterating over searchqueryresult and getting each object individually.

dcloud commented 9 years ago

Now there is a CSVExportMixin for objects in 4f143c208bb9f40960eec420b7587622339ce8a1. Seems like we're good here.