draft a plan for an api to the raw data

[cacraig]

Plan for Raw Data Expansion

Archive reports once per day into a JSON store. Create an API to return aggregates, and subsets of this data. Archive the data in a datastore (JSON store - RethinkDB, or even Postgres >9.3 with JSON support). Handle aggregations server-side. We would store Agency level data, and perhaps provide parameters in order to allow sub-agencies to access more concise samples of the data (ie. USPTO from the DOC).

Frequency: Daily, and Minute.

API:

Up for debate, but one user asked for top downloads by domain. Which is already satisfied by the agency level dropdowns.

We can expose these reports like such:

HTTP [GET] /api/v1.0/<agency>/<report_name>?sort=…&limit=…&start_date=…&end_date=…

defaults:

sort= asc
limit =100
start_date = 10 days ago
end_date = today

report names:

• pages ( get me the top pages between two dates for some domain.)
• browsers
• os
• devices
• users
• visits or page_views

Tasks:

• Set up JSON datastore app on cloud.gov => RethinkDB is probably decent choice. Minimal writes, and potentially large number of reads. Security of this data isn’t of huge concern (given the data’s open nature).
• Add code to analytics-reporter to write to the datastore whenever a daily report is run.
• Build API’s for access to this archived data. Do we want a new Repo for this, or should we put this in analytics-reporter? Do we need API limiting?

[gbinal]

This is good. The planning phase is :+1: for me. We should take this to new issues for the actual work.