Archive reports once per day into a JSON store. Create an API to return aggregates, and subsets of this data. Archive the data in a datastore (JSON store - RethinkDB, or even Postgres >9.3 with JSON support). Handle aggregations server-side. We would store Agency level data, and perhaps provide parameters in order to allow sub-agencies to access more concise samples of the data (ie. USPTO from the DOC).
Frequency: Daily, and Minute.
API:
Up for debate, but one user asked for top downloads by domain. Which is already satisfied by the agency level dropdowns.
sort= asc
limit =100
start_date = 10 days ago
end_date = today
report names:
pages ( get me the top pages between two dates for some domain.)
browsers
os
devices
users
visits or page_views
Tasks:
Set up JSON datastore app on cloud.gov => RethinkDB is probably decent choice. Minimal writes, and potentially large number of reads. Security of this data isn’t of huge concern (given the data’s open nature).
Add code to analytics-reporter to write to the datastore whenever a daily report is run.
Build API’s for access to this archived data. Do we want a new Repo for this, or should we put this in analytics-reporter? Do we need API limiting?
Plan for Raw Data Expansion
Archive reports once per day into a JSON store. Create an API to return aggregates, and subsets of this data. Archive the data in a datastore (JSON store - RethinkDB, or even Postgres >9.3 with JSON support). Handle aggregations server-side. We would store Agency level data, and perhaps provide parameters in order to allow sub-agencies to access more concise samples of the data (ie. USPTO from the DOC).
Frequency: Daily, and Minute.
API:
Up for debate, but one user asked for top downloads by domain. Which is already satisfied by the agency level dropdowns.
We can expose these reports like such:
HTTP [GET] /api/v1.0/
<agency>/<report_name>
?sort=…&limit=…&start_date=…&end_date=…defaults:
report names:
Tasks: