Spec out technical metrics we'd like to track

[Lsquared13]

As a product stakeholder, I want to be able to retrieve business-relevant data about our customers. As a service operator, I want to be able to retrieve metrics regarding our technical performance.

These concerns both require us to set up a system of metrics and logging. In general, we want to instrument customer operations and errors. This issue is for specifying

• What business metrics we want to track
• What technical metrics we want to track
• A high-level implementation plan, in particular what tools and services we will use and what we want the "output" to look like

[Lsquared13]

For technical metrics we may want to set up a Cloudwatch dashboard. Lambda functions also have their own dashboards, should be able to pick some of those graphs to import on our dashboard.

For business metrics we have a few options.

Data Warehouses and Data Lakes are the two models of service built for business analytics. Data Warehouses are for heavily processed data that we want to be able to do fast and complex queries on. Data Lakes are more free-form and lower cost.

Depending on exactly what we want it might not make sense to use those products. We could roll something ourselves using more basic services if what we want is simple enough.

Finally, we could purchase a third party solution, for example: https://amplitude.com

[john-osullivan]

We ended up deciding to go with Amplitude, and there's now a new issue for collecting the set of business metrics we want to track (). I've updated the title of this issue to reflect that its scope has been narrowed to technical metrics, rather than product/business ones.