Add appropriate monitoring and alerting for Via HTML

hypothesis / via

Proxies third-party PDF files and HTML pages with the Hypothesis client embedded, so you can annotate them

https://via.hypothes.is/

BSD 2-Clause "Simplified" License

21 stars 7 forks source link

Add appropriate monitoring and alerting for Via HTML #246

Closed jon-betts closed 4 years ago

jon-betts commented 4 years ago

We should decide and implement whatever monitoring and logging we would expect for the Via HTML component.

This should:

Match the alerting and monitoring capability of existing Via
Ideally provide hints about getting the right sizes for deployment (how many instances we might need etc)

Tasks:

[x] Add to Papertrail with groups
[x] Add metrics to New Relic (review pending)
[x] Add basic alerting on metrics in New Relic
[x] Add pingdom coverage (or something similar)

indigobravo commented 4 years ago

Due to be working on this the week starting: 2020-09-07

indigobravo commented 4 years ago

Deployment work has basically been completed. However, there are some updates that need to be made to the application before we can confirm the correct monitoring is in place. Until that work is complete this ticket is blocked.

jon-betts commented 4 years ago

Looks like we are missing newrelic agent

indigobravo commented 4 years ago

I have started to put together the ViaHTML - Operational Dashboard. As it stands we only have the infrastructure agent reporting, so that means we aren't capturing application statics.

We have infra monitoring for:

High CPU
High Load Average
High Memory Consumption
High Disk Usage

indigobravo commented 4 years ago

Further work has been done here and also tested.

Elastic Beanstalk autoscaling has been configured based on CPU usage. These settings will need tweaking as we go live. As it stands they are set to:

Scale-out / increased node count when CPU AVG goes above 60% for 4 minutes.
Scale-In / decrease node count when CPU AVG drops below 30%.

We have a NewRelic ViaHTML Status - Scripted check that checks the /_status endpoint and sends notifications to Slack if it is not available.

We are using all of our Pingdom checks. So I can not configure anymore without either deleting an existing, or upgrading our account. I feel the NewRelic check provides the same function.

jon-betts commented 4 years ago

Sean has given this a look over and thinks it's ok

indigobravo commented 4 years ago

ViaHTML incidents raised in NewRelic will now trigger a PagerDuty notification.