search

NYCPlanning / labs-infrastructure

Server infrastructure setup code
Other
2 stars 5 forks source link

Test Prometheus & Grafana as monitoring and alerting solution #76

Open bfreeds opened 4 years ago

bfreeds commented 4 years ago

Our apps currently have an inconsistent mixture of monitoring tools, which means when troubleshooting an app to diagnose bugs or performance we have to switch between contexts.

To make an informed decision about a monitoring, logging, and alerting solution, we should test Prometheus & Grafana and analyze this solution vs. New Relic.

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit

Prometheus has wide community adoption, is open source, and is supported by the Cloud Native Computing Foundation. Read here for a good intro to prometheus & grafana.

Grafana

Grafana is the open source analytics and monitoring solution for every database Live demo of grafana dashboard

Current Monitoring

Heroku add-ons for ceqr-app: image

Heroku add-ons for zap-api: image

The paid plan for ceqr-app's New Relic was a stop-gap to address immediate debugging needs, however the free plan through Heroku provides the following functionality: image

New Relic APM was installed on zap-api to address an immediate need (it was easy to add through Heroku and would be consistent with ceqr-app's setup.

For a long-term, holistic approach to maintenance and operations for Labs' portfolio, and to make smoother to troubleshoot and manage apps, we should have consistent:

  1. Monitoring
  2. Logging
  3. Alerting
allthesignals commented 4 years ago

My concern is with installing these things if they're not already available as add-ons there is a prometheus build pack https://github.com/heroku/heroku-buildpack-prometheus

so we'll have to give it a try, but doesn't look maintained . prometheus installation instructions https://prometheus.io/docs/prometheus/latest/installation/