Improve monitoring infrastructure

[tomusher]

We currently have lots of different monitoring and alerting tools that are either too noisy, or too quiet.

We should simplify the platforms we use and come up with a simple monitoring infrastructure.

• [ ] Inventory and document all our monitoring infra
• [ ] get access to all monitoring infra

[jamilasnell]

@tomusher Can you elaborate on this ticket? What exactly should we be focusing on for this issue?

[tomusher]

We need to:

• Perform an inventory of the existing monitoring infrastructure
• Review which, if any, we can shut down to attempt to consolidate on fewer monitoring platforms
• Ensure we have a documented approach to monitoring and alerting for downtime.

[tomusher]

Currently we have:

• Scalyr for log aggregation (being migrated to Papertrail)
• Splunk On-Call for on-call paging
• StatusGator for monitoring dependent services
• Healthchecks.io for cron monitoring
• AWS Cloudwatch for health checks

I have also found reference to (but not logins for):

• Pingdom
• Logentries

Questions to be discussed:

• Do we need/use a paging service?
• Are StatusGator alerts useful?
• What sort of monitoring do we need? Do we want specific health checks, and does what we have cover those requirements? Do we just need uptime monitoring?

[tomusher]

First action taken here was to clean up some of the noise we get via Statusgator.

I've unsubscribed from all notifications channels on Statusgator except for Github, changing the alert status to only notify us when it's down, not when there are warnings.

[jencohoon]

Issue is 2 years old. And it looks like this has been completed.