search

scoutapp / roadmap

The public roadmap for Scout application monitoring.
https://scoutapp.com
16 stars 2 forks source link

Slack bot: checking the health of deploys #2

Closed itsderek23 closed 6 years ago

itsderek23 commented 7 years ago

Watching monitoring data after a deploy can be as boring. An automated health check that kicks off following a deploy, surfacing any issues, would address this.

Basic idea:

  1. Following a deploy, we begin checking the health of your app every 3 minutes for 30 minutes total.
  2. If we notice abnormal behavior (ex: a spike in response time, an endpoint that suddenly jumps in % time consumed, etc) we'll post a single message to Slack. If you've integrated Scout w/GitHub, we'll mention the developers behind the latest deploy.

This would leverage some of the logic in #1.

Inspired by the GitHub Haystack bot.

itsderek23 commented 7 years ago

We ran an experiment for this, collecting data from a number of apps.

Based on the sample size, only 1% of deploys showed a visible change in app health following a deploy. This would be a high signal-to-noise ratio for a chat bot - for every 100 "deploy ok" messages, there would be just 1 "deploy bad" message.