Improve automated nightly error reporting

bricerisingalgorand commented 3 years ago

Summary

We need reporting on the information that is collected nightly.

Scope/Requirements

This will be a script that collects information about the daily pipelines and errors in telemetry and send them to slack. It will be put in a docker image and run daily using our task framework.

Telemetry - make the errors more generic so we can collect/count them https://ci.algorand.network/job/Network%20Report/569/console
Rel nightly build (travis) - what specifically failed, specific errors
Perf dashboard (datadog) - especially significant perf drops

Urgency/Relative Priority

We want to make sure this reporting is more visible. It would be a convenient nice to have so that we have better observability on what's going on in our daily testing.

winder commented 3 years ago

This is the Jenkins task that sends out the nightly network status email which was informing the meatware system. A Jenkins slack plugin would get us most of the way towards an Evan bot.

I'm surprised this is a low priority task, it's pretty important that the dev team be aware of network issues.

algojack commented 3 years ago

We are thinking slack apps? but open to other ideas

Parts: Slack Bot

needs to run daily in the morning (ET) (Daily M-Sun)

Lacework

Already going to #devops-alerts
Needs to be scanned and added to nightly report by bot
(stretch goal) pagerduty critical alerts in another ticket

CI/CD

Focus on CircleCI
Need alerts with details to go to #devops-alerts
Needs to be scanned and added to nightly report by bot
(stretch goal) use test insights to get exact test failures (possible?)

Perf Tests

Already going to to #devops-alerts
(Optional) Can go and look into datadog instead of slack
Bot needs to scan #devops-alerts and put it in report (with links?)

Network Report

hide/don't display/send when there are no errors (only display on report)
Make an automated nightly test to see if it works (so we know when it doesn't)
Bot scans #devops-alerts and adds
Number of hosts that have the error
Feature: Add number of hosts effected/Organize it by host/Summarize it "better"

algorand / go-algorand