EVM Monitoring and Alerting - Phase 2

eosnetworkfoundation / engineering

A workspace for documentation by Engineering primarily regarding process

MIT License

0 stars 0 forks source link

EVM Monitoring and Alerting - Phase 2 #73

Closed wanderingbort closed 4 months ago

wanderingbort commented 1 year ago

Place holder for future items

Needs to wait for phase 1 to be used and feedback prepared

### Tasks

kj4ezj commented 12 months ago

While responding to an alert setup during phase one of this project, the customers shared a graph of CPU utilization and explained the instance needs to be restarted.

This suggests to me that CPU utilization and instance uptime or restarts might be good metrics to plan for dashboarding and/or alerting in phase 2.

kj4ezj commented 11 months ago

We will need to provide the customer some way to declare a maintenance period during which alarms should be suppressed.

2023-10-24 15-36-33 - EVM alarms during maintenance period

kj4ezj commented 10 months ago

Notes on customer feedback during the EVM video call:

Eight hour time limit on the maintenance window(s) during which alarms will be suppressed.
A mechanism to automatically restart nodes when health checks have been failing for a specified duration.

wanderingbort commented 4 months ago

Blocked by infrastructure transition to EOS Labs

wanderingbort commented 4 months ago

Closing as waiting for something to do