Particular / ServiceControl

Backend for ServiceInsight and ServicePulse
https://docs.particular.net/servicecontrol/
Other
52 stars 47 forks source link

Implement alerts clearing #43

Closed dannycohen closed 11 years ago

dannycohen commented 11 years ago

As Opie, I want to view alerts details and mark an alert as "cleared" after I took corrective actions

Visualizations:

  1. An alert can be set to be "active" or "cleared"
  2. Alert list displays only alerts whose status is "active"
  3. To Clear an alert - select one or more alerts (by checking the checkbox in the first column of each row) and click the "Clear Alerts" button
  4. If no heartbeat alerts remain active for an endpoint, the relevant indicators revers to green
  5. It may be possible that an indicator is green but it has an active alert.
    • See case 2 below which covers the following flow of events:
      1. Heartbeat failure causes the creation of an alert and the indicator is red
      2. The endpoint recuperates after a few minutes and heartbeat messages are received
      3. The previously created alert is still active - indicating there was a failure at some point in the past
      4. The indicator is green - indicating that in the heartbeat messages are received in the the present

Notes:

Demo / Acceptance tests:

Case 1:

  1. Deploy the heartbeat plugin in 3 of the 5 Video Store sample endpoints (in all except the ContentManagement and Operations endpoints)
  2. Run the Video Store sample
  3. Kill the "Sales" endpoint
  4. Indicator should turn red within 1 minute
  5. Number below Heartbeat Indicator should be 2 in green and 1 in red
  6. Click on heartbeat indicator
  7. 3 endpoint heartbeat indicators should be displayed, 2 green, 1 red
  8. The name of each endpoint and the number seconds elapsed since last heartbeat messages was received noted next to each indicator
  9. Click on the endpoint name "Sales"
  10. The alerts list is displayed, filtered to show only the heartbeat alerts for the "Sales" endpoint
  11. Select all the alerts for the "Sales" endpoint
  12. Click the "Clear Alerts" button
  13. The cleared alerts are removed from the alerts list
  14. The endpoint heartbeat indicator for "Sales" endpoint is green (i.e. 3 green indicators) and so is the overall heartbeat indicator (#40)
  15. Since we did not revive the "Sales" endpoint, the "Sales" heartbeat Indicator should turn red within 1 minute (and a new heartbeat alert should be created)

Case 2:

  1. Deploy the heartbeat plugin in 3 of the 5 Video Store sample endpoints (in all except the ContentManagement and Operations endpoints)
  2. Run the Video Store sample
  3. Kill the "Sales" endpoint
  4. Indicator should turn red within 1 minute
  5. Number below Heartbeat Indicator should be 2 in green and 1 in red
  6. Click on heartbeat indicator
  7. 3 endpoint heartbeat indicators should be displayed, 2 green, 1 red
  8. The name of each endpoint and the number seconds elapsed since last heartbeat messages was received noted next to each indicator
  9. Click on the endpoint name "Sales"
  10. The alerts list is displayed, filtered to show only the heartbeat alerts for the "Sales" endpoint
  11. An active heartbeat alerts exists for the "Sales" endpoint
  12. Create a new instance of the "Sales" endpoint
  13. "Sales" endpoint Indicator should turn green within 1 minute (since it starts receiving heartbeat messages from the endpoint)
  14. Next to the "Sales" endpoint indicator there is a small exclamation mark indicating there is an active alert
  15. Click on the endpoint name "Sales"
  16. The alerts list is displayed, filtered to show only the heartbeat alerts for the "Sales" endpoint
  17. The same active heartbeat alerts exists for the "Sales" endpoint (see step 9 above)
  18. Select the alert and click the "Clear Alerts" button
  19. The active alert is removed from the alerts list
  20. In the "Sales" endpoint indicator the small exclamation mark is no longer visible (indicating there are no longer any active alert)
dannycohen commented 11 years ago

Moved to https://github.com/Particular/ServicePulse/issues/13