search

Particular / ServiceControl

Backend for ServiceInsight and ServicePulse
https://docs.particular.net/servicecontrol/
Other
52 stars 47 forks source link

Implement custom checks & corresponding alerts #45

Closed dannycohen closed 11 years ago

dannycohen commented 11 years ago

As Opie, I would like to check application-specific resources, and raise alerts when these resources fail

Visualization:

  1. Based on Opie's requests, Dave adds application-specific resource checks by implementing a heartbeat provider (see #24 - and here)
  2. When resource checks indicate a problem, an alert is raised with the relevant details (specifically: alert type and free text alert information)
  3. Opie sees the alerts in the alerts list
  4. Opie is able to clear these alerts (like all other alerts)

Notes:

Demo / Acceptance tests:

Case 1:

  1. In the Video Store sample, add a custom check to the heartbeat provider (e.g. check if the "resources" directory exists)
  2. Make sure the "resources" directory exists
  3. Run the Video Store sample
  4. No custom alerts should be raised for a missing "resources" directory
  5. Delete the "resources" directory
  6. Within 1 minute, an alert appears in the alerts list indicating that the "resources" directory does not exist
  7. Wait for 2 more minutes; no additional alerts appear
  8. Select the alert, and clear it by clicking the "Clear Alerts" button
  9. Within 1 minute, a new alert appears in the alerts list indicating that the "resources" directory does not exist
  10. Re-create the "resources" directory
  11. The existing alert indicating that the "resources" directory did not exist, remains visible in the alerts list
  12. Select the alert, and clear it by clicking the "Clear Alerts" button
  13. No additional alerts appear
johnsimons commented 11 years ago

Opie adds application-specific resource checks by implementing a heartbeat provider (see #24 - and here)

This is not true! It is not the OPs responsibility to implement a heartbeat, that responsibility is with the system developer. An OPs person would not know what checks to perform as they would not be familiar with the internals of the system.

dannycohen commented 11 years ago

@johnsimons - Agreed. Updated accordingly to Dave.

dannycohen commented 11 years ago

See "custom checks" specific wireframes: https://particular.mybalsamiq.com/projects/operations/34.%20Dashboard%20-%20Custom%20Alerts

indualagarsamy commented 11 years ago

Added support for periodic checks. https://github.com/Particular/ServiceControl/commit/d01a68f453a091ef4c3e9a8d31e54767912d2c9c

indualagarsamy commented 11 years ago

@dannycohen - I believe we need a user story in ServicePulse that talks about the custom checks indicator, which I believe is done. Also please separate the acknowledgement / clearing the alerts into a separate unit of work. Since ServiceControl has the necessary API support for Custom Checks,.I am closing this one in ServiceControl.

dannycohen commented 11 years ago

@indualagarsamy -

I believe we need a user story in ServicePulse that talks about the custom checks indicator

This GH issue should have been opened in SP. Water under the bridge.

I believe is done

I agree.

Also please separate the acknowledgement / clearing the alerts into a separate unit of work

Not sure I understand your meaning. Do you mean https://github.com/Particular/ServicePulse/issues/13 ?