Implement Heartbeat indicator

[andreasohlund]

As Opie, I want to be aware of endpoints that are not emitting heartbeats in a timely manner

Visualization:

1. Indicator should be green if no endpoints are failing
   • Definition of endpoint failure: no heartbeat messages were received from endpoint during the past 1 minute
   • Number of heartbeat emitting endpoints should be displayed in green under the indicator icon
2. Indicator should be red if at least one endpoint is failing    * Number of heartbeat failing endpoints should be displayed in red under the indicator icon (and next to the healthy heartbeats)

Notes:

• An endpoint can be set to be "active" or "inactive"; Heartbeat indicator applies only to endpoint whose status is "active"

Demo / Acceptance tests:

Case 1:

1. Deploy the heartbeat plugin in 3 of the 5 Video Store sample endpoints (in all except the ContentManagement and Operations endpoints)
2. Run the Video Store sample
3. Wait for 1 minute while heartbeat messages are sent and processed
4. Heartbeat indicator should be green and the number 3 should be displayed

Case 2:

1. Kill the Sales endpoint
2. Indicator should turn red within 1 minute
3. Number below Heartbeat Indicator should be 2 in green and 1 in red

Case 3:

1. Restart Sales endpoint
2. Within 1 minute the heartbeat indicator should be green and the number 3 should be displayed

Case 4:

1. Deploy the heartbeat plugin in the Operations endpoint and restart it
2. Within 1 minute the heartbeat indicator should be green and the number 4 should be displayed

[johnsimons]

Nice, I can work on this tomorrow

On Monday, September 2, 2013, Andreas Öhlund wrote:

VIP

Visualization:

-

Indicator should be green if no endpoints are failing

Indicator should be red if at least one endpoint is failing

Design:

Demo / Acceptance tests

Case 1:

1. Run the videostore sample
2. Heartbeat indicator should be green and the number 4 should be displayed

Case 2:

1. Kill the sales endpoint
2. Indicator should turn red within X seconds
3. Number below inidcator should be 1

Case 3:

1. Restart sales endpoint
2. Indicator should turn green

Related to Epic GH link

— Reply to this email directly or view it on GitHubhttps://github.com/Particular/ServiceControl/issues/40 .

Regards John Simons NServiceBus

[dannycohen]

Comment regarding 1 minute timing:

• For demo purposes we can have a much shorter interval (e.g. 10 seconds timeout for receiving a heartbeat message, and a setting per endpoint to send a heartbeat message every 5 seconds),
• I doubt we would like to have heartbeat messages being sent in production at short intervals (such as every 5 seconds etc.)
• Lets leave it as TBD or configurable for further user feedback

[andreasohlund]

I like the "active/inactive" => user can use that to mute endpoints that are known to be down and we can use that our self to turn the active on and off when we start to monitor endpoints starting/stopping for legitimate reasons

[indualagarsamy]

@dannycohen If you are ok with the current display for heartbeats, i.e. the text displays 'n' active endpoints, and a circle at the top indicates the number of failed endpoints, then this issue can be closed. @andreasohlund - Since I am using a template that allows the button with the circle, I chose to display the number in a circle on the indicator itself as opposed to * Number of heartbeat failing endpoints should be displayed in red under the indicator icon (and next to the healthy heartbeats)

[dannycohen]

@indualagarsamy - current implementation is good. I'm missing the "location" aspect of the indicators / dashboard grid (i.e. which endpoints ? on which machines ? etc.) but that is something that is more relevant to #41 and #42

[indualagarsamy]

also i believe some of these user stories need to be in service pulse. The indicator being red or green is a service pulse concern, not service control. @dannycohen @andreasohlund @johnsimons - thoughts?

[dannycohen]

@indualagarsamy - Agreed. I'll close and move them to SP repo.

[indualagarsamy]

@andreasohlund - I believe ServiceControl has needed stuff for the heartbeat indicator. I am closing this issue. Feel free to reopen if you think we are still missing something in SC for the heartbeats feature.