Production - [Alerting] On-Prem Machines Heartbeating By Queue alert

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

• Percentage Heartbeating {QueueName=raspbian.11.arm32.iot.open} 66.66666666666666

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-d2356d84cf3e43ea952d81de941eaa76

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

Go to rule

[oleksandr-didyk]

Seems to have self-resolved, will continue monitoring

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

• Percentage Heartbeating {QueueName=raspbian.11.arm32.iot} 0

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

• Percentage Heartbeating {QueueName=osx.13.amd64.iphone} 66.66666666666666
• Percentage Heartbeating {QueueName=windows.10.amd64.20h2.owl.perf} 75

Go to rule

[oleksandr-didyk]

Rebooted DNCENGMAC134 in osx.13.amd64.iphone Rebooted PERFOWL010, PERFOWL014 and PERFOWL015 in windows.10.amd64.20h2.owl.perf

Will continue monitoring

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

One or more queues of on-prem Helix machines had a heartbeat rate below 80%. This may indicate a deployment, the addition of new machines that are not yet active, or a systemic problem with the machines.

Go to rule

[ilyas1974]

Systems are online as expected. Closing alert.