Production - [Alerting] Apple simulator failure rate alert

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-137} 100

Go to rule

@dotnet/dnceng, @dotnet/prodconsvcs, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-36d07fceeaf0472b804d8358b2198eac

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-137} 83
• FailureRate {Machine=dci-mac-build-142} 83

Go to rule

[andriipatsula]

Machine dci-mac-build-137 in osx.1200.amd64.open has been disabled Machine dci-mac-build-142 in osx.1200.amd64.open has been disabled

[andriipatsula]

~Machines are not available. Cannot ping them~ I can ping dci-mac-build-137 and cannot dci-mac-build-142

[andriipatsula]

logged in to dci-mac-build-137 via KVM and see "you computed was restarted because of a problem". Performed the from Description and instructions for this alert and rebooted.

[andriipatsula]

Machine dci-mac-build-137 in osx.1200.amd64.open has been enabled

[andriipatsula]

dci-mac-build-142 it seems the IP listed in OsobInventory differs from the real one.

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-142} 83

Go to rule

[andriipatsula]

reassigning to @oleksandr-didyk

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-142} 83
• FailureRate {Machine=dci-mac-build-143} 85

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-135} 100

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100
• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100
• FailureRate {Machine=dci-mac-build-136} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-134} 100

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-141} 100

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-141} 100

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-148} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-148} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-148} 83

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-104} 100

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

• FailureRate {Machine=dci-mac-build-135} 100

Go to rule

[MilenaHristova]

I reset the simulators on dci-mac-build-135. Closing this issue as it's too long, if there is something left to handle a new one will pop up