Production - [Alerting] DotNetEng Status Failed Requests/Hour alert

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 48

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-d2dd705a6c724ed68fcf6955561c06dd

[!IMPORTANT] This error started on Jun 28th and the error showed is related to Octokit, in order to fix this, we need to rollout octokit with version 13.0, the changes are already in on this pr: https://github.com/dotnet/dnceng/pull/3248, but it hasn't rollout, when that rollout we can close this issue.

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 22

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 24

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 29

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 22

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 30

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 23

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 24

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 32

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 23

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 23

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 21

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 25

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 27

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 30

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 25

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 25

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 25

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 21

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 26

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 27

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 23

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 24

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 24

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 23

Go to rule

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated. To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 28

Go to rule