Production - [Alerting] Autoscale: Minutes to scale-up from zero machine alert

[dotnet-eng-status[bot]]

:broken_heart: Metric state changed to alerting

Scale up issue: A queue has been waiting for a machine to scale up for more than 45 minutes, there are no machines in this queue, which could cause a lot of work to get stuck.

Wiki link for investigation and mitigation steps here

• WaitTime {Queue=ubuntu.2204.amd64.open.svc} 50

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-54aa0d7e647e46ff9e880bf6ae532b99

[dotnet-eng-status[bot]]

:green_heart: Metric state changed to ok

Scale up issue: A queue has been waiting for a machine to scale up for more than 45 minutes, there are no machines in this queue, which could cause a lot of work to get stuck.

Wiki link for investigation and mitigation steps here

Go to rule

[riarenas]

ooh, not arm! looking

[riarenas]

The queue doesn't currently have any broken machines to investigate, but we did see a spike in VMs getting deleted by the VM cleaner around this time, which points to a transient problem.