dotnet / dnceng

.NET Engineering Services
MIT License
24 stars 19 forks source link

Add VM cleaner panel to Grafana #3597

Open riarenas opened 4 months ago

riarenas commented 4 months ago

We should integrate the telemetry from the dead vm cleaner into our grafana dashboards. We can use @AlitzelMendez's query that checks for machines we are just about to clean up as a starting point:

traces
| where cloud_RoleName == "fabric:/Helix/DeadVMCleaner"
| where message contains "exists but has lacked a heartbeat for 2 checks"
| extend queueId_ = tostring(customDimensions.queueId)
| summarize count() by queueId_, bin(timestamp, 1d)

Release Note Category

garath commented 4 months ago

Just FYI, there are some state already at the "VM Cleaner" row of the "Backend Status" dashboard.

Splits by queue will be very good addition!