search

dotnet / dnceng

.NET Engineering Services
MIT License
24 stars 18 forks source link

Auto scale configuration is not cleaned up when Helix queue is deleted #1425

Open riarenas opened 10 months ago

riarenas commented 10 months ago

The reunion and xaml helix queues were removed in https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/32886 on August 1st.

Since then, the autoscaler has been logging errors of the form:

The Resource 'Microsoft.Compute/virtualMachineScaleSets/windows.amd64.server2022.open.reunion-a-scaleset' under resource group 'windows.amd64.server2022.open.reunion-westus2-v2-rg' was not found.

This happens because the blob container that the auto scaler reads from still has configuration files for these deleted queues.

image

We should make it so that deleting a queue in the helix-machines YAML also cleans up the auto scaler resources.

Release Note Category

riarenas commented 10 months ago

I removed the files for the reunion and xaml queues so that we clean up the exception logging a bit.

riarenas commented 10 months ago

I also cleaned up the ubuntu 1604 config files. These queues were removed in https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/34407

garath commented 4 months ago

Since these config files are re-deployed with each rollout, I'm deleting any that are older than today (May 1, the latest rollout). I confirmed that unmonitored queues still get this update, so they are safe.

Here is a list of what will be deleted:

az storage blob list --account-name helixosconfig --container-name configuration --query "sort_by([?properties.lastModified < ``2024-05-01``].{Name:name, LastModified:properties.lastModified}, &Name)" --output table
Name LastModified
debian.9.amd64.arcade.zip 2022-06-29T15:26:59+00:00
debian.9.amd64.open.arcade.zip 2022-06-29T15:26:59+00:00
debian.9.amd64.open.rt.zip 2022-06-29T15:27:42+00:00
debian.9.amd64.open.svc.zip 2022-06-29T15:28:56+00:00
debian.9.amd64.open.zip 2022-06-29T15:20:52+00:00
debian.9.amd64.rt.zip 2022-06-29T15:27:40+00:00
debian.9.amd64.svc.zip 2022-06-29T15:28:56+00:00
debian.9.amd64.zip 2022-06-29T15:20:52+00:00
osx.1015.amd64.appletv.open.zip 2023-08-02T20:45:01+00:00
osx.1015.amd64.appletv.zip 2023-08-02T20:45:01+00:00
osx.1015.amd64.iphone.open.zip 2023-06-07T00:58:45+00:00
osx.1015.amd64.iphone.zip 2023-06-07T00:58:45+00:00
osx.1015.amd64.open.zip 2023-08-02T20:45:01+00:00
osx.1100.amd64.scouting.open.zip 2022-05-03T19:07:52+00:00
osx.1100.amd64.scouting.zip 2022-05-03T19:07:52+00:00
osx.1200.amd64.scouting.open.zip 2022-06-29T15:20:43+00:00
osx.1200.amd64.scouting.zip 2022-06-29T15:20:43+00:00
raspbian.9.arm32.iot.open.zip 2022-12-07T21:57:06+00:00
raspbian.9.arm32.iot.zip 2022-12-07T21:57:06+00:00
redhat.7.amd64.arcade.zip 2023-11-09T16:07:26+00:00
redhat.7.amd64.open.arcade.zip 2023-11-09T16:07:26+00:00
ubuntu.1604.amd64.open.rt.zip 2023-10-25T20:05:40+00:00
ubuntu.1604.amd64.open.svc.zip 2023-10-25T20:18:16+00:00
ubuntu.1604.amd64.open.zip 2023-10-25T19:50:48+00:00
ubuntu.1604.amd64.perf.open.zip 2022-04-20T19:38:16+00:00
ubuntu.1604.amd64.perf.zip 2022-04-20T19:38:16+00:00
ubuntu.1604.amd64.rt.zip 2023-10-25T20:05:40+00:00
ubuntu.1604.amd64.svc.zip 2023-10-25T20:18:16+00:00
ubuntu.1604.amd64.zip 2023-10-25T19:50:47+00:00
ubuntu.1804.amd64.android.arcade.zip 2023-10-18T02:39:36+00:00
ubuntu.1804.amd64.android.open.arcade.zip 2023-10-18T02:39:36+00:00
ubuntu.1804.amd64.arcade.zip 2023-10-18T02:39:36+00:00
ubuntu.1804.amd64.open.arcade.zip 2023-10-18T02:39:36+00:00
ubuntu.1804.armarch.repro.zip 2022-04-20T19:27:41+00:00
ubuntu.2004.ppc64el.experimental.open.zip 2022-06-18T00:57:53+00:00
ubuntu.2004.ppc64el.experimental.zip 2022-06-18T00:57:52+00:00
windows.10.amd64.19h1.snake.perf.open.zip 2022-12-07T21:57:20+00:00
windows.10.amd64.client.open.reunion.zip 2023-08-02T21:06:49+00:00
windows.10.amd64.client.reunion.zip 2023-08-02T21:06:49+00:00
windows.10.amd64.client20h2.open.reunion.zip 2023-08-02T21:06:50+00:00
windows.10.amd64.client20h2.reunion.zip 2023-08-02T21:06:49+00:00
windows.10.amd64.client20h2.xaml.zip 2023-08-02T21:06:35+00:00
windows.10.amd64.client21h1.open.reunion.zip 2022-12-07T21:59:18+00:00
windows.10.amd64.client21h1.open.rt.zip 2022-12-07T22:00:47+00:00
windows.10.amd64.client21h1.open.svc.zip 2022-12-07T22:01:30+00:00
windows.10.amd64.client21h1.open.xaml.zip 2022-12-07T21:58:06+00:00
windows.10.amd64.client21h1.open.zip 2022-12-07T21:57:18+00:00
windows.10.amd64.client21h1.reunion.zip 2022-12-07T21:59:18+00:00
windows.10.amd64.client21h1.rt.zip 2022-12-07T22:00:46+00:00
windows.10.amd64.client21h1.svc.zip 2022-12-07T22:01:30+00:00
windows.10.amd64.client21h1.xaml.zip 2022-12-07T21:58:06+00:00
windows.10.amd64.client21h1.zip 2022-12-07T21:57:18+00:00
windows.10.amd64.clientpre.vs2019.pre.svc.zip 2022-11-16T19:17:08+00:00
windows.10.amd64.clientpre.vs2019.pre.zip 2022-11-16T18:46:55+00:00
windows.10.amd64.clientrs1.perf.zip 2022-04-20T19:38:32+00:00
windows.10.amd64.clientrs5.open.reunion.zip 2023-08-02T21:06:49+00:00
windows.10.amd64.clientrs5.open.xaml.zip 2023-08-02T21:06:35+00:00
windows.10.amd64.clientrs5.reunion.zip 2023-08-02T21:06:49+00:00
windows.10.amd64.clientrs5.xaml.zip 2023-08-02T21:06:35+00:00
windows.10.amd64.nperf.zip 2022-04-20T19:38:36+00:00
windows.10.amd64.server19h1.es.open.rt.zip 2022-03-23T17:37:02+00:00
windows.10.amd64.server19h1.es.open.svc.zip 2022-03-23T17:38:55+00:00
windows.10.amd64.server19h1.es.open.zip 2022-03-23T17:34:02+00:00
windows.10.amd64.server19h1.es.svc.zip 2022-03-23T17:38:54+00:00
windows.10.amd64.server19h1.es.zip 2022-03-23T17:34:03+00:00
windows.10.amd64.server20h2.open.rt.zip 2022-07-27T19:22:48+00:00
windows.10.amd64.server20h2.open.svc.zip 2022-07-27T19:49:36+00:00
windows.10.amd64.server20h2.open.zip 2022-07-27T18:25:40+00:00
windows.10.amd64.server20h2.rt.zip 2022-07-27T19:22:47+00:00
windows.10.amd64.server20h2.svc.zip 2022-07-27T19:53:41+00:00
windows.10.amd64.server20h2.zip 2022-07-27T18:25:39+00:00
windows.10.amd64.serverrs1.perf.open.zip 2022-04-20T19:38:33+00:00
windows.10.arm32.iot.open.zip 2022-12-07T21:57:19+00:00
windows.10.arm32.iot.zip 2022-12-07T21:57:20+00:00
windows.10.arm32.open.zip 2022-12-07T21:57:20+00:00
windows.10.arm32.zip 2022-12-07T21:57:19+00:00
windows.10.arm64.appcompat.zip 2022-12-07T21:57:19+00:00
windows.10.arm64.open.zip 2023-01-30T16:58:32+00:00
windows.10.arm64.tof.zip 2022-12-07T21:57:19+00:00
windows.10.arm64.zip 2023-01-30T16:58:32+00:00
windows.10.arm64v8.open.zip 2023-01-30T16:58:32+00:00
windows.10.arm64v8.zip 2023-01-30T16:58:32+00:00
windows.11.amd64.client.open.reunion.zip 2023-08-02T21:06:49+00:00
windows.11.amd64.client.open.xaml.zip 2023-08-02T21:06:35+00:00
windows.11.amd64.client.reunion.zip 2023-08-02T21:06:49+00:00
windows.11.amd64.client.xaml.zip 2023-08-02T21:06:35+00:00
windows.11.amd64.clientpre.open.svc.zip 2022-06-18T01:08:31+00:00
windows.11.amd64.clientpre.open.zip 2022-06-18T00:57:42+00:00
windows.11.amd64.clientpre.svc.zip 2022-06-18T01:08:31+00:00
windows.11.amd64.clientpre.zip 2022-06-18T00:57:42+00:00
windows.11.arm64.perf.surf.zip 2023-11-09T15:52:43+00:00
windows.7.amd64.open.rt.zip 2024-04-17T19:55:18+00:00
windows.7.amd64.open.svc.zip 2024-04-17T19:55:46+00:00
windows.7.amd64.open.zip 2024-04-17T19:52:08+00:00
windows.7.amd64.rt.zip 2024-04-17T19:55:18+00:00
windows.7.amd64.svc.zip 2024-04-17T19:55:46+00:00
windows.7.amd64.zip 2024-04-17T19:52:08+00:00
windows.amd64.server2022.open.reunion.zip 2023-08-02T21:06:49+00:00
windows.amd64.server2022.reunion.zip 2023-08-02T21:06:49+00:00
windows.server.amd64.vs2019.pre.open.svc.zip 2022-04-20T19:58:22+00:00
windows.server.amd64.vs2019.pre.open.zip 2022-04-20T19:38:23+00:00
windows.server.amd64.vs2019.pre.svc.zip 2022-04-20T19:58:23+00:00
windows.server.amd64.vs2019.pre.zip 2022-04-20T19:38:23+00:00
dougbu commented 4 months ago

Are we planning to keep this open to remind us to take manual action in the future❔ If not, does something track automating more deletions related to deleted queues❔

garath commented 4 months ago

I think this issue is tracking a better, automated solution in the future.

It also seems like a good place to note impact and manual interventions in the meantime (which can serve as a measure of the impact this solution would have).