Closed dougbu closed 1 year ago
HeartbeatExport
| where ingestion_time() > ago(3d)
| where LastHeartbeat > ago(1h)
| where ScriptVersion == "20230628.1"
| summarize max(ScriptVersion) by Queue
| order by max_ScriptVersion, Queue
shows exactly what we expect after last week's rollout reversion for some on-premise machines:
Queue name | Script version |
---|---|
ubuntu.1804.amd64.tiger.perf.open | 20230628.1 |
ubuntu.1804.amd64.tiger.perf | 20230628.1 |
osx.13.arm64.open | 20230628.1 |
osx.13.arm64 | 20230628.1 |
HeartbeatExport
| where ingestion_time() > ago(3d)
| where LastHeartbeat > ago(1h)
| where ScriptVersion != "20230711.2" and ScriptVersion != "20230628.1" and ScriptVersion != "unset"
| summarize max(ScriptVersion) by Queue, Machine
| order by max_ScriptVersion asc, Queue asc, Machine asc
on the other hand, shows almost machines that aren't running the current helix-scripts/ version for some reason (likely not intentionally).
Queue Machine max_ScriptVersion
windows.10.arm64v8.open DDARM64-053 20230127.1
windows.10.arm64v8.open DDARM64-057 20230127.1
windows.10.arm64v8.open DDARM64-058 20230127.1
windows.10.arm64v8.open DDARM64-062 20230127.1
windows.10.arm64v8.open DDARM64-064 20230127.1
windows.10.arm64v8.open DDARM64-065 20230127.1
windows.10.arm64v8.open DDARM64-066 20230127.1
windows.10.arm64v8.open DDARM64-067 20230127.1
windows.10.arm64v8.open DDARM64-070 20230127.1
windows.10.arm64v8.open DDARM64-072 20230127.1
windows.10.arm64v8.open DDARM64-073 20230127.1
windows.10.arm64v8.open DDARM64-075 20230127.1
windows.10.arm64v8.open DDARM64-077 20230127.1
windows.10.arm64v8.open DDARM64-078 20230127.1
windows.10.arm64v8.open DDARM64-081 20230127.1
windows.10.arm64v8.open DDARM64-083 20230127.1
windows.10.arm64v8.open DDARM64-089 20230127.1
windows.10.arm64v8.open DDARM64-090 20230127.1
windows.10.arm64v8.open DDARM64-093 20230127.1
windows.10.arm64v8.open DDARM64-095 20230127.1
windows.10.arm64v8.open DDARM64-101 20230127.1
windows.10.arm64v8.open DDARM64-102 20230127.1
windows.10.arm64v8.open DDARM64-104 20230127.1
windows.10.arm64v8.open DDARM64-105 20230127.1
windows.10.arm64v8.open DDARM64-106 20230127.1
windows.10.arm64v8.open DDARM64-107 20230127.1
windows.10.arm64v8.open DDARM64-108 20230127.1
windows.10.arm64v8.open DDARM64-112 20230127.1
windows.10.arm64v8.open DDARM64-116 20230127.1
windows.10.arm64v8.open DDARM64-117 20230127.1
windows.10.arm64v8.open DDARM64-122 20230127.1
windows.10.arm64v8.open DDARM64-123 20230127.1
windows.10.arm64v8.open DDARM64-125 20230127.1
windows.10.arm64v8.open DDARM64-126 20230127.1
windows.10.arm64v8.open DDARM64-129 20230127.1
windows.10.arm64v8.open DDARM64-130 20230127.1
windows.10.arm64v8.open DDARM64-132 20230127.1
windows.10.arm64v8.open DDARM64-139 20230127.1
windows.10.arm64v8.open DDARM64-140 20230127.1
windows.10.arm64v8.open DDARM64-142 20230127.1
windows.10.arm64v8.open DDARM64-145 20230127.1
windows.10.arm64v8.open DDARM64-153 20230127.1
windows.10.arm64v8.open DDARM64-154 20230127.1
windows.10.arm64v8.open DDARM64-161 20230127.1
windows.10.arm64v8.open DDARM64-162 20230127.1
windows.10.arm64v8.open DDARM64-163 20230127.1
windows.10.arm64v8.open DDARM64-165 20230127.1
windows.10.arm64v8.open DDARM64-166 20230127.1
windows.10.arm64v8.open DDARM64-167 20230127.1
windows.10.arm64v8.open DDARM64-168 20230127.1
windows.10.arm64v8.open DDARM64-169 20230127.1
windows.10.arm64v8.open DDARM64-172 20230127.1
windows.10.arm64v8.open DDARM64-174 20230127.1
windows.10.arm64v8.open DDARM64-177 20230127.1
windows.10.arm64v8.open DDARM64-178 20230127.1
windows.11.amd64.cet.open DNCENGCET-001 20230531.2
windows.11.amd64.cet.open DNCENGCET-002 20230531.2
windows.11.amd64.cet.open DNCENGCET-003 20230531.2
windows.11.amd64.cet.open DNCENGCET-004 20230531.2
windows.11.amd64.cet.open DNCENGCET-005 20230531.2
windows.11.amd64.cet.open DNCENGCET-006 20230531.2
windows.11.amd64.cet.open DNCENGCET-007 20230531.2
windows.11.amd64.cet.open DNCENGCET-008 20230531.2
windows.11.amd64.cet.open DNCENGCET-009 20230531.2
windows.11.amd64.cet.open DNCENGCET-010 20230531.2
windows.11.amd64.cet.open DNCENGCET-011 20230531.2
windows.11.amd64.cet.open DNCENGCET-012 20230531.2
windows.11.amd64.cet.open DNCENGCET-013 20230531.2
windows.11.amd64.cet.open DNCENGCET-014 20230531.2
windows.11.amd64.cet.open DNCENGCET-015 20230531.2
windows.11.amd64.cet.open DNCENGCET-016 20230531.2
windows.11.amd64.cet.open DNCENGCET-017 20230531.2
windows.11.amd64.cet.open DNCENGCET-018 20230531.2
windows.11.amd64.cet.open DNCENGCET-019 20230531.2
windows.11.amd64.cet.open DNCENGCET-020 20230531.2
windows.11.amd64.cet.open DNCENGCET-021 20230531.2
windows.11.amd64.cet.open DNCENGCET-022 20230531.2
windows.11.amd64.cet.open DNCENGCET-023 20230531.2
windows.11.amd64.cet.open DNCENGCET-024 20230531.2
windows.11.amd64.cet.open DNCENGCET-025 20230531.2
windows.11.amd64.cet.open DNCENGCET-026 20230531.2
windows.11.amd64.cet.open DNCENGCET-027 20230531.2
windows.11.amd64.cet.open DNCENGCET-028 20230531.2
windows.11.amd64.cet.open DNCENGCET-029 20230531.2
windows.11.amd64.cet.open DNCENGCET-030 20230531.2
windows.11.amd64.cet.open DNCENGCET-031 20230531.2
windows.11.amd64.cet.open DNCENGCET-032 20230531.2
windows.11.amd64.cet.open DNCENGCET-033 20230531.2
windows.11.amd64.cet.open DNCENGCET-034 20230531.2
windows.11.amd64.cet.open DNCENGCET-035 20230531.2
windows.11.amd64.cet.open DNCENGCET-036 20230531.2
windows.11.amd64.cet.open DNCENGCET-037 20230531.2
windows.11.amd64.cet.open DNCENGCET-038 20230531.2
windows.11.amd64.cet.open DNCENGCET-039 20230531.2
windows.11.amd64.cet.open DNCENGCET-040 20230531.2
windows.11.amd64.cet.open DNCENGCET-041 20230531.2
windows.11.amd64.cet.open DNCENGCET-042 20230531.2
windows.11.amd64.cet.open DNCENGCET-043 20230531.2
windows.11.amd64.cet.open DNCENGCET-044 20230531.2
windows.11.amd64.cet.open DNCENGCET-045 20230531.2
windows.11.amd64.cet DNCENGCET-046 20230531.2
windows.11.amd64.cet DNCENGCET-047 20230531.2
windows.11.amd64.cet DNCENGCET-048 20230531.2
windows.11.amd64.cet DNCENGCET-049 20230531.2
windows.11.amd64.cet DNCENGCET-050 20230531.2
windows.10.amd64.android.open DNCENGWIN-001 20230531.2
windows.10.amd64.android.open DNCENGWIN-002 20230531.2
windows.10.amd64.android.open DNCENGWIN-003 20230531.2
windows.10.amd64.android.open DNCENGWIN-004 20230531.2
windows.10.amd64.android.open DNCENGWIN-005 20230531.2
windows.10.amd64.android.open DNCENGWIN-006 20230531.2
windows.10.amd64.android.open DNCENGWIN-007 20230531.2
windows.10.amd64.android.open DNCENGWIN-008 20230531.2
windows.10.amd64.android.open DNCENGWIN-009 20230531.2
windows.10.amd64.android.open DNCENGWIN-010 20230531.2
windows.10.amd64.android.open DNCENGWIN-011 20230531.2
windows.10.amd64.android.open DNCENGWIN-012 20230531.2
windows.10.amd64.android.open DNCENGWIN-013 20230531.2
windows.10.amd64.android.open DNCENGWIN-014 20230531.2
windows.10.amd64.android.open DNCENGWIN-015 20230531.2
windows.10.amd64.android.open DNCENGWIN-016 20230531.2
windows.10.amd64.android.open DNCENGWIN-017 20230531.2
windows.10.amd64.android.open DNCENGWIN-018 20230531.2
windows.10.amd64.android.open DNCENGWIN-019 20230531.2
windows.10.amd64.android.open DNCENGWIN-020 20230531.2
windows.10.amd64.android.open DNCENGWIN-021 20230531.2
windows.10.amd64.android.open DNCENGWIN-022 20230531.2
windows.10.amd64.android.open DNCENGWIN-023 20230531.2
windows.10.amd64.android.open DNCENGWIN-024 20230531.2
windows.10.amd64.android.open DNCENGWIN-025 20230531.2
windows.10.amd64.android.open DNCENGWIN-026 20230531.2
windows.10.amd64.android.open DNCENGWIN-027 20230531.2
windows.10.amd64.android.open DNCENGWIN-028 20230531.2
windows.10.amd64.android.open DNCENGWIN-029 20230531.2
windows.10.amd64.android.open DNCENGWIN-030 20230531.2
windows.10.amd64.android.open DNCENGWIN-031 20230531.2
windows.10.amd64.android.open DNCENGWIN-032 20230531.2
windows.10.amd64.android.open DNCENGWIN-033 20230531.2
windows.10.amd64.android.open DNCENGWIN-034 20230531.2
windows.10.amd64.android.open DNCENGWIN-035 20230531.2
windows.10.amd64.android.open DNCENGWIN-036 20230531.2
windows.10.amd64.android.open DNCENGWIN-037 20230531.2
windows.10.amd64.android.open DNCENGWIN-038 20230531.2
windows.10.amd64.android.open DNCENGWIN-039 20230531.2
windows.10.amd64.android.open DNCENGWIN-040 20230531.2
windows.10.amd64.android.open DNCENGWIN-041 20230531.2
windows.10.amd64.android.open DNCENGWIN-042 20230531.2
windows.10.amd64.android.open DNCENGWIN-043 20230531.2
windows.10.amd64.android.open DNCENGWIN-044 20230531.2
windows.10.amd64.android.open DNCENGWIN-045 20230531.2
windows.10.amd64.android.open DNCENGWIN-046 20230531.2
windows.10.amd64.android.open DNCENGWIN-047 20230531.2
windows.10.amd64.android.open DNCENGWIN-048 20230531.2
windows.10.amd64.android.open DNCENGWIN-049 20230531.2
windows.10.amd64.android.open DNCENGWIN-050 20230531.2
windows.10.amd64.android.open DNCENGWIN-051 20230531.2
windows.10.amd64.android.open DNCENGWIN-053 20230531.2
windows.10.amd64.android.open DNCENGWIN-054 20230531.2
windows.10.amd64.android.open DNCENGWIN-055 20230531.2
windows.10.amd64.android.open DNCENGWIN-056 20230531.2
windows.10.amd64.android.open DNCENGWIN-057 20230531.2
windows.10.amd64.android.open DNCENGWIN-058 20230531.2
windows.10.amd64.android.open DNCENGWIN-059 20230531.2
windows.10.amd64.android.open DNCENGWIN-060 20230531.2
windows.10.amd64.android.open DNCENGWIN-061 20230531.2
windows.10.amd64.android.open DNCENGWIN-062 20230531.2
windows.10.amd64.pixel.perf DNCENGWIN-063 20230531.2
windows.10.amd64.android.open DNCENGWIN-064 20230531.2
windows.10.amd64.pixel.perf DNCENGWIN-065 20230531.2
windows.10.amd64.android.open DNCENGWIN-066 20230531.2
windows.10.amd64.android.open DNCENGWIN-067 20230531.2
windows.10.amd64.android.open DNCENGWIN-068 20230531.2
windows.10.amd64.android.open DNCENGWIN-069 20230531.2
windows.10.amd64.android.open DNCENGWIN-070 20230531.2
windows.10.amd64.android.open DNCENGWIN-071 20230531.2
windows.10.amd64.android.open DNCENGWIN-072 20230531.2
windows.10.amd64.android.open DNCENGWIN-073 20230531.2
windows.10.amd64.android.open DNCENGWIN-074 20230531.2
windows.10.amd64.android.open DNCENGWIN-075 20230531.2
windows.10.amd64.android.open DNCENGWIN-076 20230531.2
windows.10.amd64.android.open DNCENGWIN-077 20230531.2
windows.10.amd64.android.open DNCENGWIN-078 20230531.2
windows.10.amd64.android.open DNCENGWIN-079 20230531.2
windows.10.amd64.android.open DNCENGWIN-080 20230531.2
windows.10.amd64.android.open DNCENGWIN-081 20230531.2
windows.10.amd64.android.open DNCENGWIN-082 20230531.2
windows.10.amd64.android.open DNCENGWIN-083 20230531.2
windows.10.amd64.android.open DNCENGWIN-084 20230531.2
windows.10.amd64.android.open DNCENGWIN-085 20230531.2
windows.10.amd64.android.open DNCENGWIN-086 20230531.2
windows.10.amd64.android.open DNCENGWIN-087 20230531.2
windows.10.amd64.android.open DNCENGWIN-088 20230531.2
windows.10.amd64.android.open DNCENGWIN-089 20230531.2
windows.10.amd64.android.open DNCENGWIN-090 20230531.2
windows.10.amd64.android.open DNCENGWIN-091 20230531.2
windows.10.amd64.android.open DNCENGWIN-092 20230531.2
windows.10.amd64.android.open DNCENGWIN-093 20230531.2
windows.10.amd64.android.open DNCENGWIN-094 20230531.2
windows.10.amd64.android.open DNCENGWIN-095 20230531.2
windows.10.amd64.android.open DNCENGWIN-096 20230531.2
windows.10.amd64.android.open DNCENGWIN-097 20230531.2
windows.10.amd64.android.open DNCENGWIN-098 20230531.2
windows.10.amd64.android.open DNCENGWIN-099 20230531.2
windows.10.amd64.android.open DNCENGWIN-100 20230531.2
windows.10.amd64.android.open DNCENGWIN-101 20230531.2
windows.10.amd64.android.open DNCENGWIN-102 20230531.2
windows.10.amd64.android.open DNCENGWIN-103 20230531.2
windows.10.amd64.android.open DNCENGWIN-104 20230531.2
windows.10.amd64.android.open DNCENGWIN-105 20230531.2
windows.10.amd64.android.open DNCENGWIN-106 20230531.2
windows.10.amd64.android.open DNCENGWIN-107 20230531.2
windows.10.amd64.android.open DNCENGWIN-108 20230531.2
windows.10.amd64.android.open DNCENGWIN-109 20230531.2
windows.10.amd64.android.open DNCENGWIN-110 20230531.2
windows.10.amd64.android.open DNCENGWIN-111 20230531.2
windows.10.amd64.android.open DNCENGWIN-112 20230531.2
windows.10.amd64.android.open DNCENGWIN-113 20230531.2
windows.10.amd64.android.open DNCENGWIN-114 20230531.2
windows.10.amd64.android.open DNCENGWIN-115 20230531.2
windows.10.amd64.android.open DNCENGWIN-116 20230531.2
windows.10.amd64.android.open DNCENGWIN-117 20230531.2
windows.10.amd64.android.open DNCENGWIN-118 20230531.2
windows.10.amd64.android.open DNCENGWIN-119 20230531.2
windows.10.amd64.android.open DNCENGWIN-120 20230531.2
windows.10.amd64.android.open DNCENGWIN-121 20230531.2
windows.10.amd64.android.open DNCENGWIN-122 20230531.2
windows.10.amd64.android.open DNCENGWIN-123 20230531.2
windows.10.amd64.android.open DNCENGWIN-124 20230531.2
windows.10.amd64.android.open DNCENGWIN-125 20230531.2
windows.10.amd64.galaxy.perf DNCENGWIN-126 20230531.2
windows.10.amd64.galaxy.perf DNCENGWIN-127 20230531.2
windows.server.arm64.perf DotCJade-06 20230531.2
windows.server.arm64.perf DotCJade-07 20230531.2
windows.server.arm64.perf DotCJade-08 20230531.2
windows.server.arm64.perf DotCJade-09 20230531.2
windows.server.arm64.perf DotCJade-10 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL001 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL002 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL003 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL004 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL005 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL006 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL007 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL008 20230531.2
ubuntu.1804.amd64.owl.perf PERFOWL009 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL010 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL011 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL012 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL013 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL014 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL015 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL016 20230531.2
windows.10.amd64.20h2.owl.perf PERFOWL017 20230531.2
windows.10.arm64.perf.surf PERFSURFX01 20230531.2
windows.10.arm64.perf.surf PERFSURFX02 20230531.2
windows.10.arm64.perf.surf PERFSURFX03 20230531.2
windows.10.arm64.perf.surf PERFSURFX04 20230531.2
windows.10.arm64.perf.surf PERFSURFX05 20230531.2
windows.10.arm64.perf.surf PERFSURFX06 20230531.2
windows.10.arm64.perf.surf PERFSURFX07 20230531.2
windows.10.arm64.perf.surf PERFSURFX08 20230531.2
windows.10.arm64.perf.surf PERFSURFX09 20230531.2
windows.10.arm64.perf.surf PERFSURFX10 20230531.2
windows.11.amd64.tiger.perf PERFTIGER100 20230531.2
windows.11.amd64.tiger.perf PERFTIGER101 20230531.2
windows.11.amd64.tiger.perf PERFTIGER102 20230531.2
windows.11.amd64.tiger.perf PERFTIGER103 20230531.2
windows.11.amd64.tiger.perf PERFTIGER104 20230531.2
windows.11.amd64.tiger.perf PERFTIGER105 20230531.2
windows.11.amd64.tiger.perf PERFTIGER106 20230531.2
windows.11.amd64.tiger.perf PERFTIGER107 20230531.2
windows.11.amd64.tiger.perf PERFTIGER108 20230531.2
windows.11.amd64.tiger.perf PERFTIGER109 20230531.2
windows.11.amd64.tiger.perf PERFTIGER110 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER111 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER112 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER113 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER114 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER115 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER116 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER117 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER118 20230531.2
windows.11.amd64.tiger.perf PERFTIGER119 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER120 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER121 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER122 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER123 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER124 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER125 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER126 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER127 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER128 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER129 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER130 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER131 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER132 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER133 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER134 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER135 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER136 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER137 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER138 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER139 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER140 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER141 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER142 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER143 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER144 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER145 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER146 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER147 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER148 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER149 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER150 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER151 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER152 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER153 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER154 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER155 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER156 20230531.2
windows.10.amd64.19h1.tiger.perf PERFTIGER157 20230531.2
ubuntu.2204.amd64.tiger.perf PERFTIGER168 20230531.2
ubuntu.2204.amd64.tiger.perf PERFTIGER169 20230531.2
ubuntu.2204.amd64.tiger.perf PERFTIGER170 20230531.2
windows.10.amd64.19h1.tiger.perf.open PERFTIGER180 20230531.2
windows.10.amd64.19h1.tiger.perf.open PERFTIGER181 20230531.2
windows.10.amd64.19h1.tiger.perf.open PERFTIGER182 20230531.2
windows.10.amd64.19h1.tiger.perf.open PERFTIGER183 20230531.2
windows.10.amd64.19h1.tiger.perf.open PERFTIGER184 20230531.2
windows.10.amd64.server2022.es.open.rt a0016IV 20230531.2
osx.13.arm64.m2.perf dci-macm2-build-048 20230531.2
osx.13.arm64.m2.perf dci-macm2-build-049 20230531.2
osx.13.arm64.m2.perf dci-macm2-build-050 20230531.2
raspbian.11.arm32.iot.open ddvsorpil016 20230531.2
raspbian.11.arm32.iot.open ddvsorpil017 20230531.2
raspbian.11.arm32.iot.open ddvsorpil019 20230531.2
raspbian.11.arm32.iot ddvsorpil020 20230531.2
ubuntu.2004.arm64.perf dotcjade-11 20230531.2
ubuntu.2004.arm64.perf dotcjade-14 20230531.2
ubuntu.2004.arm64.perf dotcjade-15 20230531.2
ubuntu.2004.arm64.perf dotcjade-16 20230531.2
ubuntu.2004.arm64.perf dotcjade-17 20230531.2
ubuntu.2004.arm64.perf dotcjade-18 20230531.2
ubuntu.2004.ppc64le.experimental.open dotnet-1 20230531.2
ubuntu.2004.ppc64le.experimental.open dotnet-4 20230531.2
ubuntu.2004.s390x.experimental.open ms01 20230531.2
ubuntu.2004.s390x.experimental.open ms02 20230531.2
@ilyas1974 I think you were going to look at a few more machines than I checked over the weekend (a few osx.13.arm64.m2.perf
and windows.10.amd64.19h1.tiger.perf[.open]
machines, where I saw the 403
issues). Right now, I'm focusing on #368.
Let's take in this "sections"
Let's take in this "sections"
20230127.1
- All systems are set to be retired per https://portal.microsofticm.com/imp/v3/incidents/details/405548854/home
- Can be disregarded for the purposes of any investigation.
20230131.7 and 20230307.1
- We moved off these particular systems but never removed them from the heartbeat table. I have corrected this.
I didn't see anything w/ ScriptVersion
20230131.7 or 20230307.1 but wouldn't be surprised if your queries were more correct than mine.
For these and the to-be-retired systems using 20230127.1, why do they continue heart-beating❔ Note the | where LastHeartbeat > ago(1h)
in my query.
20230531.2
- Need to investigate what changes we deployed around 5/31 to cause these systems not to get updates.
👍
20230628.1
- Similar to 20230531.2 - PERF and macOS , need to investigate why this class of system is not taking updates.
These are the systems we reverted to the previous helix-scripts/ version because their installed artefacts could not use the latest Python wheels. This category is entirely expected and should be fixed once we resolve #368.
/btw it was a few of the machines reporting ScriptVersion
20230531.2 where I noticed the 403
errors in the agent logs. Your suspicion about the configuration download SAS token expiring before the machines updated may explain that. But, it's oddly consistent — seemingly (from a brief glance at the machine counts) affecting all machines in a few queues.
I have a feeling that because of their "ago" the first two groups of systems (ScriptVersion
20230131.7 or 20230307.1) are older than what we keep in Kusto. The heart beat table doesn't delete anything we've designated as "on premises" so they can and will stick around.
RE: ScriptVersion
20230127.1 - Retirements. I did not place a high priority of getting the systems retired as we were going through the re-org at the time. Typically we do try and remove systems we plan on retiring from Helix within a relatively short period of time.
ScriptVersion
20230531.2 is indeed an experienced SAS token (left over from the secret rotation ~6/5. The main reason it hasn't been addressed is because the SAS token in question only effected Helix updates to the systems - customer workloads continue to run without issue.
Resolved the script version issues on all systems expect those with version 20230531.2. Currently working on them. https://portal.microsofticm.com/imp/v3/incidents/details/408185319/home for the Ubuntu based systems.
All but 12 of the systems have been resolved.
I have created https://portal.microsofticm.com/imp/v3/incidents/details/411459136/home to have DDFUN address these remaining systems.
All systems have not been updated.
Our on-premise machines are not all running the current helix-scripts/ version, as shown in![image](https://github.com/dotnet/dnceng/assets/6431421/e94c0134-3162-4e24-8852-3655beee9263)
Under the covers, this inconsistency could hide compliance issues because the current helix-scripts/ content may be incompatible w/ the artefacts we should be using on these machines. Updating the on-premise artefacts is however a separate problem (tactically but not strategically part of #368).
Only a subset of the held-back machines (all or most of those using the
20230628.1
version) are intentionally in that state. Others are unable to download the latest configurations e.g.osx.13.arm64.m2.perf
andwindows.10.amd64.19h1.tiger.perf[.open]
machines are getting403
responses.See https://helixagentlogs.blob.core.windows.net/agent-logs/osx.13.arm64.m2.perf/dci-macm2-build-048/2023-07-14T18.19.29Z.log for example:
One likely root cause of the
403
problem (thanx @ilyas1974 and @garath for talking this through) is that the machines did not get new client settings before the contained SAS token expired. Unfortunately, the SAS token is in the client settings JSON file (which is in the Zip the above traceback was trying to download).We may find other problems impacting the automation that regularly moves to the latest helix-scripts/ version.
First order of business is to turn the snapshot above into a list of impacted queues and machines. Second is to look for trends e.g. perhaps confirm the
403
problem is all we need to fix.Release Note Category
Release Note Description
Found and resolved issues causing our on-premise machines to continue running old versions of our Helix client Python wheels.