Get helix-scripts/ up to date on on-premise machines

dougbu commented 1 year ago

Our on-premise machines are not all running the current helix-scripts/ version, as shown in

Under the covers, this inconsistency could hide compliance issues because the current helix-scripts/ content may be incompatible w/ the artefacts we should be using on these machines. Updating the on-premise artefacts is however a separate problem (tactically but not strategically part of #368).

Only a subset of the held-back machines (all or most of those using the 20230628.1 version) are intentionally in that state. Others are unable to download the latest configurations e.g. osx.13.arm64.m2.perf and windows.10.amd64.19h1.tiger.perf[.open] machines are getting 403 responses.

See https://helixagentlogs.blob.core.windows.net/agent-logs/osx.13.arm64.m2.perf/dci-macm2-build-048/2023-07-14T18.19.29Z.log for example:

2023-07-14T19:05:13.415Z    ERROR   run_client.py   scriptutil(88)  check_for_updates   Error trying to update scripts / config : Traceback (most recent call last):
  File "/private/etc/helix/scripts/helix/scriptutil.py", line 61, in check_for_updates
    self._replace_helix_settings()
  File "/private/etc/helix/scripts/helix/scriptutil.py", line 125, in _replace_helix_settings
    self._download_to(self.settings_uri, settings_path)
  File "/private/etc/helix/scripts/helix/scriptutil.py", line 148, in _download_to
    raise Exception("Failed to download '{}', the status code was {}.".format(str(uri).split("?")[0], response.status_code))
Exception: Failed to download 'https://helixosconfig.blob.core.windows.net/configuration/osx.13.arm64.m2.perf.zip', the status code was 403.

One likely root cause of the 403 problem (thanx @ilyas1974 and @garath for talking this through) is that the machines did not get new client settings before the contained SAS token expired. Unfortunately, the SAS token is in the client settings JSON file (which is in the Zip the above traceback was trying to download).

We may find other problems impacting the automation that regularly moves to the latest helix-scripts/ version.

First order of business is to turn the snapshot above into a list of impacted queues and machines. Second is to look for trends e.g. perhaps confirm the 403 problem is all we need to fix.

Release Note Category

[ ] Feature changes/additions
[x] Bug fixes
[x] Internal Infrastructure Improvements

Release Note Description

Found and resolved issues causing our on-premise machines to continue running old versions of our Helix client Python wheels.

dougbu commented 1 year ago

HeartbeatExport
| where ingestion_time() > ago(3d)
| where LastHeartbeat > ago(1h)
| where ScriptVersion == "20230628.1"
| summarize max(ScriptVersion) by Queue
| order by max_ScriptVersion, Queue

shows exactly what we expect after last week's rollout reversion for some on-premise machines:

Queue name	Script version
ubuntu.1804.amd64.tiger.perf.open	20230628.1
ubuntu.1804.amd64.tiger.perf	20230628.1
osx.13.arm64.open	20230628.1
osx.13.arm64	20230628.1

dougbu commented 1 year ago

HeartbeatExport
| where ingestion_time() > ago(3d)
| where LastHeartbeat > ago(1h)
| where ScriptVersion != "20230711.2" and ScriptVersion != "20230628.1" and ScriptVersion != "unset"
| summarize max(ScriptVersion) by Queue, Machine
| order by max_ScriptVersion asc, Queue asc, Machine asc

on the other hand, shows almost machines that aren't running the current helix-scripts/ version for some reason (likely not intentionally).

Queue   Machine max_ScriptVersion
windows.10.arm64v8.open DDARM64-053 20230127.1
windows.10.arm64v8.open DDARM64-057 20230127.1
windows.10.arm64v8.open DDARM64-058 20230127.1
windows.10.arm64v8.open DDARM64-062 20230127.1
windows.10.arm64v8.open DDARM64-064 20230127.1
windows.10.arm64v8.open DDARM64-065 20230127.1
windows.10.arm64v8.open DDARM64-066 20230127.1
windows.10.arm64v8.open DDARM64-067 20230127.1
windows.10.arm64v8.open DDARM64-070 20230127.1
windows.10.arm64v8.open DDARM64-072 20230127.1
windows.10.arm64v8.open DDARM64-073 20230127.1
windows.10.arm64v8.open DDARM64-075 20230127.1
windows.10.arm64v8.open DDARM64-077 20230127.1
windows.10.arm64v8.open DDARM64-078 20230127.1
windows.10.arm64v8.open DDARM64-081 20230127.1
windows.10.arm64v8.open DDARM64-083 20230127.1
windows.10.arm64v8.open DDARM64-089 20230127.1
windows.10.arm64v8.open DDARM64-090 20230127.1
windows.10.arm64v8.open DDARM64-093 20230127.1
windows.10.arm64v8.open DDARM64-095 20230127.1
windows.10.arm64v8.open DDARM64-101 20230127.1
windows.10.arm64v8.open DDARM64-102 20230127.1
windows.10.arm64v8.open DDARM64-104 20230127.1
windows.10.arm64v8.open DDARM64-105 20230127.1
windows.10.arm64v8.open DDARM64-106 20230127.1
windows.10.arm64v8.open DDARM64-107 20230127.1
windows.10.arm64v8.open DDARM64-108 20230127.1
windows.10.arm64v8.open DDARM64-112 20230127.1
windows.10.arm64v8.open DDARM64-116 20230127.1
windows.10.arm64v8.open DDARM64-117 20230127.1
windows.10.arm64v8.open DDARM64-122 20230127.1
windows.10.arm64v8.open DDARM64-123 20230127.1
windows.10.arm64v8.open DDARM64-125 20230127.1
windows.10.arm64v8.open DDARM64-126 20230127.1
windows.10.arm64v8.open DDARM64-129 20230127.1
windows.10.arm64v8.open DDARM64-130 20230127.1
windows.10.arm64v8.open DDARM64-132 20230127.1
windows.10.arm64v8.open DDARM64-139 20230127.1
windows.10.arm64v8.open DDARM64-140 20230127.1
windows.10.arm64v8.open DDARM64-142 20230127.1
windows.10.arm64v8.open DDARM64-145 20230127.1
windows.10.arm64v8.open DDARM64-153 20230127.1
windows.10.arm64v8.open DDARM64-154 20230127.1
windows.10.arm64v8.open DDARM64-161 20230127.1
windows.10.arm64v8.open DDARM64-162 20230127.1
windows.10.arm64v8.open DDARM64-163 20230127.1
windows.10.arm64v8.open DDARM64-165 20230127.1
windows.10.arm64v8.open DDARM64-166 20230127.1
windows.10.arm64v8.open DDARM64-167 20230127.1
windows.10.arm64v8.open DDARM64-168 20230127.1
windows.10.arm64v8.open DDARM64-169 20230127.1
windows.10.arm64v8.open DDARM64-172 20230127.1
windows.10.arm64v8.open DDARM64-174 20230127.1
windows.10.arm64v8.open DDARM64-177 20230127.1
windows.10.arm64v8.open DDARM64-178 20230127.1
windows.11.amd64.cet.open   DNCENGCET-001   20230531.2
windows.11.amd64.cet.open   DNCENGCET-002   20230531.2
windows.11.amd64.cet.open   DNCENGCET-003   20230531.2
windows.11.amd64.cet.open   DNCENGCET-004   20230531.2
windows.11.amd64.cet.open   DNCENGCET-005   20230531.2
windows.11.amd64.cet.open   DNCENGCET-006   20230531.2
windows.11.amd64.cet.open   DNCENGCET-007   20230531.2
windows.11.amd64.cet.open   DNCENGCET-008   20230531.2
windows.11.amd64.cet.open   DNCENGCET-009   20230531.2
windows.11.amd64.cet.open   DNCENGCET-010   20230531.2
windows.11.amd64.cet.open   DNCENGCET-011   20230531.2
windows.11.amd64.cet.open   DNCENGCET-012   20230531.2
windows.11.amd64.cet.open   DNCENGCET-013   20230531.2
windows.11.amd64.cet.open   DNCENGCET-014   20230531.2
windows.11.amd64.cet.open   DNCENGCET-015   20230531.2
windows.11.amd64.cet.open   DNCENGCET-016   20230531.2
windows.11.amd64.cet.open   DNCENGCET-017   20230531.2
windows.11.amd64.cet.open   DNCENGCET-018   20230531.2
windows.11.amd64.cet.open   DNCENGCET-019   20230531.2
windows.11.amd64.cet.open   DNCENGCET-020   20230531.2
windows.11.amd64.cet.open   DNCENGCET-021   20230531.2
windows.11.amd64.cet.open   DNCENGCET-022   20230531.2
windows.11.amd64.cet.open   DNCENGCET-023   20230531.2
windows.11.amd64.cet.open   DNCENGCET-024   20230531.2
windows.11.amd64.cet.open   DNCENGCET-025   20230531.2
windows.11.amd64.cet.open   DNCENGCET-026   20230531.2
windows.11.amd64.cet.open   DNCENGCET-027   20230531.2
windows.11.amd64.cet.open   DNCENGCET-028   20230531.2
windows.11.amd64.cet.open   DNCENGCET-029   20230531.2
windows.11.amd64.cet.open   DNCENGCET-030   20230531.2
windows.11.amd64.cet.open   DNCENGCET-031   20230531.2
windows.11.amd64.cet.open   DNCENGCET-032   20230531.2
windows.11.amd64.cet.open   DNCENGCET-033   20230531.2
windows.11.amd64.cet.open   DNCENGCET-034   20230531.2
windows.11.amd64.cet.open   DNCENGCET-035   20230531.2
windows.11.amd64.cet.open   DNCENGCET-036   20230531.2
windows.11.amd64.cet.open   DNCENGCET-037   20230531.2
windows.11.amd64.cet.open   DNCENGCET-038   20230531.2
windows.11.amd64.cet.open   DNCENGCET-039   20230531.2
windows.11.amd64.cet.open   DNCENGCET-040   20230531.2
windows.11.amd64.cet.open   DNCENGCET-041   20230531.2
windows.11.amd64.cet.open   DNCENGCET-042   20230531.2
windows.11.amd64.cet.open   DNCENGCET-043   20230531.2
windows.11.amd64.cet.open   DNCENGCET-044   20230531.2
windows.11.amd64.cet.open   DNCENGCET-045   20230531.2
windows.11.amd64.cet    DNCENGCET-046   20230531.2
windows.11.amd64.cet    DNCENGCET-047   20230531.2
windows.11.amd64.cet    DNCENGCET-048   20230531.2
windows.11.amd64.cet    DNCENGCET-049   20230531.2
windows.11.amd64.cet    DNCENGCET-050   20230531.2
windows.10.amd64.android.open   DNCENGWIN-001   20230531.2
windows.10.amd64.android.open   DNCENGWIN-002   20230531.2
windows.10.amd64.android.open   DNCENGWIN-003   20230531.2
windows.10.amd64.android.open   DNCENGWIN-004   20230531.2
windows.10.amd64.android.open   DNCENGWIN-005   20230531.2
windows.10.amd64.android.open   DNCENGWIN-006   20230531.2
windows.10.amd64.android.open   DNCENGWIN-007   20230531.2
windows.10.amd64.android.open   DNCENGWIN-008   20230531.2
windows.10.amd64.android.open   DNCENGWIN-009   20230531.2
windows.10.amd64.android.open   DNCENGWIN-010   20230531.2
windows.10.amd64.android.open   DNCENGWIN-011   20230531.2
windows.10.amd64.android.open   DNCENGWIN-012   20230531.2
windows.10.amd64.android.open   DNCENGWIN-013   20230531.2
windows.10.amd64.android.open   DNCENGWIN-014   20230531.2
windows.10.amd64.android.open   DNCENGWIN-015   20230531.2
windows.10.amd64.android.open   DNCENGWIN-016   20230531.2
windows.10.amd64.android.open   DNCENGWIN-017   20230531.2
windows.10.amd64.android.open   DNCENGWIN-018   20230531.2
windows.10.amd64.android.open   DNCENGWIN-019   20230531.2
windows.10.amd64.android.open   DNCENGWIN-020   20230531.2
windows.10.amd64.android.open   DNCENGWIN-021   20230531.2
windows.10.amd64.android.open   DNCENGWIN-022   20230531.2
windows.10.amd64.android.open   DNCENGWIN-023   20230531.2
windows.10.amd64.android.open   DNCENGWIN-024   20230531.2
windows.10.amd64.android.open   DNCENGWIN-025   20230531.2
windows.10.amd64.android.open   DNCENGWIN-026   20230531.2
windows.10.amd64.android.open   DNCENGWIN-027   20230531.2
windows.10.amd64.android.open   DNCENGWIN-028   20230531.2
windows.10.amd64.android.open   DNCENGWIN-029   20230531.2
windows.10.amd64.android.open   DNCENGWIN-030   20230531.2
windows.10.amd64.android.open   DNCENGWIN-031   20230531.2
windows.10.amd64.android.open   DNCENGWIN-032   20230531.2
windows.10.amd64.android.open   DNCENGWIN-033   20230531.2
windows.10.amd64.android.open   DNCENGWIN-034   20230531.2
windows.10.amd64.android.open   DNCENGWIN-035   20230531.2
windows.10.amd64.android.open   DNCENGWIN-036   20230531.2
windows.10.amd64.android.open   DNCENGWIN-037   20230531.2
windows.10.amd64.android.open   DNCENGWIN-038   20230531.2
windows.10.amd64.android.open   DNCENGWIN-039   20230531.2
windows.10.amd64.android.open   DNCENGWIN-040   20230531.2
windows.10.amd64.android.open   DNCENGWIN-041   20230531.2
windows.10.amd64.android.open   DNCENGWIN-042   20230531.2
windows.10.amd64.android.open   DNCENGWIN-043   20230531.2
windows.10.amd64.android.open   DNCENGWIN-044   20230531.2
windows.10.amd64.android.open   DNCENGWIN-045   20230531.2
windows.10.amd64.android.open   DNCENGWIN-046   20230531.2
windows.10.amd64.android.open   DNCENGWIN-047   20230531.2
windows.10.amd64.android.open   DNCENGWIN-048   20230531.2
windows.10.amd64.android.open   DNCENGWIN-049   20230531.2
windows.10.amd64.android.open   DNCENGWIN-050   20230531.2
windows.10.amd64.android.open   DNCENGWIN-051   20230531.2
windows.10.amd64.android.open   DNCENGWIN-053   20230531.2
windows.10.amd64.android.open   DNCENGWIN-054   20230531.2
windows.10.amd64.android.open   DNCENGWIN-055   20230531.2
windows.10.amd64.android.open   DNCENGWIN-056   20230531.2
windows.10.amd64.android.open   DNCENGWIN-057   20230531.2
windows.10.amd64.android.open   DNCENGWIN-058   20230531.2
windows.10.amd64.android.open   DNCENGWIN-059   20230531.2
windows.10.amd64.android.open   DNCENGWIN-060   20230531.2
windows.10.amd64.android.open   DNCENGWIN-061   20230531.2
windows.10.amd64.android.open   DNCENGWIN-062   20230531.2
windows.10.amd64.pixel.perf DNCENGWIN-063   20230531.2
windows.10.amd64.android.open   DNCENGWIN-064   20230531.2
windows.10.amd64.pixel.perf DNCENGWIN-065   20230531.2
windows.10.amd64.android.open   DNCENGWIN-066   20230531.2
windows.10.amd64.android.open   DNCENGWIN-067   20230531.2
windows.10.amd64.android.open   DNCENGWIN-068   20230531.2
windows.10.amd64.android.open   DNCENGWIN-069   20230531.2
windows.10.amd64.android.open   DNCENGWIN-070   20230531.2
windows.10.amd64.android.open   DNCENGWIN-071   20230531.2
windows.10.amd64.android.open   DNCENGWIN-072   20230531.2
windows.10.amd64.android.open   DNCENGWIN-073   20230531.2
windows.10.amd64.android.open   DNCENGWIN-074   20230531.2
windows.10.amd64.android.open   DNCENGWIN-075   20230531.2
windows.10.amd64.android.open   DNCENGWIN-076   20230531.2
windows.10.amd64.android.open   DNCENGWIN-077   20230531.2
windows.10.amd64.android.open   DNCENGWIN-078   20230531.2
windows.10.amd64.android.open   DNCENGWIN-079   20230531.2
windows.10.amd64.android.open   DNCENGWIN-080   20230531.2
windows.10.amd64.android.open   DNCENGWIN-081   20230531.2
windows.10.amd64.android.open   DNCENGWIN-082   20230531.2
windows.10.amd64.android.open   DNCENGWIN-083   20230531.2
windows.10.amd64.android.open   DNCENGWIN-084   20230531.2
windows.10.amd64.android.open   DNCENGWIN-085   20230531.2
windows.10.amd64.android.open   DNCENGWIN-086   20230531.2
windows.10.amd64.android.open   DNCENGWIN-087   20230531.2
windows.10.amd64.android.open   DNCENGWIN-088   20230531.2
windows.10.amd64.android.open   DNCENGWIN-089   20230531.2
windows.10.amd64.android.open   DNCENGWIN-090   20230531.2
windows.10.amd64.android.open   DNCENGWIN-091   20230531.2
windows.10.amd64.android.open   DNCENGWIN-092   20230531.2
windows.10.amd64.android.open   DNCENGWIN-093   20230531.2
windows.10.amd64.android.open   DNCENGWIN-094   20230531.2
windows.10.amd64.android.open   DNCENGWIN-095   20230531.2
windows.10.amd64.android.open   DNCENGWIN-096   20230531.2
windows.10.amd64.android.open   DNCENGWIN-097   20230531.2
windows.10.amd64.android.open   DNCENGWIN-098   20230531.2
windows.10.amd64.android.open   DNCENGWIN-099   20230531.2
windows.10.amd64.android.open   DNCENGWIN-100   20230531.2
windows.10.amd64.android.open   DNCENGWIN-101   20230531.2
windows.10.amd64.android.open   DNCENGWIN-102   20230531.2
windows.10.amd64.android.open   DNCENGWIN-103   20230531.2
windows.10.amd64.android.open   DNCENGWIN-104   20230531.2
windows.10.amd64.android.open   DNCENGWIN-105   20230531.2
windows.10.amd64.android.open   DNCENGWIN-106   20230531.2
windows.10.amd64.android.open   DNCENGWIN-107   20230531.2
windows.10.amd64.android.open   DNCENGWIN-108   20230531.2
windows.10.amd64.android.open   DNCENGWIN-109   20230531.2
windows.10.amd64.android.open   DNCENGWIN-110   20230531.2
windows.10.amd64.android.open   DNCENGWIN-111   20230531.2
windows.10.amd64.android.open   DNCENGWIN-112   20230531.2
windows.10.amd64.android.open   DNCENGWIN-113   20230531.2
windows.10.amd64.android.open   DNCENGWIN-114   20230531.2
windows.10.amd64.android.open   DNCENGWIN-115   20230531.2
windows.10.amd64.android.open   DNCENGWIN-116   20230531.2
windows.10.amd64.android.open   DNCENGWIN-117   20230531.2
windows.10.amd64.android.open   DNCENGWIN-118   20230531.2
windows.10.amd64.android.open   DNCENGWIN-119   20230531.2
windows.10.amd64.android.open   DNCENGWIN-120   20230531.2
windows.10.amd64.android.open   DNCENGWIN-121   20230531.2
windows.10.amd64.android.open   DNCENGWIN-122   20230531.2
windows.10.amd64.android.open   DNCENGWIN-123   20230531.2
windows.10.amd64.android.open   DNCENGWIN-124   20230531.2
windows.10.amd64.android.open   DNCENGWIN-125   20230531.2
windows.10.amd64.galaxy.perf    DNCENGWIN-126   20230531.2
windows.10.amd64.galaxy.perf    DNCENGWIN-127   20230531.2
windows.server.arm64.perf   DotCJade-06 20230531.2
windows.server.arm64.perf   DotCJade-07 20230531.2
windows.server.arm64.perf   DotCJade-08 20230531.2
windows.server.arm64.perf   DotCJade-09 20230531.2
windows.server.arm64.perf   DotCJade-10 20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL001  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL002  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL003  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL004  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL005  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL006  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL007  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL008  20230531.2
ubuntu.1804.amd64.owl.perf  PERFOWL009  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL010  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL011  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL012  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL013  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL014  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL015  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL016  20230531.2
windows.10.amd64.20h2.owl.perf  PERFOWL017  20230531.2
windows.10.arm64.perf.surf  PERFSURFX01 20230531.2
windows.10.arm64.perf.surf  PERFSURFX02 20230531.2
windows.10.arm64.perf.surf  PERFSURFX03 20230531.2
windows.10.arm64.perf.surf  PERFSURFX04 20230531.2
windows.10.arm64.perf.surf  PERFSURFX05 20230531.2
windows.10.arm64.perf.surf  PERFSURFX06 20230531.2
windows.10.arm64.perf.surf  PERFSURFX07 20230531.2
windows.10.arm64.perf.surf  PERFSURFX08 20230531.2
windows.10.arm64.perf.surf  PERFSURFX09 20230531.2
windows.10.arm64.perf.surf  PERFSURFX10 20230531.2
windows.11.amd64.tiger.perf PERFTIGER100    20230531.2
windows.11.amd64.tiger.perf PERFTIGER101    20230531.2
windows.11.amd64.tiger.perf PERFTIGER102    20230531.2
windows.11.amd64.tiger.perf PERFTIGER103    20230531.2
windows.11.amd64.tiger.perf PERFTIGER104    20230531.2
windows.11.amd64.tiger.perf PERFTIGER105    20230531.2
windows.11.amd64.tiger.perf PERFTIGER106    20230531.2
windows.11.amd64.tiger.perf PERFTIGER107    20230531.2
windows.11.amd64.tiger.perf PERFTIGER108    20230531.2
windows.11.amd64.tiger.perf PERFTIGER109    20230531.2
windows.11.amd64.tiger.perf PERFTIGER110    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER111    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER112    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER113    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER114    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER115    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER116    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER117    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER118    20230531.2
windows.11.amd64.tiger.perf PERFTIGER119    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER120    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER121    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER122    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER123    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER124    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER125    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER126    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER127    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER128    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER129    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER130    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER131    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER132    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER133    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER134    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER135    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER136    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER137    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER138    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER139    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER140    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER141    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER142    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER143    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER144    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER145    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER146    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER147    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER148    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER149    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER150    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER151    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER152    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER153    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER154    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER155    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER156    20230531.2
windows.10.amd64.19h1.tiger.perf    PERFTIGER157    20230531.2
ubuntu.2204.amd64.tiger.perf    PERFTIGER168    20230531.2
ubuntu.2204.amd64.tiger.perf    PERFTIGER169    20230531.2
ubuntu.2204.amd64.tiger.perf    PERFTIGER170    20230531.2
windows.10.amd64.19h1.tiger.perf.open   PERFTIGER180    20230531.2
windows.10.amd64.19h1.tiger.perf.open   PERFTIGER181    20230531.2
windows.10.amd64.19h1.tiger.perf.open   PERFTIGER182    20230531.2
windows.10.amd64.19h1.tiger.perf.open   PERFTIGER183    20230531.2
windows.10.amd64.19h1.tiger.perf.open   PERFTIGER184    20230531.2
windows.10.amd64.server2022.es.open.rt  a0016IV 20230531.2
osx.13.arm64.m2.perf    dci-macm2-build-048 20230531.2
osx.13.arm64.m2.perf    dci-macm2-build-049 20230531.2
osx.13.arm64.m2.perf    dci-macm2-build-050 20230531.2
raspbian.11.arm32.iot.open  ddvsorpil016    20230531.2
raspbian.11.arm32.iot.open  ddvsorpil017    20230531.2
raspbian.11.arm32.iot.open  ddvsorpil019    20230531.2
raspbian.11.arm32.iot   ddvsorpil020    20230531.2
ubuntu.2004.arm64.perf  dotcjade-11 20230531.2
ubuntu.2004.arm64.perf  dotcjade-14 20230531.2
ubuntu.2004.arm64.perf  dotcjade-15 20230531.2
ubuntu.2004.arm64.perf  dotcjade-16 20230531.2
ubuntu.2004.arm64.perf  dotcjade-17 20230531.2
ubuntu.2004.arm64.perf  dotcjade-18 20230531.2
ubuntu.2004.ppc64le.experimental.open   dotnet-1    20230531.2
ubuntu.2004.ppc64le.experimental.open   dotnet-4    20230531.2
ubuntu.2004.s390x.experimental.open ms01    20230531.2
ubuntu.2004.s390x.experimental.open ms02    20230531.2

dougbu commented 1 year ago

@ilyas1974 I think you were going to look at a few more machines than I checked over the weekend (a few osx.13.arm64.m2.perf and windows.10.amd64.19h1.tiger.perf[.open] machines, where I saw the 403 issues). Right now, I'm focusing on #368.

ilyas1974 commented 1 year ago

Let's take in this "sections"

20230127.1
- All systems are set to be retired per https://portal.microsofticm.com/imp/v3/incidents/details/405548854/home
- Can be disregarded for the purposes of any investigation.
20230131.7 and 20230307.1
- We moved off these particular systems but never removed them from the heartbeat table. I have corrected this.
20230531.2
- Need to investigate what changes we deployed around 5/31 to cause these systems not to get updates.
20230628.1
- Similar to 20230531.2 - PERF and macOS , need to investigate why this class of system is not taking updates.

dougbu commented 1 year ago

Let's take in this "sections"

20230127.1

All systems are set to be retired per https://portal.microsofticm.com/imp/v3/incidents/details/405548854/home

Can be disregarded for the purposes of any investigation.

20230131.7 and 20230307.1

We moved off these particular systems but never removed them from the heartbeat table. I have corrected this.

I didn't see anything w/ ScriptVersion 20230131.7 or 20230307.1 but wouldn't be surprised if your queries were more correct than mine.

For these and the to-be-retired systems using 20230127.1, why do they continue heart-beating❔ Note the | where LastHeartbeat > ago(1h) in my query.

20230531.2

Need to investigate what changes we deployed around 5/31 to cause these systems not to get updates.

👍

20230628.1

Similar to 20230531.2 - PERF and macOS , need to investigate why this class of system is not taking updates.

These are the systems we reverted to the previous helix-scripts/ version because their installed artefacts could not use the latest Python wheels. This category is entirely expected and should be fixed once we resolve #368.

dougbu commented 1 year ago

/btw it was a few of the machines reporting ScriptVersion 20230531.2 where I noticed the 403 errors in the agent logs. Your suspicion about the configuration download SAS token expiring before the machines updated may explain that. But, it's oddly consistent — seemingly (from a brief glance at the machine counts) affecting all machines in a few queues.

ilyas1974 commented 1 year ago

I have a feeling that because of their "ago" the first two groups of systems (ScriptVersion 20230131.7 or 20230307.1) are older than what we keep in Kusto. The heart beat table doesn't delete anything we've designated as "on premises" so they can and will stick around.
RE: ScriptVersion 20230127.1 - Retirements. I did not place a high priority of getting the systems retired as we were going through the re-org at the time. Typically we do try and remove systems we plan on retiring from Helix within a relatively short period of time.
ScriptVersion 20230531.2 is indeed an experienced SAS token (left over from the secret rotation ~6/5. The main reason it hasn't been addressed is because the SAS token in question only effected Helix updates to the systems - customer workloads continue to run without issue.

ilyas1974 commented 1 year ago

Resolved the script version issues on all systems expect those with version 20230531.2. Currently working on them. https://portal.microsofticm.com/imp/v3/incidents/details/408185319/home for the Ubuntu based systems.

ilyas1974 commented 1 year ago

All but 12 of the systems have been resolved.

I have created https://portal.microsofticm.com/imp/v3/incidents/details/411459136/home to have DDFUN address these remaining systems.

ilyas1974 commented 1 year ago

All systems have not been updated.

dotnet / dnceng

Get helix-scripts/ up to date on on-premise machines #377

Release Note Category

Release Note Description