Keep artefacts up to date on on-premise machines

[dougbu]

As discussed in the tactical #379 issue, our artefacts are not up to date on many on-premise machines. This for example means recent changes to helix-scripts/ content were unable to be used on some machines. Now, as part of Epic: Helix Machines Improvements, we should fund work to automate as much of this as we can. Running artefact updates on a regular cadence will go a long way toward improving our security stance and avoid problems likely we encountered in the 12 July rollout.

The main idea here is to keep artefacts up to date without having to reimage all on-premise machines.

Release Note Category

• [x] Feature changes/additions
• [ ] Bug fixes
• [ ] Internal Infrastructure Improvements

Release Note Description

We are now automatically keeping configuration and installations up to date on all on-premise machines.

[dougbu]

from #379: we should remember to clean a few things up before downloading and executing run.sh:

• clean up any executing Docker, Python, and emulator processes
• remove systemctl entries and similar for Helix startup
• remove the helixbot and helix-runner users and the kvm user group
• perhaps remove or disconnect any data disks
• remove a few directory trees:
  • C:\arcade-tools (on Windows)
  • /etc/helix
  • /etc/helix-prep
  • /etc/local.d (on Alpine, if we have any on-prem Alpine machines)
  • /home/helixbot
  • /Users/helix-runner (on OSX)
  • /usr/local/nodejs
  • /usr/local/v8
  • Windows equivalents of most of the above directories
  • clear the /cores directory on OSX

there's probably more to think about but this should be a fairly good start

[dougbu]

@garath please copy 'n paste what you did for #379 or, if it was scripted, please check the script in somewhere that @riarenas and I can see

[dougbu]

@dotnet/dnceng (aka @ilyas1974, @mmitche, @markwilkie, and everyone on Mark's team): Should we create a "Proactive security mitigation" milestone for compliance and security issues that are visible to us but not part of existing mandates❔ for example, this issue might become something we get mandates about if we did a better job reporting what was installed on our on-prem machines (which are all used for Helix testing and maybe some non-Helix performance work).