Deterministic Windows patching

[PatrickLang]

Problem Statement

Windows Update's automatic behavior doesn't match what's needed for AKS-Engine or AKS in a few ways:

• Some optional updates may need to be applied, but not others
• Some patches may need to be avoided if we find failures in testing. For example, the April & May 2019 cumulative updates had changes that intermittently broke Kubernetes service connectivity and DNS

It would need to be configurable so that we can use it to patch VHDs with the known-good list of patches from Windows Update services, but also used for private testing to apply new patches in testing that may be in private storage blobs.

Related solutions

Service Fabric Patch Orchestration Application

doc, github can orchestrate upgrades across multiple nodes, and handles stopping the Service Fabric service cleanly before reboot and starts it after upgrades complete. It also has a central service to make sure nodes are not taken down simultaneously. That would work for "hotpatching" scenarios if adapted to work with Kubernetes.

As a standalone service, it may still be possible to use this to control what updates are installed. This config would let you include/exclude specific patches using a query language.

https://github.com/microsoft/Service-Fabric-POA/blob/0accfa6d7b6218e1b9543e7403bd7a34905906f0/src/PatchOrchestrationApplication/PatchOrchestrationApplication/ApplicationPackageRoot/ApplicationManifest.xml#L27-L28

windows-patching extension

https://github.com/Azure/aks-engine/tree/master/extensions/windows-patches

This uses the AKS-Engine extension support which calls an extra script as a VM extension. It has a few downsides:

1. No VM Scale Set Support. This means it can't work in AKS
2. It requires full paths to MSU or EXE files. It can't find them using Windows Update

[CecileRobertMichon]

@marosset closing this one as it seems outdated but feel free to reopen if needed.