microsoft / Windows-Containers

Welcome to our Windows Containers GitHub community! Ask questions, report bugs, and suggest features -- let's work together.
MIT License
421 stars 64 forks source link

2022-10 LCU seems to have broken workload #281

Closed ghost closed 1 year ago

ghost commented 2 years ago

Describe the bug We run a set of onprem Azure Pipelines agents via swarm mode using a custom image containing the latest VS2019 Build Tools and a number of MSI-based dependencies; this image is rebuilt monthly when the base image is updated.

Following the installation of KB5018421 on the hosts and rebuilding the image, when the pipeline agent tries to MSBuild a solution the container terminates unexpectedly with status Exited (2147483697), with a matching error condition reported in Event Viewer.

To Reproduce In the container, MSBuild a solution containing C# projects.

(Our build process has a fair few moving parts and the solution contains 150+ projects, isolating this to a simple rep case may take more time than we have available. Our scenario used to work cleanly, and other steps running in the container such as git clone and PowerShell scripts seem to be working fine as before.)

Expected behavior Expected the build process to remain as it was before the update -- build errors to be reported via Azure DevOps rather than container termination.

Configuration:

Additional context

We're at something of a loss with this one; a guess is that something is killing the entrypoint process, but it's unclear what or how, or how to proceed with further diagnostics.

ghost commented 2 years ago

Can also rep this with the latest Moby runtime on a clean host VM.

ghost commented 1 year ago

This issue appears to have been resolved by the 2022-11 updated base image.

It turns out (via LogMonitor) the underlying error was a crash in LSASS.exe from netlogon.dll inside the container, possibly related to running the containers with a domain-joined gMSA, which would then trigger the "reboot in 60 seconds" behaviour which terminated the container unexpectedly.