search

dotnet / docker-tools

This is a repo to house some common tools for our various docker repos.
MIT License
117 stars 45 forks source link

Automate testing on internal Windows builds #387

Open mthalman opened 4 years ago

mthalman commented 4 years ago

New builds of Windows are produced on a regular basis. We should setup automation to respond to the availability of a new Windows image by building and testing our Windows-based images for .NET Core and .NET Fx.

MichaelSimons commented 4 years ago

This is referring to the servicing builds for existing releases of Windows.

This isn't trivial to do for .NET FX because we would need to modify the Dockerfiles to install the LCU.

@mthalman - Can describe the benefit this would provide to us? Can you also provide a strawman plan of the necessary changes needed to support building/testing these images?

@MichaelSimons - Can you find any past issues that this type of testing would have caught?

MichaelSimons commented 4 years ago

The one time I can think of that this would have caught an issue would have been Dec 2018 servicing in the .NET Fx repo. There was a regression in Windows Server Core where applying the LCU failed after installing FoD (Feature-on-Demand) packages. The Windows team thought they had this scenario covered in their tests but there were subtle differences in how they were testing. A fix wasn't made available until the next patch cycle. The two questions I have are.

  1. Did windows address their testing to cover this scenario? A broader question at hand is how well are they testing their base images.
  2. Even if we caught this issue before the release, would it have held the release or would the issue have been fixed?
mthalman commented 4 years ago

This kind of testing would have also helped to identify the 2B container on 1B host issue that recently occurred. Identifying issues like this early on would have a huge impact, hopefully preventing breaks like this from ever being publicly published and reaching customers.

mthalman commented 4 years ago

A design proposal will be discussed on this topic.

mthalman commented 4 years ago

At this point in time, we've setup a process to do manual pre-release testing of .NET Framework images using internally available patched updates for existing versions of Windows as well as pre-releases of new Windows versions.

For .NET Core, there is also a testing process to do manual testing of pre-release Windows versions.

The remaining work is to automate the process of testing .NET Core images on internally available patched updates for existing versions of Windows. So this issue will remain open for that work.

mthalman commented 1 year ago

This work is required in order to support the full implementation of internal .NET builds, specifically for COSMIC.