search

microsoft / dotnet-framework-docker

The repo for the official docker images for .NET Framework on Windows Server Core.
https://hub.docker.com/_/microsoft-dotnet-framework
MIT License
698 stars 334 forks source link

Use Dockerfile paths to determine which test data is in scope #1026

Closed mthalman closed 1 year ago

mthalman commented 1 year ago

This updates the test project to reflect the changes that were made in https://github.com/dotnet/dotnet-docker/pull/4207 and to conform to the new test script contract defined by https://github.com/dotnet/docker-tools/pull/1082. This means the test project no longer filters tests by product version but rather by Dockerfile path.

This change made it very cumbersome to attempt to adapt the existing SkippableTheoryAttribute class to use the Dockerfile path data in order to determine which tests to skip. So instead of doing that, I got rid of it entirely and went with a different approach. Each of the test projects defined its own set of test data. This data was largely duplicated amongst the classes except for WCF and SDK tests which needed to limit which versions they tested (hence the need for SkippableTheoryAttribute).

I consolidated all of that test data into one set which all the test classes use. Then I made use of the Xunit.SkippableFact NuGet package. This provides its own SkippableTheoryAttribute that, when used, allows the code to throw a SkipException to indicate that the test should be skipped. It also provides some helper methods that will throw a SkipException based on a condition. This then allowed me to reimplement the skip logic based on version in a much more straightforward manner.

mthalman commented 1 year ago

I made some changes to handle a case where it was still running into the XUnit issue of having an empty set for the member data. That can happen when the paths filter causes the image for a given image type to be filtered out. For example, a 3.5 test leg would have the following paths specified:

There is no wcf path here, by design, so this causes the WCF tests to end up with an empty set of image data and XUnit throws. To work around this, I've added code to compensate for this scenario by injecting a null ImageDescriptor as a placeholder to allow the test to detect and skip execution in that case. I've also added some validation logic to ensure we only end up in that state for this expected scenario. This also applies to the SDK-only tests.