JakobL-MSFT
commented
4 months ago I tested this further. Here are results. Essence: Works really well.
Here is what I ran:
Build-Samples $ewdk_ge_release_eeap_26058 "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_ge_release_eeap_26063 "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_ge_release_latest "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_ge_release_sigma "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_ge_release_sigma_dev "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_rs_prerelease "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_rs_sparc "user/jakobl/build_catch_sporadic_failures"
Build-Samples $ewdk_ge_release_prs "user/jakobl/build_catch_sporadic_failures"So 8 full builds. Each with 130 drivers and 4 flavors. That is 520 combinations. Total of 4120 combinations across all 8 EWDKs.
Sporadic Failures:
- Host x64: I observed 3 Sporadic failures (0.07%) each on form 'error MSB6003: The specified task executable "foo.exe" could not be run. System.IO.IOException: The requested operation cannot be performed on a file with a user-mapped section open.'
- Host arm64: I observed 0 Sporadic failures.
Actual Failures:
- I observed a total of 0 Failures.
Runtime:
- x64 desktop: 163 minutes or 20 minutes per EWDK or 2.3 second per build combination.
- arm64 8280 laptop: 318 minutes or 40 minutes per EWDK or 4.6 second per build combination.
Log for Host x64:
...
WDSTest: 2024-02-20 18:57:34: Build-Samples EWDK_ge_release_26058_240209-1555 user/jakobl/build_catch_sporadic_failures {
Build Environment: (8 different EWDKs)
Build Number: (Varies)
Samples: 130
Configurations: 2 (Debug Release)
Platforms: 2 (x64 arm64)
InfVerif_AdditionalOptions: /samples
Combinations: 520
LogicalProcessors: 12
ThrottleFactor: 5
ThrottleLimit: 60
...
Failed: 0
Sporadic: 0
...
Build sporadic errors in Sample storage.class.classpnp; Configuration: Release; Platform: arm64 {
D:\wdstest\wdstest-tmp\ewdk\EWDK_ge_release_26063_240216-1326\Program Files\Windows Kits\10\build\10.0.26063.0\WindowsDriver.common.targets(937,5): error MSB6003: The specified task executable "tracewpp.exe" could not be run. System.IO.IOException: The requested operation cannot be performed on a file with a user-mapped section open. [D:\wdstest\wdstest-tmp\run\storage\class\classpnp\src\classpnp.vcxproj]
...
Failed: 0
Sporadic: 1
---
Failed: 0
Sporadic: 0
---
Failed: 0
Sporadic: 0
---
Build sporadic errors in Sample wmi.wmisamp; Configuration: Debug; Platform: x64 {
D:\wdstest\wdstest-tmp\ewdk\EWDK_ge_release_sigma_dev_26062_240216-1700\Program Files\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(2027,5): error MSB6003: The specified task executable "rc.exe" could not be run. System.IO.IOException: The requested operation cannot be performed on a file with a user-mapped section open. [D:\wdstest\wdstest-tmp\run\wmi\wmisamp\WmiSamp.vcxproj]
...
Failed: 0
Sporadic: 1
...
Build sporadic errors in Sample nfp.net; Configuration: Release; Platform: x64 {
D:\wdstest\wdstest-tmp\ewdk\EWDK_rs_prerelease_27562_240216-1514\Program Files\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(1151,5): error MSB6003: The specified task executable "link.exe" could not be run. System.IO.IOException: The requested operation cannot be performed on a file with a user-mapped section open. [D:\wdstest\wdstest-tmp\run\nfp\net\exe\NetNfpControl.vcxproj]
...
Failed: 0
Sporadic: 1
...
Failed: 0
Sporadic: 0
...
Failed: 0
Sporadic: 0
...
WDSTest: 2024-02-20 21:41:02: Build-Samples EWDK_ge_release_prs_26063_240216-1342 user/jakobl/build_catch_sporadic_failures }
This PR enable retry logic to catch sporadic failures.
We /can/ do this now because samples in general are building extremely cleanly and Failures are mostly unexpected events. (If we expect lots of failures, then retrying would take a lot of time)
We /should/ do this now because we have dramatically extended the number of combinations each build takes and the number of branches we test against. What is perhaps only happening in 1 of 5000 combinations will statistically happens once if you build 125 drivers across 4 flavors across 10 builds.
How it works: Specifically, whenever a build failure occur for a given (Driver, Configuration, Platform) combination (say, video/indirectdisplay, Debug, arm64) the tool will re-run up to three times. If all three builds results in failure, then the tool will mark the combination as indeed a failure. If, however, on the second or third attempt, a successful build occurs, then the tool will mark the combination "Sporadic".
Note: We should still pursue "Sporadic" failures. This PR helps isolate these from true Failures and allows other progress to me made.
Example:
(Assume 3 passing samples, 1 sample with a Failure, and 1 sample with a Sporadic failure)