search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.91k stars 1.86k forks source link

Helix test failures are double counted #7044

Open ericstj opened 3 months ago

ericstj commented 3 months ago

Describe the bug When a helix test fails gracefully the failure should be reported and the runner should return 0.

To Reproduce Examine a normal test failure for a build. You'll see the test failure reported and the "work item" for the test. The latter is reported as a failure since the runner returned a non-zero exit value.

Expected behavior Only the failing test is reported.

Screenshots, Code, Sample Projects image.

Example build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=583517&view=ms.vss-test-web.build-test-results-tab&runId=14034000&resultId=100083&paneView=debug

ericstj commented 3 months ago

Looks like xunit.runner will return either 0 or 1 for normal cases: https://github.com/xunit/xunit/blob/c10b7a2884edcb69a4afe0af6bf241f5565e99e9/src/xunit.console/ConsoleRunner.cs#L101C24-L101C33

I think we just need to swallow the 1 value.