search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.04k stars 1.88k forks source link

PR Validation Flaky Test #6684

Open michaelgsharp opened 1 year ago

michaelgsharp commented 1 year ago

Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest is flaky on arm/arm64 devices. Disabling the test for now on non x86/x64 devices until we figure out the issue.

Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest [FAIL]
      Assert.Equal() Failure
      Expected: 0
      Actual:   2
      Stack Trace:
        /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs(201,0): at Microsoft.ML.RunTests.BaseTestBaseline.Done()
        /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs(156,0): at Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest()
      Output:
        Running 'MulticlassLogisticRegression' on 'iris'
          Running as: TrainTest tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} data=/root/helix/work/correlation/test/data/iris.txt seed=1 test=/root/helix/work/correlation/test/data/iris.txt xf=Term{col=Label} out={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip} dout={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt} norm=no
        maml.exe TrainTest test=/root/helix/work/correlation/test/data/iris.txt tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} norm=No dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt data=/root/helix/work/correlation/test/data/iris.txt out=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip seed=1 xf=Term{col=Label}
        
        
        Not adding a normalizer.
        
        
        Beginning optimization
        
        
        num vars: 15
        
        
        improvement criterion: Mean Improvement
        
        
        L1 regularization selected 15 of 15 weights.
        
        
        Not training a calibrator because it is not needed.
        
        
        
        
        Confusion table
        
        
                  ||========================
        
        
        PREDICTED ||     0 |     1 |     2 | Recall
        
        
        TRUTH     ||========================
        
        
                0 ||    50 |     0 |     0 | 1.0000
        
        
                1 ||     0 |    48 |     2 | 0.9600
        
        
                2 ||     0 |     1 |    49 | 0.9800
        
        
                  ||========================
        
        
        Precision ||1.0000 |0.9796 |0.9608 |
        
        
        Accuracy(micro-avg): 0.980000
        
        
        Accuracy(macro-avg): 0.980000
        
        
        Log-loss:           0.072171
        
        
        Log-loss reduction: 0.934307
        
        
        
        
        OVERALL RESULTS
        
        
        ---------------------------------------
        
        
        Accuracy(micro-avg): 0.980000 (0.0000)
        
        
        Accuracy(macro-avg): 0.980000 (0.0000)
        
        
        Log-loss:           0.072171 (0.0000)
        
        
        Log-loss reduction: 0.934307 (0.0000)
        
        
        
        
        ---------------------------------------
        
        
        Physical memory usage(MB): 120
        
        
        Virtual memory usage(MB): 3549
        
        
        5/15/2023 5:29:00 PM     Time elapsed(s): 0.028
        
        
        
        
        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt
        Output matches baseline: 'MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt'
        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/netcoreapp/MulticlassLogisticRegression-TrainTest-iris.txt
        Output matches baseline: 'MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt'
        maml.exe Test dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt data=/root/helix/work/correlation/test/data/iris.txt in=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip seed=1
        
        
        
        
        Confusion table
        
        
                  ||========================
        
        
        PREDICTED ||     0 |     1 |     2 | Recall
        
        
        TRUTH     ||========================
        
        
                0 ||    50 |     0 |     0 | 1.0000
        
        
                1 ||     0 |    48 |     2 | 0.9600
        
        
                2 ||     0 |     1 |    49 | 0.9800
        
        
                  ||========================
        
        
        Precision ||1.0000 |0.9796 |0.9608 |
        
        
        Accuracy(micro-avg): 0.980000
        
        
        Accuracy(macro-avg): 0.980000
        
        
        Log-loss:           0.072171
        
        
        Log-loss reduction: 0.934307
        
        
        
        
        OVERALL RESULTS
        
        
        ---------------------------------------
        
        
        Accuracy(micro-avg): 0.980000 (0.0000)
        
        
        Accuracy(macro-avg): 0.980000 (0.0000)
        
        
        Log-loss:           0.072171 (0.0000)
        
        
        Log-loss reduction: 0.934307 (0.0000)
        
        
        
        
        ---------------------------------------
        
        
        Physical memory usage(MB): 120
        
        
        Virtual memory usage(MB): 3549
        
        
        5/15/2023 5:29:00 PM     Time elapsed(s): 0.008
        
        
        
        
        Suffix of length 27 compared against sequence of length 34
        Running 'MulticlassLogisticRegression' on 'iris'
          Running as: CV tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} data=/root/helix/work/correlation/test/data/iris.txt seed=1 xf=Term{col=Label} dout={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt} norm=no threads-
        maml.exe CV tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} threads=- norm=No dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt data=/root/helix/work/correlation/test/data/iris.txt seed=1 xf=Term{col=Label}
        
        
        Not adding a normalizer.
        
        
        Not adding a normalizer.
        
        
        Beginning optimization
        
        
        num vars: 15
        
        
        improvement criterion: Mean Improvement
        
        
        L1 regularization selected 15 of 15 weights.
        
        
        Not training a calibrator because it is not needed.
        
        
        Physical memory usage(MB): 120
        
        
        Virtual memory usage(MB): 3549
        
        
        5/15/2023 5:29:00 PM     Time elapsed(s): 0.018
        
        
        
        
        ***** Unexpected failure. Please refer to https://aka.ms/MLNetIssue to file an issue with details *****
        
        
        ***** Error log has been saved to '/tmp/TLC/Error_20230515_172900_e06f381a-3a62-4df3-b493-09ff5a4df87d.log', please refer to https://aka.ms/MLNetIssue to file an issue with details *****
        
        
        ===== Begin detailed dump =====
        
        
        (1) Unexpected exception: One or more errors occurred. (Assert failed: Assertion Failed
        
        
        Expected: True
        
        
        Actual:   False), 'System.AggregateException'
        
        
           at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
        
        
           at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
        
        
           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.GetCrossValidationTasks() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 471
        
        
           at Microsoft.ML.Data.CrossValidationCommand.RunCore(IChannel ch, String cmd) in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 202
        
        
           at Microsoft.ML.Data.CrossValidationCommand.Run() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 134
        
        
           at Microsoft.ML.Tools.Maml.MainCore(IHostEnvironment env, String args, Boolean alwaysPrintStacktrace) in /__w/1/s/src/Microsoft.ML.Maml/MAML.cs:line 130
        
        
        (2) Unexpected exception: Assert failed: Assertion Failed
        
        
        Expected: True
        
        
        Actual:   False, 'Xunit.Sdk.TrueException'
        
        
           at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
        
        
           at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
        
        
           at Microsoft.ML.Internal.Internallearn.Test.GlobalBase.AssertHandler(String msg, IExceptionContext ectx) in /__w/1/s/test/Microsoft.ML.TestFramework/GlobalBase.cs:line 80
        
        
           at Microsoft.ML.Runtime.Contracts.DbgFailCore(String msg, IExceptionContext ctx) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 781
        
        
           at Microsoft.ML.Runtime.Contracts.DbgFail(IExceptionContext ctx) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 790
        
        
           at Microsoft.ML.Runtime.Contracts.Assert(IExceptionContext ctx, Boolean f) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 845
        
        
           at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplVec`1.Fetch(Int32 idx, VBuffer`1& value) in /__w/1/s/src/Microsoft.ML.Data/DataView/CacheDataView.cs:line 1389
        
        
           at Microsoft.ML.Data.CacheDataView.RowCursor`1.<>c__DisplayClass8_0`1.<CreateGetterDelegateCore>b__0(TValue& value) in /__w/1/s/src/Microsoft.ML.Data/DataView/CacheDataView.cs:line 496
        
        
           at Microsoft.ML.Trainers.FeatureFloatVectorCursor.Accept() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 855
        
        
           at Microsoft.ML.Trainers.FloatLabelCursor.Accept() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 928
        
        
           at Microsoft.ML.Trainers.TrainingCursorBase.MoveNext() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 555
        
        
           at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainCore(IChannel ch, RoleMappedData data) in /__w/1/s/src/Microsoft.ML.StandardTrainers/Standard/LogisticRegression/LbfgsPredictorBase.cs:line 605
        
        
           at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainModelCore(TrainContext context) in /__w/1/s/src/Microsoft.ML.StandardTrainers/Standard/LogisticRegression/LbfgsPredictorBase.cs:line 458
        
        
           at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Microsoft.ML.ITrainer<Microsoft.ML.IPredictor>.Train(TrainContext context) in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerEstimatorBase.cs:line 100
        
        
           at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Microsoft.ML.ITrainer.Train(TrainContext context) in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerEstimatorBase.cs:line 168
        
        
           at Microsoft.ML.Data.TrainUtils.TrainCore(IHostEnvironment env, IChannel ch, RoleMappedData data, ITrainer trainer, RoleMappedData validData, IComponentFactory`1 calibrator, Int32 maxCalibrationExamples, Nullable`1 cacheData, IPredictor inputPredictor, RoleMappedData testData) in /__w/1/s/src/Microsoft.ML.Data/Commands/TrainCommand.cs:line 280
        
        
           at Microsoft.ML.Data.TrainUtils.Train(IHostEnvironment env, IChannel ch, RoleMappedData data, ITrainer trainer, RoleMappedData validData, IComponentFactory`1 calibrator, Int32 maxCalibrationExamples, Nullable`1 cacheData, IPredictor inputPredictor, RoleMappedData testData) in /__w/1/s/src/Microsoft.ML.Data/Commands/TrainCommand.cs:line 255
        
        
           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.RunFold(Int32 fold) in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 523
        
        
           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.<>c__DisplayClass24_0.<GetCrossValidationTasks>b__0() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 463
        
        
           at System.Threading.Tasks.Task`1.InnerInvoke()
        
        
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
        
        
        --- End of stack trace from previous location ---
        
        
           at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
        
        
        ====== End detailed dump =====
        
        
        *** Predictor returned -2
        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris-out.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/netcoreapp/MulticlassLogisticRegression-CV-iris-out.txt
        *** Failure #1: Output and baseline mismatch at line 3, expected 'Beginning optimization' but got 'Not adding a normalizer.' : 'MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris-out.txt'
                Void Fail(System.String, System.Object[]) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 224
                Boolean CheckEqualityFromPathsCore(System.String, System.String, System.String, Int32, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 546
                Boolean CheckEqualityCore(System.String, System.String, System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 409
                Boolean CheckEqualityNormalized(System.String, System.String, System.String, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 377
                Boolean CheckEqualityNormalized(Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs 79
                Void Run(RunContext, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 169
                Void Run_CV(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 442
                Void RunOneAllTests(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 363
                Void MulticlassLRTest() /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs 155
                System.Object InvokeMethod(System.Object, System.Span`1[System.Object] ByRef, System.Signature, Boolean, Boolean)  0
                System.Object Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)  0
                System.Object CallTestMethod(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 150
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 257
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task <InvokeTestMethodAsync>b__1()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs 48
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task AggregateAsync(System.Func`1[System.Threading.Tasks.Task])  0
                System.Threading.Tasks.Task <InvokeTestMethodAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 242
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 90
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task RunAsync(System.Func`1[System.Threading.Tasks.Task])  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 241
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object)  0
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestInvoker.cs 112
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 206
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Decimal] <RunAsync>b__47_0()  0
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
                System.Threading.Tasks.Task`1[System.Decimal] RunAsync() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 189
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 84
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 67
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)  0
                System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] <RunAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs 149
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
                Void MoveNext()  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs 82
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 136
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCasesAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 106
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 213
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestMethodsAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 171
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase]) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs 158
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 130
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassesAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 101
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase], System.Threading.CancellationTokenSource) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 235
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] <RunTestCollectionsAsync>b__2() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 184
                Void InnerInvoke()  0
                Void <.cctor>b__272_0(System.Object)  0
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
                Boolean ExecuteEntry()  0
                Void <.cctor>b__8_0(System.Object)  0
                Void RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 106
                Void <WorkerThreadProc>b__0(System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void WorkerThreadProc() /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
                Void <QueueUserWorkItem>b__5_0(System.Object) /_/src/common/XunitWorkerThread.cs 37
                Void InnerInvoke()  0
                Void <.cctor>b__272_0(System.Object)  0
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
                Void ExecuteEntryUnsafe(System.Threading.Thread)  0
                Void <.cctor>b__10_0(System.Object)  0
                Void StartCallback()  0
        *** Failure #2: Output file not found: /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt
                Void Fail(System.String, System.Object[]) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 224
                Boolean CheckOutFile(System.String) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 922
                Boolean CheckEqualityCore(System.String, System.String, System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 398
                Boolean CheckEquality(System.String, System.String, System.String, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 367
                Boolean CheckEquality(Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs 73
                Void Run(RunContext, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 218
                Void Run_CV(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 442
                Void RunOneAllTests(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 363
                Void MulticlassLRTest() /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs 155
                System.Object InvokeMethod(System.Object, System.Span`1[System.Object] ByRef, System.Signature, Boolean, Boolean)  0
                System.Object Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)  0
                System.Object CallTestMethod(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 150
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 257
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task <InvokeTestMethodAsync>b__1()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs 48
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task AggregateAsync(System.Func`1[System.Threading.Tasks.Task])  0
                System.Threading.Tasks.Task <InvokeTestMethodAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 242
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 90
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task RunAsync(System.Func`1[System.Threading.Tasks.Task])  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 241
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object)  0
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestInvoker.cs 112
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 206
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Decimal] <RunAsync>b__47_0()  0
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
                System.Threading.Tasks.Task`1[System.Decimal] RunAsync() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 189
                System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 84
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 67
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)  0
                System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] <RunAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs 149
                Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
                Void MoveNext()  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs 82
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 136
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCasesAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 106
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 213
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestMethodsAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 171
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase]) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs 158
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 130
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassesAsync()  0
                Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 101
                Void Start[TStateMachine](TStateMachine ByRef)  0
                Void Start[TStateMachine](TStateMachine ByRef)  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase], System.Threading.CancellationTokenSource) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 235
                System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] <RunTestCollectionsAsync>b__2() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 184
                Void InnerInvoke()  0
                Void <.cctor>b__272_0(System.Object)  0
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
                Boolean ExecuteEntry()  0
                Void <.cctor>b__8_0(System.Object)  0
                Void RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 106
                Void <WorkerThreadProc>b__0(System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void WorkerThreadProc() /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
                Void <QueueUserWorkItem>b__5_0(System.Object) /_/src/common/XunitWorkerThread.cs 37
                Void InnerInvoke()  0
                Void <.cctor>b__272_0(System.Object)  0
                Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
                Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
                Void ExecuteEntryUnsafe(System.Threading.Thread)  0
                Void <.cctor>b__10_0(System.Object)  0
                Void StartCallback()  0
        Test MulticlassLRTest: completed normally: failed
        Test MulticlassLRTest is using netcoreapp configuration specific baselines.
michaelgsharp commented 1 year ago

Also disabling TestGridSearchTrialRunner2 for same reason.