search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.04k stars 1.89k forks source link

Legacy tests - partially disabled tests #5139

Closed frank-dong-ms-zz closed 4 years ago

frank-dong-ms-zz commented 4 years ago

We have some tests that are partially disabled based on OS, arch or netcore version, resolve them or at least root cause them and let test running on all platform.

Test Category Status Owner
MulticlassTreeFeaturizedLRTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
LinearClassifierTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
BinaryClassifierLogisticRegressionTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
BinaryClassifierSymSgdTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
BinaryClassifierTesterThresholdingTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
FastForestClassificationTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
FastTreeBinaryClassificationTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
FastTreeBinaryClassificationCategoricalSplitTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
FastTreeBinaryClassificationNoOpGroupIdTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
FastTreeHighMinDocsTest [X64Fact("x86 output differs from Baseline")] resolve Mustafa
CommandCrossValidationKeyLabelWithFloatKeyValues [X64Fact("x86 output differs from Baseline")] resolve Mustafa
TestTreeEnsembleCombiner [X64Fact("x86 fails. Associated GitHubIssue: https://github.com/dotnet/machinelearning/issues/1216")] resolve Frank
TestTreeEnsembleCombinerWithCategoricalSplits [X64Fact("x86 fails. Associated GitHubIssue: https://github.com/dotnet/machinelearning/issues/1216")] resolve Frank
TestEnsembleCombiner [X64Fact("x86 fails. Associated GitHubIssue: https://github.com/dotnet/machinelearning/issues/1216")] resolve Frank
TestMulticlassEnsembleCombiner [X64Fact("x86 fails. Associated GitHubIssue: https://github.com/dotnet/machinelearning/issues/1216")] TextLoader is throwing exception: Unhandled exception at 0x15B296B3 (coreclr.dll) in dotnet.exe.14324.dmp: 0xC0000005: Access violation writing location 0x17A41000. resolve Frank
ExprBind [X64Fact("sin(1e+30) gives different value on x86."), TestCategory("Expr Language")] Resolve   Math.Sin has valid value range from approximately -9223372036854775295 to approximately 9223372036854775295: https://docs.microsoft.com/en-us/dotnet/api/system.math.sin?view=netcore-3.1. If value is out of range, Math.Sin will return value passed in and not throwing exception.   Below is some test result:   Math.Sin(1e+30):   NetCoreApp X64: 0.0093314689311758247 NetCoreApp X86: -0.75626273033357649 Net FX: 1e+30     Math.Sin(1e+10):   NetCoreApp X64: -0.48750602508751067 NetCoreApp X86: -0.48750602507627 Net FX:         -0.48750602507627     So this is acceptable to skip in x86 and net framework, but we should consider to use Math.Sin within its valid range. Frank
EntryPointPipelineEnsembleGetSummary [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
TestCrossValidationMacro [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
MulticlassLRTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
MulticlassLRNonNegativeTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
BinaryClassifierLogisticRegressionBinNormTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
DefaultCalibratorPerceptronTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
PAVCalibratorPerceptronTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
BinaryClassifierLDSvmTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
BinaryClassifierLDSvmNoBiasTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
CommandTrainMlrWithStats [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
KmeansOnnxConversionTest [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
SavePipeSsaSpikeNoData [LessThanNetCore30OrNotNetCoreFact("netcoreapp3.1 output differs from Baseline")] resolve Frank
EnsemblesMultiClassBootstrapSelectorTest [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
EnsemblesMultiAveragerTest [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
EnsemblesMultiVotingCombinerTest [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
EnsemblesMultiStackCombinerTest [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
ChangePointDetectionWithSeasonality [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
ChangePointDetectionWithSeasonalityPredictionEngineNoColumn [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
ChangePointDetectionWithSeasonalityPredictionEngine [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
SsaForecast [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
SsaForecastPredictionEngine [LessThanNetCore30OrNotNetCoreFact("output on .NetCore 3.0 differs. Tracked on issue 3856 in GitHub.")] resolve Frank
BinaryClassifierLogisticRegressionNonNegativeTest [LessThanNetCore30OrNotNetCoreAndX64Fact("netcoreapp3.1 and x86 output differs from Baseline")] resolve Frank
BinaryClassifierLogisticRegressionGaussianNormTest [LessThanNetCore30OrNotNetCoreAndX64Fact("netcoreapp3.1 and x86 output differs from Baseline")] resolve Frank
RandomCalibratorPerceptronTest [LessThanNetCore30OrNotNetCoreAndX64Fact("netcoreapp3.1 and x86 output differs from Baseline")] resolve Frank
BinaryClassifierSymSgdTest RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms Root cause: Linux uses a version of MKL that doesn't support conditional numerical reproducibility the same way as Windows runs.   The different during different OS as well as unstable result for Linux is from MKL library (method cblas_sdot). Frank
CommandTrainingBinaryFactorizationMachineWithValidation RuntimeInformation.IsOSPlatform Resolve - Lower precision on Linux platforms   This test is skipped due to missing dataset, also Linux uses lower precision due to below issue: https://github.com/dotnet/machinelearning/issues/404 Frank
CommandTrainingBinaryFactorizationMachineWithValidationAndInitialization RuntimeInformation.IsOSPlatform Resolve - Lower precision on Linux platforms   Linux uses lower precision due to below issue and PR: https://github.com/dotnet/machinelearning/issues/404 https://github.com/dotnet/machinelearning/pull/1206 Frank
IrisLightGbm RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms resolve PR #5080 Mustafa
IrisLightGbmWithLoadColumnName RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms Resolve PR #5080 Mustafa
IrisVectorLightGbm RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms Resolve PR #5080 Mustafa
IrisVectorLightGbmWithLoadColumnName RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms Resolve PR #5080 Mustafa
IrisSdcaMaximumEntropy RuntimeInformation.IsOSPlatform Skipped on Non-Windows platforms Resolve PR #5080 Mustafa
TextNormalizingOnnxConversionTest RuntimeInformation.IsOSPlatform Skipped on Linux platforms   Skipped due to below locale exception: Failed to construct locale with name:en_US.UTF-8:locale::facet::_S_create_c_locale name not valid:Please, install necessary language-pack-XX and configure locales   https://github.com/dotnet/machinelearning/issues/5093   Mustafa will work on a fix later Mustafa
TensorFlowTransformCifar RuntimeInformation.IsOSPlatform Resolved - Different expected results between Windows vs. Linux/OSX, runs only on 64-bit due to TF   By Design, this difference is explained below:   // taco_invalidpixelformat.jpg has '8207' pixel format on Windows but this format translates to Format32bppRgb             // on macOS and Linux, hence on Windows this image's pixel format is converted in resize transformer to Format32bppArgb             // and on linux and macOS it is not converted in resize transform since pixel format 'Format32bppRgb' can be resized but             // in ImagePixelExtractingTransformer it is converted to Format32bppArgb since there we just support two             // pixel formats, i.e Format32bppArgb and Format16bppArgb. Frank
TensorFlowImageClassificationDefault RuntimeInformation.IsOSPlatform Resolved – already runs on all OS's, runs only on 64-bit due to TF Mustafa
TensorFlowImageClassification RuntimeInformation.IsOSPlatform Resolved – already runs on all OS's, runs only on 64-bit due to TF Mustafa
MatrixFactorizationSimpleTrainAndPredict RuntimeInformation.IsOSPlatform Lower precision on Linux, skipped on OSX Mustafa Mustafa
OneClassMatrixFactorizationInMemoryDataZeroBaseIndex RuntimeInformation.IsOSPlatform Resolved - Lower precision on non-Windows platforms   There are randomness during initialization in native matrix factorization library(default_random_engine). This default_random_engine seems have different implementation on different OS. https://stackoverflow.com/questions/32730906/random-generates-same-number-in-linux-but-not-in-windows/32731387 Frank
MulticlassLRTest Debug Release Different Baseline resolve Mustafa
RandomCalibratorPerceptronTest Debug Release Different Baseline resolve Mustafa
LinearClassifierTest Debug Release Different Baseline resolve Mustafa
MulticlassTreeFeaturizedLRTest Debug Release Different Baseline resolve Mustafa
FastForestClassificationTest Debug Release Different Baseline resolve Mustafa
BinaryClassifierLogisticRegressionNormTest Debug Release Different Baseline resolve Mustafa
frank-dong-ms-zz commented 4 years ago

Thanks to @mstfbl all these legacy tests have been resolved. CC @harishsk