search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.05k stars 1.89k forks source link

I cannot save the model for some weird reason. #5030

Closed coutaq closed 4 years ago

coutaq commented 4 years ago

System information

Issue

-Here's the error code: `System.ArgumentException HResult=0x80070057 Message=The path is not of a legal form. Source=mscorlib StackTrace: at System.IO.Path.NewNormalizePath(String path, Int32 maxPathLength, Boolean expandShortPaths) at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths) at System.IO.Path.GetFullPathInternal(String path) at System.IO.Path.GetFullPath(String path) at System.Diagnostics.FileVersionInfo.GetFullPathWithAssert(String fileName) at System.Diagnostics.FileVersionInfo.GetVersionInfo(String fileName) at Microsoft.ML.RepositoryWriter.CreateNew(Stream stream, IExceptionContext ectx, Boolean useFileSystem) at Microsoft.ML.ModelOperationsCatalog.Save(ITransformer model, DataViewSchema inputSchema, Stream stream) at Microsoft.ML.ModelOperationsCatalog.Save(ITransformer model, DataViewSchema inputSchema, String filePath) at MachineLearningTest.ModelBuilder.SaveModel(MLContext mlContext, ITransformer mlModel, String modelRelativePath, DataViewSchema modelInputSchema) in C:\Users\Michael\source\repos\new\MachineLearingTest\ModelBuilder.cs:line 48 at MachineLearningTest.ModelBuilder.CreateModel() in C:\Users\Michael\source\repos\new\MachineLearingTest\ModelBuilder.cs:line 43 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

This exception was originally thrown at this call stack: System.IO.Path.NewNormalizePath(string, int, bool) System.IO.Path.NormalizePath(string, bool, int, bool) System.IO.Path.GetFullPathInternal(string) System.IO.Path.GetFullPath(string) System.Diagnostics.FileVersionInfo.GetFullPathWithAssert(string) System.Diagnostics.FileVersionInfo.GetVersionInfo(string) Microsoft.ML.RepositoryWriter.CreateNew(System.IO.Stream, Microsoft.ML.Runtime.IExceptionContext, bool) Microsoft.ML.ModelOperationsCatalog.Save(Microsoft.ML.ITransformer, Microsoft.ML.DataViewSchema, System.IO.Stream) Microsoft.ML.ModelOperationsCatalog.Save(Microsoft.ML.ITransformer, Microsoft.ML.DataViewSchema, string) MachineLearningTest.ModelBuilder.SaveModel(Microsoft.ML.MLContext, Microsoft.ML.ITransformer, string, Microsoft.ML.DataViewSchema) in ModelBuilder.cs ... [Call Stack Truncated] `

Source code / logs

And here's the code: `// This file was auto-generated by ML.NET Model Builder.

using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Windows.Forms; using Microsoft.ML; using Microsoft.ML.Data; using Microsoft.ML.Trainers.LightGbm;

namespace MachineLearningTest { public static class ModelBuilder { private static string TRAIN_DATA_FILEPATH = Program.dataPath; private static string MODEL_FILEPATH = @"MLModel.zip"; // Create MLContext to be shared across the model creation workflow objects // Set a random seed for repeatable/deterministic results across multiple trainings. private static MLContext mlContext = new MLContext(seed: 1);

    public static void CreateModel()
    {
        // Load Data
        IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
                                        path: TRAIN_DATA_FILEPATH,
                                        hasHeader: true,
                                        separatorChar: ',',
                                        allowQuoting: true,
                                        allowSparse: false);

        // Build training pipeline
        IEstimator<ITransformer> trainingPipeline = BuildTrainingPipeline(mlContext);

        // Evaluate quality of Model
        //Evaluate(mlContext, trainingDataView, trainingPipeline);

        // Train Model
        ITransformer mlModel = TrainModel(mlContext, trainingDataView, trainingPipeline);

        // Save model
        SaveModel(mlContext, mlModel, MODEL_FILEPATH, trainingDataView.Schema);
    }
    private static void SaveModel(MLContext mlContext, ITransformer mlModel, string modelRelativePath, DataViewSchema modelInputSchema)
    {
        // This causes the exception
            mlContext.Model.Save(mlModel, modelInputSchema, (GetAbsolutePath(modelRelativePath)));
    }

    public static IEstimator<ITransformer> BuildTrainingPipeline(MLContext mlContext)
    {
        // Data process configuration with pipeline data transformations 
        var dataProcessPipeline = mlContext.Transforms.Conversion.MapValueToKey("White", "White")
                                  .Append(mlContext.Transforms.Concatenate("Features", new[] { "Hue", "Saturation", "Brightness" }));
        // Set the training algorithm 
        var trainer = mlContext.MulticlassClassification.Trainers.LightGbm(new LightGbmMulticlassTrainer.Options() { NumberOfIterations = 20, LearningRate = 0.05916024f, NumberOfLeaves = 4, MinimumExampleCountPerLeaf = 1, UseCategoricalSplit = false, HandleMissingValue = true, MinimumExampleCountPerGroup = 200, MaximumCategoricalSplitPointCount = 8, CategoricalSmoothing = 20, L2CategoricalRegularization = 1, UseSoftmax = true, Booster = new GradientBooster.Options() { L2Regularization = 0, L1Regularization = 0.5 }, LabelColumnName = "White", FeatureColumnName = "Features" })
                                  .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));

        var trainingPipeline = dataProcessPipeline.Append(trainer);

        return trainingPipeline;
    }

    public static ITransformer TrainModel(MLContext mlContext, IDataView trainingDataView, IEstimator<ITransformer> trainingPipeline)
    {
        Console.WriteLine("=============== Training  model ===============");
        ITransformer model = trainingPipeline.Fit(trainingDataView);

        Console.WriteLine("=============== End of training process ===============");
        return model;
    }

    private static void Evaluate(MLContext mlContext, IDataView trainingDataView, IEstimator<ITransformer> trainingPipeline)
    {
        // Cross-Validate with single dataset (since we don't have two datasets, one for training and for evaluate)
        // in order to evaluate and get the model's accuracy metrics
        Console.WriteLine("=============== Cross-validating to get model's accuracy metrics ===============");
        var crossValidationResults = mlContext.MulticlassClassification.CrossValidate(trainingDataView, trainingPipeline, numberOfFolds: 5, labelColumnName: "White");
        PrintMulticlassClassificationFoldsAverageMetrics(crossValidationResults);
    }

    public static string GetAbsolutePath(string relativePath)
    {
        FileInfo _dataRoot = new FileInfo(typeof(Program).Assembly.Location);
        string assemblyFolderPath = _dataRoot.Directory.FullName;

        string fullPath = Path.Combine(assemblyFolderPath, relativePath);
        return fullPath;
    }
    public static void PrintMulticlassClassificationMetrics(MulticlassClassificationMetrics metrics)
    {
        Console.WriteLine($"************************************************************");
        Console.WriteLine($"*    Metrics for multi-class classification model   ");
        Console.WriteLine($"*-----------------------------------------------------------");
        Console.WriteLine($"    MacroAccuracy = {metrics.MacroAccuracy:0.####}, a value between 0 and 1, the closer to 1, the better");
        Console.WriteLine($"    MicroAccuracy = {metrics.MicroAccuracy:0.####}, a value between 0 and 1, the closer to 1, the better");
        Console.WriteLine($"    LogLoss = {metrics.LogLoss:0.####}, the closer to 0, the better");
        for (int i = 0; i < metrics.PerClassLogLoss.Count; i++)
        {
            Console.WriteLine($"    LogLoss for class {i + 1} = {metrics.PerClassLogLoss[i]:0.####}, the closer to 0, the better");
        }
        Console.WriteLine($"************************************************************");
    }

    public static void PrintMulticlassClassificationFoldsAverageMetrics(IEnumerable<TrainCatalogBase.CrossValidationResult<MulticlassClassificationMetrics>> crossValResults)
    {
        var metricsInMultipleFolds = crossValResults.Select(r => r.Metrics);

        var microAccuracyValues = metricsInMultipleFolds.Select(m => m.MicroAccuracy);
        var microAccuracyAverage = microAccuracyValues.Average();
        var microAccuraciesStdDeviation = CalculateStandardDeviation(microAccuracyValues);
        var microAccuraciesConfidenceInterval95 = CalculateConfidenceInterval95(microAccuracyValues);

        var macroAccuracyValues = metricsInMultipleFolds.Select(m => m.MacroAccuracy);
        var macroAccuracyAverage = macroAccuracyValues.Average();
        var macroAccuraciesStdDeviation = CalculateStandardDeviation(macroAccuracyValues);
        var macroAccuraciesConfidenceInterval95 = CalculateConfidenceInterval95(macroAccuracyValues);

        var logLossValues = metricsInMultipleFolds.Select(m => m.LogLoss);
        var logLossAverage = logLossValues.Average();
        var logLossStdDeviation = CalculateStandardDeviation(logLossValues);
        var logLossConfidenceInterval95 = CalculateConfidenceInterval95(logLossValues);

        var logLossReductionValues = metricsInMultipleFolds.Select(m => m.LogLossReduction);
        var logLossReductionAverage = logLossReductionValues.Average();
        var logLossReductionStdDeviation = CalculateStandardDeviation(logLossReductionValues);
        var logLossReductionConfidenceInterval95 = CalculateConfidenceInterval95(logLossReductionValues);

        Console.WriteLine($"*************************************************************************************************************");
        Console.WriteLine($"*       Metrics for Multi-class Classification model      ");
        Console.WriteLine($"*------------------------------------------------------------------------------------------------------------");
        Console.WriteLine($"*       Average MicroAccuracy:    {microAccuracyAverage:0.###}  - Standard deviation: ({microAccuraciesStdDeviation:#.###})  - Confidence Interval 95%: ({microAccuraciesConfidenceInterval95:#.###})");
        Console.WriteLine($"*       Average MacroAccuracy:    {macroAccuracyAverage:0.###}  - Standard deviation: ({macroAccuraciesStdDeviation:#.###})  - Confidence Interval 95%: ({macroAccuraciesConfidenceInterval95:#.###})");
        Console.WriteLine($"*       Average LogLoss:          {logLossAverage:#.###}  - Standard deviation: ({logLossStdDeviation:#.###})  - Confidence Interval 95%: ({logLossConfidenceInterval95:#.###})");
        Console.WriteLine($"*       Average LogLossReduction: {logLossReductionAverage:#.###}  - Standard deviation: ({logLossReductionStdDeviation:#.###})  - Confidence Interval 95%: ({logLossReductionConfidenceInterval95:#.###})");
        Console.WriteLine($"*************************************************************************************************************");

    }

    public static double CalculateStandardDeviation(IEnumerable<double> values)
    {
        double average = values.Average();
        double sumOfSquaresOfDifferences = values.Select(val => (val - average) * (val - average)).Sum();
        double standardDeviation = Math.Sqrt(sumOfSquaresOfDifferences / (values.Count() - 1));
        return standardDeviation;
    }

    public static double CalculateConfidenceInterval95(IEnumerable<double> values)
    {
        double confidenceInterval95 = 1.96 * CalculateStandardDeviation(values) / Math.Sqrt((values.Count() - 1));
        return confidenceInterval95;
    }
}

} `

gvashishtha commented 4 years ago

Is it possible you are saving in a directory with a long name? I wonder if you are exceeding the maxPathLength parameter.

coutaq commented 4 years ago

Is it possible you are saving in a directory with a long name? I wonder if you are exceeding the maxPathLength parameter.

I don't think it is; I was saving to was C:\Users\Michael\source\repos\new\MachineLearingTest\bin\x64\Debug\MLModel.zip, but I just tried to save to D:\MLModel.zip and it still was 'illegal'.

gvashishtha commented 4 years ago

OK, can you share a link to some fileshare with your whole solution so I can try to reproduce the error?

coutaq commented 4 years ago

yes sure here you go https://github.com/coutaq/machine-learning-test

gvashishtha commented 4 years ago

OK, thank you. I was able to reproduce the issue. Looks like it in this line, the value of the RepositoryWriter assembly location is being passed as an empty string, which is causing the error you're seeing.

@antoniovs1029 can you help me go deeper here? Is this an expected issue?

antoniovs1029 commented 4 years ago

Hi, @coutaq . So I'm still not sure why you're getting this exception.

I noticed that if I revert the last commit on your repo, the model is correctly saved to disk and it's possible to load it back as well without any problem. In that last commit you actually didn't change anything in your ML.NET pipeline, so I think that the exception you're now getting isn't a bug in ML.NET but actually in the way things are being referenced and used on your latest commit.

It's still unclear to me what you're trying to do on your last commit, but here are some things I've noticed that you might want to explore to fix your exception:

  1. In your last commit you changed your dependency of <package id="Microsoft.ML.LightGbm" version="1.5.0-preview2" targetFramework="net472" /> to <package id="Microsoft.ML.LightGbm" version="1.4.0" targetFramework="net472" />

I don't know why that change was made, since all the other dependencies on Microsoft.ML nugets were left with version "1.5.0-preview2". Users shouldn't mix ML.NETs nugets of different releases (i.e. all of them should be either "1.4.0" or "1.5.0-preview2"). Mixing them up can cause different problems, including problems with saving and loading models (e.g. issue #4868)

I did try to bring back the Microsoft.ML.LightGbm nuget to version 1.5.0-preview2 on your project, but that didn't solve your exception. I did this by using the Nuget package manager, but I don't know if that would be enough, since I am unfamiliar to how you have set up your project, and I wouldn't know either how the other changes on your last commit affect this dependency. So please, try updating back this dependency in all places you consider necessary, and let us know if that works for you.

  1. It's strange to me that in the last commit it looks like the Native folder in ML.NET got added to your repository. Is this something you added manually? If so, what was your goal? In general, that folder doesn't need to be copied, and its contents don't need to be built by the user. When building ML.NET nugets we actually build those native libraries, and include the resulting DLLs inside the ML.NET's nugets, so there's no need for users to rebuild them. So it's unusual for a user to be building them on their project, and perhaps this is also related to your exception (and I wouldn't know if it's related to the point 1. I mentioned above).
coutaq commented 4 years ago

Hi there! I'm not entirely sure why I changed the ML.LightGbm's version, most likely by mistake for I am new to nuget and libraries in C#. I did download it using the package manager, and changing the versions to match does not seem to work for me either. The native folder is indeed from Microsoft.ML, I manually downloaded it to try to figure out what the hell was an 'illegal path' and what code threw that exception so I could work around it. I removed the native folder, updated all Microsoft.ML libraries to 1.5.0-preview2, and yet the path remains illegal. I don't really know where to go from here, what am I doing wrong??

antoniovs1029 commented 4 years ago

Mmm I see. Why don't you try reverting back to this other commit? https://github.com/coutaq/machine-learning-test/commit/ffefbd4c87428e1aa817e169f3fbf1cb8c6d51b8

As I mentioned earlier, I tried that and it saved the model correctly. It seems that between that commit and the next one (which broke saving the model) you didn't change much of your code, but you did changed several dependencies and downloaded the Native ML.NET's folders (which you didn't actually need). I would recommend reverting to that commit, confirm that the model is saved correctly, and then work from there adding whatever change you actually wanted to introduce in your project (without introducing the changes for the LightGBM nuget or the Native folders, for instance), doing small commits and checking that you're still able to save your model until you can point to the exact change that broke your ability to save the model.

Another thing that I've just noticed on the commit that broke your ability to save the model, is that on that commit you actually erased this file from your project: https://github.com/coutaq/machine-learning-test/blob/ffefbd4c87428e1aa817e169f3fbf1cb8c6d51b8/MachineLearingTest/Properties/AssemblyInfo.cs

The AssemblyInfo.cs contains important information about your assembly, and I don't know why would you erase it. As @gvashishtha pointed out yesterday, the exception you're getting from ML.NET comes from this line when retrieving information regarding an assembly. So I wonder if this file being deleted is related to your exception. If you don't want to revert to the previous commit I've pointed to, then I'd suggest bringing that file back and see if that fixes your issue.

frank-dong-ms-zz commented 4 years ago

@antoniovs1029 please follow up on this issue, if no new question from user you can close the issue.

antoniovs1029 commented 4 years ago

Hi, @coutaq . Since I haven't received a response after my last comment, I'll close this issue.

If you've found the change on your dependencies / project settings that started to cause this issue, please let us know and reopen this issue so that I can take a closer look. Thanks.