search

dotnet / machinelearning-modelbuilder

Simple UI tool to build custom machine learning models.
Creative Commons Attribution 4.0 International
262 stars 54 forks source link

Exception has been thrown by the target of an invocation. #2716

Open Symbai opened 1 year ago

Symbai commented 1 year ago

System Information (please complete the following information):

Describe the bug

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 154   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

To Reproduce

  1. I created a console project using .NET 7
  2. I added a ML scenario for "Text Classification"
  3. I added my CSV data (only 10 rows)
  4. I selected the columns
  5. I clicked on "Train"
  6. A popup appears showing me the training time etc. looks all good
  7. Exception occurs

Expected behavior I expect to proceed to "Evaluate" step. If I manually go to this tab and press the "Predict" button, I get the same exception as above.

Screenshots

Additional context Last log entry in VS is Generate code behind files

LittleLittleCloud commented 1 year ago

Are you training on GPU, and would you mind sharing your mbconfig

Symbai commented 1 year ago

I was testing both

mbconfig:

{
  "Scenario": "TextClassification",
  "DataSource": {
    "Type": "TabularFile",
    "Version": 1,
    "FilePath": "C:\\Users\\User\\Desktop\\test.csv",
    "Delimiter": ",",
    "DecimalMarker": ".",
    "HasHeader": false,
    "ColumnProperties": [
      {
        "ColumnName": "col0",
        "ColumnPurpose": "Label",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      },
      {
        "ColumnName": "col1",
        "ColumnPurpose": "Text",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      }
    ]
  },
  "Environment": {
    "Type": "LocalCPU",
    "Version": 1
  },
  "RunHistory": {
    "Version": 3,
    "Type": "Result",
    "Trials": [
      {
        "Version": 1,
        "Type": "Trial",
        "TrainerName": "TextClassificationMulti",
        "Score": 0.1,
        "RuntimeInSeconds": 5.194,
        "Parameter": {
          "_SCHEMA_": "e0 * e1 * e2",
          "e0": {
            "OutputColumnName": "col0",
            "InputColumnName": "col0",
            "AddKeyValueAnnotationsAsText": false
          },
          "e1": {
            "LabelColumnName": "col0",
            "Sentence1ColumnName": "col1",
            "ScoreColumnName": "Score",
            "OutputColumnName": "PredictedLabel",
            "BatchSize": 32,
            "MaxEpochs": 10,
            "Architecture": "Roberta"
          },
          "e2": {
            "OutputColumnName": "PredictedLabel",
            "InputColumnName": "PredictedLabel"
          }
        }
      }
    ],
    "Estimators": {
      "e0": "MapValueToKey",
      "e1": "TextClassificationMulti",
      "e2": "MapKeyToValue"
    },
    "Schema": "e0 * e1 * e2",
    "MetricName": "MacroAccuracy",
    "ModelFilePath": "C:\\Users\\User\\source\\repos\\MlTest\\MlTest\\MLModel2.mlnet"
  },
  "Type": "TrainingConfig",
  "Version": 4,
  "TrainingOption": {
    "Version": 1,
    "Type": "TextClassficationTrainingOption",
    "TrainingTime": 2147483647,
    "LabelColumn": "col0",
    "TextColumn": "col1",
    "ValidationOption": {
      "Version": 0,
      "Type": "CrossValidationValidationOption",
      "NumberOfFolds": 5
    }
  }
}

https://github.com/dotnet/machinelearning-modelbuilder/assets/14368203/7b84ff3d-c6df-40d5-a224-7a0e8a183fea

LittleLittleCloud commented 1 year ago

Yea please. Thanks!

daikoz commented 1 year ago

same issue with same scenario with: ML.NET version 17.17.0.2332602 Microsoft Visual Studio 2022 Version 17.6.4

Symbai commented 11 months ago

I am just experiencing it again on a different project. All I did was choosing a new TSV file and clicked on 'Train again'. But I also created a brand new project and got the same immediately on first train. Here is some updated info:

Exception:

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String trainingConfigurationFolder, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 155   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

VS Log:

Set log file path to C:\Users\XXX\AppData\Local\Temp\MLVSTools\logs\MLModel1-YC9ZBU.txt
start text classification 
restore "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" --configfile "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\NuGet.config" -r win-x64 /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj"
publish "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" -r win-x64 -c Release --no-self-contained -o "C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5" --no-restore /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\\" /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj\\"
start installing runtime in C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5
  Determining projects to restore...
  Restored C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj (in 418 ms).
MSBuild version 17.8.0-preview-23418-03+0125fc9fb for .NET
C:\Program Files\dotnet\sdk\8.0.100-rc.1.23455.8\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj]
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\Release\netstandard2.0\win-x64\torchsharp.gpu.dll
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\
install runtime successfully
Use cross validation with fold: 5
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
|                          Experiment Results                        |
|--------------------------------------------------------------------|
|                               Summary                              |
|--------------------------------------------------------------------|
|ML Task: text classification                                        |
|Dataset: H:\xxxx.tsv                                        |
|Label : col0                                                        |
|Total experiment time :   463,1580 Secs                             |
|Total number of models explored: 1                                  |
|--------------------------------------------------------------------|
|                        Top 1 models explored                       |
|--------------------------------------------------------------------|
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
Generate code behind files

Log file:

(Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update TrainViewModel_163.TrainingDidNotSucceed (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TryOutRow (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TopFiveRecommendationLabel (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutItemIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutUserIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG fail calling update handler GetSampleCodeSnippetAsync (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG Exception has been thrown by the target of an invocation. (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG    at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ToolWindows.ConsumeViewModel.<GetSampleCodeSnippetAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<>c__DisplayClass15_0.<<Initialize>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<TryExecuteNotifyDependencyNodeDelegateAsync>d__20.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

image image

LittleLittleCloud commented 11 months ago

Looks like you are running with cross-validation? The fix for cross-validation hasn't been fix yet so you might need to use another validation strategy

Symbai commented 11 months ago

Unbelievable, you're right. I switched from cross-validation (which seems to be the default option because I never changed that) to split-validation and this fixed it. But cross-validation seems to be the better one, any ETA when it will be fixed? Or can you at least say if the team is working on it?

//edit: With the first run using split-validation the results were acceptable. Now I've trained again and I get completely different prediction results comparing to the first run. Not just slightly different values but completely(!) different results. Even though the data which is used for training hasn't changed.

LittleLittleCloud commented 11 months ago

@Symbai Actually, looks like the issue has already been fixed by me,,,, can you try upgrade model builder to v17.17.* and give it another trial?

Symbai commented 11 months ago

The VS extension I have says its the latest version: 17.17.0.2332602 see screenshot in my previous post https://github.com/dotnet/machinelearning-modelbuilder/issues/2716#issuecomment-1737813544

//edit: Reinstalled the extension but still the same

LittleLittleCloud commented 11 months ago

OK, let me double check then

Symbai commented 10 months ago

@LittleLittleCloud any update?

LittleLittleCloud commented 10 months ago

@Symbai No haven't yet, I was focusing on other stuff the last week. Will get the update back to this thread once I have any.

Also feel free to ping me for reminder

hannespreishuber commented 8 months ago
same here Use cross validation with fold: 5 Trainer MicroAccuracy Duration
0 TextClassificationMulti 0,3045 65,9930