Open Symbai opened 1 year ago
Are you training on GPU, and would you mind sharing your mbconfig
I was testing both
{
"Scenario": "TextClassification",
"DataSource": {
"Type": "TabularFile",
"Version": 1,
"FilePath": "C:\\Users\\User\\Desktop\\test.csv",
"Delimiter": ",",
"DecimalMarker": ".",
"HasHeader": false,
"ColumnProperties": [
{
"ColumnName": "col0",
"ColumnPurpose": "Label",
"ColumnDataFormat": "String",
"IsCategorical": false,
"Type": "Column",
"Version": 4
},
{
"ColumnName": "col1",
"ColumnPurpose": "Text",
"ColumnDataFormat": "String",
"IsCategorical": false,
"Type": "Column",
"Version": 4
}
]
},
"Environment": {
"Type": "LocalCPU",
"Version": 1
},
"RunHistory": {
"Version": 3,
"Type": "Result",
"Trials": [
{
"Version": 1,
"Type": "Trial",
"TrainerName": "TextClassificationMulti",
"Score": 0.1,
"RuntimeInSeconds": 5.194,
"Parameter": {
"_SCHEMA_": "e0 * e1 * e2",
"e0": {
"OutputColumnName": "col0",
"InputColumnName": "col0",
"AddKeyValueAnnotationsAsText": false
},
"e1": {
"LabelColumnName": "col0",
"Sentence1ColumnName": "col1",
"ScoreColumnName": "Score",
"OutputColumnName": "PredictedLabel",
"BatchSize": 32,
"MaxEpochs": 10,
"Architecture": "Roberta"
},
"e2": {
"OutputColumnName": "PredictedLabel",
"InputColumnName": "PredictedLabel"
}
}
}
],
"Estimators": {
"e0": "MapValueToKey",
"e1": "TextClassificationMulti",
"e2": "MapKeyToValue"
},
"Schema": "e0 * e1 * e2",
"MetricName": "MacroAccuracy",
"ModelFilePath": "C:\\Users\\User\\source\\repos\\MlTest\\MlTest\\MLModel2.mlnet"
},
"Type": "TrainingConfig",
"Version": 4,
"TrainingOption": {
"Version": 1,
"Type": "TextClassficationTrainingOption",
"TrainingTime": 2147483647,
"LabelColumn": "col0",
"TextColumn": "col1",
"ValidationOption": {
"Version": 0,
"Type": "CrossValidationValidationOption",
"NumberOfFolds": 5
}
}
}
Yea please. Thanks!
same issue with same scenario with: ML.NET version 17.17.0.2332602 Microsoft Visual Studio 2022 Version 17.6.4
I am just experiencing it again on a different project. All I did was choosing a new TSV file and clicked on 'Train again'. But I also created a brand new project and got the same immediately on first train. Here is some updated info:
Exception:
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String trainingConfigurationFolder, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 155 at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()
VS Log:
Set log file path to C:\Users\XXX\AppData\Local\Temp\MLVSTools\logs\MLModel1-YC9ZBU.txt
start text classification
restore "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" --configfile "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\NuGet.config" -r win-x64 /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj"
publish "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" -r win-x64 -c Release --no-self-contained -o "C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5" --no-restore /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\\" /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj\\"
start installing runtime in C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5
Determining projects to restore...
Restored C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj (in 418 ms).
MSBuild version 17.8.0-preview-23418-03+0125fc9fb for .NET
C:\Program Files\dotnet\sdk\8.0.100-rc.1.23455.8\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj]
torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\Release\netstandard2.0\win-x64\torchsharp.gpu.dll
torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\
install runtime successfully
Use cross validation with fold: 5
| Trainer MacroAccuracy Duration |
|--------------------------------------------------------------------|
|0 TextClassificationMulti 0,0102 463,1580 |
|--------------------------------------------------------------------|
| Experiment Results |
|--------------------------------------------------------------------|
| Summary |
|--------------------------------------------------------------------|
|ML Task: text classification |
|Dataset: H:\xxxx.tsv |
|Label : col0 |
|Total experiment time : 463,1580 Secs |
|Total number of models explored: 1 |
|--------------------------------------------------------------------|
| Top 1 models explored |
|--------------------------------------------------------------------|
| Trainer MacroAccuracy Duration |
|--------------------------------------------------------------------|
|0 TextClassificationMulti 0,0102 463,1580 |
|--------------------------------------------------------------------|
Generate code behind files
Log file:
(Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update TrainViewModel_163.TrainingDidNotSucceed (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TryOutRow (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TopFiveRecommendationLabel (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutItemIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutUserIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG fail calling update handler GetSampleCodeSnippetAsync (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG Exception has been thrown by the target of an invocation. (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.ConsumeViewModel.<GetSampleCodeSnippetAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<>c__DisplayClass15_0.<<Initialize>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<TryExecuteNotifyDependencyNodeDelegateAsync>d__20.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
Looks like you are running with cross-validation
? The fix for cross-validation hasn't been fix yet so you might need to use another validation strategy
Unbelievable, you're right. I switched from cross-validation
(which seems to be the default option because I never changed that) to split-validation
and this fixed it. But cross-validation
seems to be the better one, any ETA when it will be fixed? Or can you at least say if the team is working on it?
//edit: With the first run using split-validation
the results were acceptable. Now I've trained again and I get completely different prediction results comparing to the first run. Not just slightly different values but completely(!) different results. Even though the data which is used for training hasn't changed.
@Symbai Actually, looks like the issue has already been fixed by me,,,, can you try upgrade model builder to v17.17.* and give it another trial?
The VS extension I have says its the latest version: 17.17.0.2332602 see screenshot in my previous post https://github.com/dotnet/machinelearning-modelbuilder/issues/2716#issuecomment-1737813544
//edit: Reinstalled the extension but still the same
OK, let me double check then
@LittleLittleCloud any update?
@Symbai No haven't yet, I was focusing on other stuff the last week. Will get the update back to this thread once I have any.
Also feel free to ping me for reminder
same here Use cross validation with fold: 5 | Trainer MicroAccuracy Duration |
---|---|
0 TextClassificationMulti 0,3045 65,9930 |
System Information (please complete the following information):
Model Builder Version (available in Manage Extensions dialog):
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/14368203/41c60304-54a4-41e1-8572-3acbebf3b757)
Visual Studio Version 17.7.0 Preview 2.0
Describe the bug
To Reproduce
Expected behavior I expect to proceed to "Evaluate" step. If I manually go to this tab and press the "Predict" button, I get the same exception as above.
Screenshots
Additional context Last log entry in VS is
Generate code behind files