Can't even start the training

[cannja]

I get this error immediately, not clear on how to resolve this...

It seemed to work fine yesterday, I created a brand new project with the same image data, but this time.. see below

2022-06-01 09:31:18.7575 INFO Set log file path to C:\Users\User\AppData\Local\Temp\MLVSTools\logs\ImageTest-R195Q6.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:49.8111 INFO start nni training (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:49.8229 INFO Experiment output folder: C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-L4K2QY (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:49.8229 DEBUG row count is unknown, count it explicitly (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:50.1969 DEBUG count elapse 374ms (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:50.1969 INFO | Trainer MicroAccuracy MacroAccuracy Duration #Iteration | (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:52.5431 DEBUG update train dataset C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-L4K2QY\train.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:52.5431 DEBUG update test dataset C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-L4K2QY\test.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:55.3054 INFO Trial 0 encounter error with message: Splitter/consolidator worker encountered exception while consuming source data (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:55.3260 DEBUG at Microsoft.ML.Data.DataViewUtils.Splitter.Batch.SetAll(OutPipe[] pipes) at Microsoft.ML.Data.DataViewUtils.Splitter.Cursor.MoveNextCore() at Microsoft.ML.Data.RootCursorBase.MoveNext() at Microsoft.ML.Vision.ImageClassificationTrainer.CacheFeaturizedImagesToDisk(IDataView input, String labelColumnName, String imageColumnName, ImageProcessor imageProcessor, String inputTensorName, String outputTensorName, String cacheFilePath, Dataset dataset, Action1 metricsCallback, Nullable1 validationFraction) at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor) at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.ModelBuilder.AutoMLService.MulticlassClassificationRunner.RunTrainTestSplit(TrialParameter parameter, TrainingConfiguration config, IDataView trainDatasetCache, IDataView testDatasetCache) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Runner/MulticlassClassificationRunner.cs:line 70 at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.Run() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 172 at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.d_19.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 181 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.d_16.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 49 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:55.3260 DEBUG update best config to trial id 0, current best metric is 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:55.3260 DEBUG create new thread 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:32:55.3260 INFO ===============================================Experiment Results=================================================

| Summary |

|ML Task: ImageClassification | |Dataset: | |Label : Label | |Total experiment time : 0.00 Secs | |Total number of models explored: 0 |

(Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:55.3260 INFO | Top 0 models explored |

| Trainer MicroAccuracy MacroAccuracy Duration #Iteration |

(Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:32:55.3571 DEBUG filePath cannot be null or empty Parameter name: filePath at Microsoft.ML.Runtime.Contracts.CheckNonEmpty(IExceptionContext ctx, String s, String paramName) at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.d15.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 231 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.d_21.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:37:42.6490 INFO start nni training (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:37:42.6490 INFO Experiment output folder: C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-MV18M5 (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:37:42.6490 DEBUG row count is unknown, count it explicitly (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:37:43.1419 DEBUG count elapse 491ms (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:37:43.1419 INFO | Trainer MicroAccuracy MacroAccuracy Duration #Iteration | (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:37:45.5122 DEBUG update train dataset C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-MV18M5\train.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:37:45.5122 DEBUG update test dataset C:\Users\User\AppData\Local\Temp\AutoML-NNI\Experiment-MV18M5\test.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:38:23.8104 INFO Trial 0 encounter error with message: Splitter/consolidator worker encountered exception while consuming source data (Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:38:23.8104 DEBUG at Microsoft.ML.Data.DataViewUtils.Splitter.Batch.SetAll(OutPipe[] pipes) at Microsoft.ML.Data.DataViewUtils.Splitter.Cursor.MoveNextCore() at Microsoft.ML.Data.RootCursorBase.MoveNext() at Microsoft.ML.Vision.ImageClassificationTrainer.CacheFeaturizedImagesToDisk(IDataView input, String labelColumnName, String imageColumnName, ImageProcessor imageProcessor, String inputTensorName, String outputTensorName, String cacheFilePath, Dataset dataset, Action1 metricsCallback, Nullable1 validationFraction) at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor) at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.ModelBuilder.AutoMLService.MulticlassClassificationRunner.RunTrainTestSplit(TrialParameter parameter, TrainingConfiguration config, IDataView trainDatasetCache, IDataView testDatasetCache) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Runner/MulticlassClassificationRunner.cs:line 70 at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.Run() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 172 at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.d19.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 181 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.d16.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 49 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:38:23.8104 DEBUG update best config to trial id 0, current best metric is 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:38:23.8104 DEBUG filePath cannot be null or empty Parameter name: filePath at Microsoft.ML.Runtime.Contracts.CheckNonEmpty(IExceptionContext ctx, String s, String paramName) at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.d_15.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 231 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.d21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:38:23.8104 DEBUG create new thread 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug) 2022-06-01 09:38:23.8104 INFO ===============================================Experiment Results=================================================

| Summary |

|ML Task: ImageClassification | |Dataset: | |Label : Label | |Total experiment time : 0.00 Secs | |Total number of models explored: 0 |

(Microsoft.ML.ModelBuilder.Utils.Logger.Info) 2022-06-01 09:38:23.8104 INFO | Top 0 models explored |

| Trainer MicroAccuracy MacroAccuracy Duration #Iteration |

(Microsoft.ML.ModelBuilder.Utils.Logger.Info)

[beccamc]

@cannja Sorry you ran into this...

1. Which VS version are you on?
2. Are you on the latest Model Builder - 16.13.6? Marketplace
3. Are you able to share the image dataset with us?

[beccamc]

Any update on the above information?

[beccamc]

We don't have enough information to investigate this issue. Please re-open with more info if this is still a problem.