GPU Training Cancelled

[wes-baldwin]

System Information (please complete the following information):

• Model Builder Version: Latest version ML.NET Model Builder GPU Support (Preview)
• Visual Studio Version: 2019

Describe the bug

• On which step of the process did you run into an issue: Training
• Clear description of the problem: After about 30 minutes the training failed with error Training Cancelled. The stack trace from the console is below...
• [Source=ImageClassificationTrainer; ImageClassificationTrainer, Kind=Trace] Phase: Bottleneck Computation, Dataset used: Train, Image Index: 10439 [Source=AutoML, Kind=Error] Pipeline crashed: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImageSource_featurized:ImageSource imageFolder=} xf=ColumnCopying{ col=Features:ImageSource_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- . Exception: Tensorflow.TensorflowException: Blas GEMM launch failed : a.shape=(10, 1001), b.shape=(10, 5), m=1001, n=5, k=10 [[{{node train/gradients/final_retrain_ops/Wx_plus_b/MatMul_grad/MatMul_1}}]] at Tensorflow.Status.Check(Boolean throwException) at Microsoft.ML.TensorFlow.TensorFlowUtils.Runner.Run() at Microsoft.ML.Vision.ImageClassificationTrainer.TrainAndEvaluateClassificationLayerCore(Int32 epoch, Single learningRate, Int32 featureFileStartOffset, ImageClassificationMetrics metrics, Int64[] labelTensorShape, Int64[] featureTensorShape, Int32 batchSize, Stream trainSetLabelReader, Stream trainSetFeatureReader, Byte[] labelBufferBytes, Byte[] featuresBufferBytes, Int32 labelBufferSizeInBytes, Int32 featureBufferSizeInBytes, Int32 featureFileRecordSize, LearningRateScheduler learningRateScheduler, DnnTrainState trainState, Runner runner, IntPtr featureBufferPtr, IntPtr labelBufferPtr, Action2 metricsAggregator) at Microsoft.ML.Vision.ImageClassificationTrainer.TrainAndEvaluateClassificationLayer(String trainBottleneckFilePath, String validationSetBottleneckFilePath) at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor) at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.AutoML.RunnerUtil.TrainAndScorePipeline[TMetrics](MLContext context, SuggestedPipeline pipeline, IDataView trainData, IDataView validData, String groupId, String labelColumn, IMetricsAgent`1 metricsAgent, ITransformer preprocessorTransform, FileInfo modelFileInfo, DataViewSchema modelInputSchema, IChannel logger) [Source=ImageClassificationTrainer; ImageClassificationTrainer, Kind=Trace] Channel started [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Trace] Channel started [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Trace] Channel disposed [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Trace] Channel finished. Elapsed 00:00:00.0001597. Training cancelled

To Reproduce Steps to reproduce the behavior:

1. Download the sample dataset from kaggle @ https://www.kaggle.com/puneet6060/intel-image-classification
2. Step thru the model builder UI (selecting the GPU environment) and select the train folder from the extracted archive.zip from kaggle
3. Click Train
4. After about 20 minutes (for my machine) it will begin displaying image indexes in the console.
5. 5-10 minutes later the stated error occurs.

Expected behavior I expected the training to complete

Screenshots

Additional context I installed all software as stated on https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/install-gpu-model-builder. My machine info:

[JakeRadMSFT]

Hey @wes-baldwin!

Sorry for the delayed response!

Would you be up for trying our private-preview build to see if it resolves your issue? If so, you can sign up here: https://aka.ms/mb-private-preview

If not - I think the "old" or released version might just need updated versions of Cuda and CuDNN.

We have new instructions here: https://github.com/dotnet/machinelearning/blob/main/docs/api-reference/tensorflow-usage.md

You'll have to uninstall Cuda 10.0 and replace it with Cuda 10.1

Sorry again for delayed response. Let me know if this resolves the issue!

[JakeRadMSFT]

@wes-baldwin I just finished running this and I don't think the above will resolve the issue for this dataset.

We're looking into why it's canceling early

[wes-baldwin]

Thanks for the response. I’ve since moved on to other projects. I don’t require a resolution. I really posted this more for the benefit of others.