Re-training an ImageClassificationTrainer

[gartangh]

System information

• OS version: Windows 10 Pro 18363
• .NET version: Core 2.1
• Platform: x64
• ML.NET version: 0.15-preview

Issue

• What did you do? I started working from this example, to perform multiclass classification using transfer learning. Training, saving, loading, and consuming the model works as intended. Now I am trying to train further on that trained model (re-training), using these 2 steps:

  • Extract pre-trained model parameters
  • Re-train model

  Similar to those steps, I came up with this piece of code:

 // Extract trained model parameters
ImageClassificationModelParameters originalParameters = ((ISingleFeaturePredictionTransformer<object>)trainedModel).Model as ImageClassificationModelParameters;
// Retrain model
MulticlassPredictionTransformer<ImageClassificationModelParameters> trainedModelFurther = mlContext.MulticlassClassification.Trainers.ImageClassification().Fit(trainImages, originalParameters);

• What happened? At the end of the last line, the method Fit(IDataView, ImageClassificationModelParameters) does not exist (docs).

• What did you expect?

  • First of all, I expected a similar method as Fit(IDataView, LinearModelParameters) in OnlineGradientDescentTrainer (docs), Fit(IDataView, MaximumEntropyModelParameters) in LbfgsMaximumEntropyMulticlassTrainer (docs), or other trainers to exist. I believe the ImageClassificationTrainer makes use of an LbfgsMaximumEntropyMulticlassTrainer behind the scenes, to classify the extracted features from the images.

  • Secondly (unrelated), the method Fit(IDataView, IDataView) in ImageClassificationTrainer seems unnecessary, as you can already specify the validation set (or a fraction of the training set to use for validation) in ImageClassificationTrainer.Options.

Question

Is it possible to re-train this model on new data, maybe using some extra steps to convert the ImageClassificationTrainer to an LbfgsMaximumEntropyMulticlassTrainer? If not, when is this functionality expected?

[antoniovs1029]

Hi, @gartangh .

As it's stated on the article you've linked ML.NET only supports retraining for some of the trainers. Unfortunately, ImageClassificationTrainer is not one of them, and currently it's not in our short term plans to add support for that.

Also, I believe that it was the case that ImageClassificationTrainer did use LbfgsMaximumEntropyMulticlassTrainer in the past, but that has changed. Previously, the ImageClassificationTrainer would use a DNN to featurize the input images, and then would use an Lbfgs to do classification. That has changed and now the ImageClassificationTrainer actually trains a DNN using tensorflow which does the featurization and classification. So it is also not possible to use the model parameters of an ImageClassificationTrainer to retrain a LbfgsMaximumEntropyMulticlassTrainer.

As for your second point, I believe the Fit(IDataView, IDataView) overload was created before the ImageClassificationTrainer.Options class was created. So even if it's now simpler to use the Optionsobject, we can't remove the Fit() overload, since that would be an API breaking change, and some users might already be using that method on their projects.

Back to your main ask, I will tag this issue as 'enhancement' to document the feature request about supporting retraining on ImageClassificationTrainer.

[gartangh]

Hi @antoniovs1029, thank you for your quick response.

Too bad, I cannot re-train an ImageClassificationTrainer for now. Training from scratch each time does not seem to be the best solution for my use-case. I feel like all trainers that inherit from TrainerEstimatorBase<TTransformer,TModel> need a generic way to train further on new data. E.g. these lines of code should work for all Trainers:

trainedModel.Options.LearningRate = 0.001f;
trainedModel.Options.ValidationSet = newValidationData;
var retrainedModel = trainedModel.Fit(newTrainData);

Where the trainedModel accepts new options like the learning rate and new validation set and just start training further from its learned weights on the new train set. I know this might be infeasible, but maybe it could function as a target to work towards.

Just out of interest, do you happen to know what kind of Trainer/optimizer the ImageClassificationTrainer uses now? (Maybe Adam?)

What concern my second point, I don't want to introduce any API breaking changes. Maybe it can be deprecated in the future? Or the documentation could advise using ImageClassificationTrainer.Options instead?

Back to my main ask, thank you for documenting my feature request by marking this issue as 'enhancement'!

[codemzs]

@gartangh @antoniovs1029 ImageClassificationTrainer never used LbfgsMaximumEntropyMulticlassTrainer. ImageClassificationTrainer was created to train a DNN model. In the past we never trained a DNN model and instead would use a pre-trained DNN model like resnet to featurize an image and then train a LbfgsMaximumEntropyMulticlassTrainer.

[antoniovs1029]

@gartangh Thanks for your suggestions regarding the API and general usability of ML.NET.

Just out of interest, do you happen to know what kind of Trainer/optimizer the ImageClassificationTrainer uses now? (Maybe Adam?)

The ImageClassificationTrainer works by taking a pretrained Tensorflow model, and retraining it with your input data (transfer learning). ML.NET provides 4 different architectures to do so: InceptionV3, Mobilenet V2, ResnetV2101 and ResnetV250. About the optimizer, I am not sure, but I believe it's SGD. I don't know if you're familiar with the samples repo, but you can read a little bit more about this there: link to sample.

ImageClassificationTrainer never used LbfgsMaximumEntropyMulticlassTrainer

Sorry, @codemzs , I misunderstood you when I asked you about this offline. My bad.

But regarding this, there's also a sample actually showing how to use a tensorflow moderl to featurize images, and then use LbfgsMaximumEntropyMulticlassTrainer to classify them (link to sample). So although it's not the same as using an ImageClassificationTrainer, it might be useful for you, @gartangh , since you would be able to retrain the LbfgsMaximumEntropyMulticlassTrainer.

[gartangh]

@antoniovs1029 , I am familiar with the 2 samples from the samples repository you sent. A couple of months ago, I used a featurizer and LbfgsMaximumEntropyMulticlassTrainer, but wanted to upgrade to the ImageClassificationTrainer, as it was easier to use.

Also thanks to you @codemzs , I now see the difference between the model composition and transfer learning methods. I was wrongly thinking the LbfgsMaximumEntropyMulticlassTrainer represented the last fully connected layers of a DNN.

One more thing: I am assuming that the featurizers from the model composition method cannot be used on GPUs, even though they are coming from pre-trained TensorFlow models, is that right?