search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.02k stars 1.88k forks source link

[ImageClassification Transfer Learning] Need to improve training perf/time #4080

Closed CESARDELATORRE closed 5 years ago

CESARDELATORRE commented 5 years ago

I know we're on the work for this issue but I just wanted to create an issue to track it down. 👍

Our current performance/time needed for training when using the new ImageClassification Transfer Learning is kind of huge compared to other frameworks, probably due to wrong approaches taken in the training process that can be solved soon.

On top of that we'll support GPU pretty soon, so that will improve training performace/time even further.

Here are some comparisons with TensorFlow.NET vs. the new ImageClassification Transfer Learning (Early preview):

A. ML.NET ImageClassification Transfer Learning training with 80 photo Files, 2 Folders/Categories --> It took 1,067 seconds --> 18 minutes

b. TensorFlow.NET training with the same 80 photo Files, 2 Folders/Categories --> It took only 54 seconds (around 1 minute)

Meaning it was around 1,800% worse in training time (18 times worse).

Another test was: TF.NET training with 3,671 photo Files, 5 Folders/Categories (Flowers image set) --> It took 1,010 seconds (16 minutes) - I didn't test that with ML.NET ImageClassification Transfer Learning training since it would have taken hours.

Discussing about it with Zeeshan, the issues and solutions for it have been identified and we expect to fix it pretty soon. 👍

justinormont commented 5 years ago

May also want to try the DNNImageFeaturizer(), which is based on ONNX though it will not refit the model.

CESARDELATORRE commented 5 years ago

Closing this issue. There was an error in the test code plus the perf itself of the training has also been improved.