Training / Predicting from in-memory Bitmap

[ibebbs]

Hi,

Just started playing with Microsoft.ML and am pretty impressed. I followed this tutorial to build an image classifier model that works reasonably well (limited training data). I now want to put this model to use but have hit an issue:

The images I want to classify will be in-memory (as a Bitmap) but the trained model seems to need the images on disk. Obviously I could save the image to a temporary file but this seems wasteful when the model will need to read it back in again. From what I can see in the source code, the "LoadRawImageBytes" transform from the [Model Builder generated] pipeline shown below doesn't have any kind of overload for loading in-memory data:

var pipeline = context.Transforms.Conversion.MapValueToKey("Label", "Label")
    .Append(context.Transforms.LoadRawImageBytes("ImageSource_featurized", null, "Image"))
    .Append(context.Transforms.CopyColumns("Features", "ImageSource_featurized"));

After a lot of searching I found this issue in which @huy-lv asks how to do pretty much exactly what I want to do. @Lynx1820 replied pointing to this sample which I have endeavoured to follow.

I now have the following pipeline:

var pipeline = context.Transforms.Conversion.MapValueToKey("Label", "Label")
    .Append(context.Transforms.ResizeImages(outputColumnName: "ScaledImage", imageWidth: 227, imageHeight: 227, inputColumnName: "Image"))
    .Append(context.Transforms.ExtractPixels(outputColumnName: "ImageSource_featurized", inputColumnName: "ScaledImage", outputAsFloatArray: false))
    .Append(context.Transforms.CopyColumns("Features", "ImageSource_featurized"));

var trainer = context.MulticlassClassification.Trainers.ImageClassification(new ImageClassificationTrainer.Options() { LabelColumnName = "Label", FeatureColumnName = "Features" })
    .Append(context.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));

var trainingPipeline = pipeline.Append(trainer);

But when I try to train the model (with trainingPipeline.Fit(trainingData)) I receive the error:

Schema mismatch for feature column 'Features': expected VarVector<Byte>, got Vector<Byte> '

Could you provide an example of how to use Transforms.ExtractPixels with MulticlassClassification.Trainers.ImageClassification or suggestions on how to train/predict from an in-memory Bitmap source?

Thanks!

[luisquintanilla]

@ibebbs take a look at this sample. I think it might help you do what you want.

https://github.com/dotnet/machinelearning/blob/master/docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ImageClassificationDefault.cs

Notice to train it uses ImageData but to score it uses InMemoryImageData. Difference being, ImageData uses a path and InMemoryImageData uses byte[].

[antoniovs1029]

Hi @ibebbs , so I read on Gitter that the sample that @luisquintanilla provided was enough for you to solve your problem. I will close this issue now. Feel free to reopen this if you still have problems with this. 😄