Image classification based on transfer learning, unable to use bitmap which load from MemoryStream.

[fuine6]

According to past examples, I have a working code, and I can load image from MemoryStream.

public class ImageData
{
    [ImageType(81, 81)]
    public Bitmap Bitmap { get; set; }

    public string Label;
}

ITransformer TrainModel(MLContext mlContext, IEnumerable<ImageData> imageDatas)
{
    var pipeline= mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: LabelTokey, inputColumnName: "Label")
        .Append(mlContext.Transforms.ResizeImages(outputColumnName: "input", imageWidth: 81, imageHeight: 81, inputColumnName: nameof(ImageData.Bitmap)))
        .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "input", interleavePixelColors: true, offsetImage: 117))
        .Append(mlContext.MulticlassClassification.Trainers.LbfgsMaximumEntropy(labelColumnName: LabelTokey, featureColumnName: "input"))
        .Append(mlContext.Transforms.Conversion.MapKeyToValue(PredictedLabelValue, "PredictedLabel"))
        .AppendCacheCheckpoint(mlContext);

    var data = mlContext.Data.LoadFromEnumerable(imageDatas);

    ITransformer model = pipeline.Fit(data);

    return model;
}

Now I try to use this example with the following code.

ITransformer TrainModel(MLContext mlContext, IEnumerable<ImageData> imageDatas)
{
    var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: LabelTokey, inputColumnName: "Label")
        .Append(mlContext.Model.ImageClassification(nameof(ImageData.Bitmap), LabelTokey,
                    arch: ImageClassificationEstimator.Architecture.InceptionV3,
                    epoch: 100,
                    batchSize: 30,
                    metricsCallback: (metrics) => Console.WriteLine(metrics)));

    var data = mlContext.Data.LoadFromEnumerable(imageDatas);

    ITransformer model = pipeline.Fit(data);// <--- Here throw a exception.

    return model;
}

I got a exception.

Schema mismatch for input column 'Bitmap': expected String, got Image<81, 81>
Parameter name: inputSchema

Can I load the image from MemoryStream?

[CESARDELATORRE]

@fuine6 - In the current preview version of the 'mlContext.Model.ImageClassification' API, the only way to train/score a model is by providing image paths.

That's a know limitation in the current Preview of the new API. Here's the issue at the ML.NET repo: https://github.com/dotnet/machinelearning/issues/4153

Please, provide your feedback in that issue at the ML.NET so the team can validate the importance of the scenario with your additional feedback.

Being able to directly provide in-memory images is a priority for the next version. I agree this is "a must" to have asap for this new high level API, currently in Preview.

Thanks for your feedback. 👍

[ashbhandare]

Closing as this support for in-memory images has been added.