dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.92k stars 1.86k forks source link

PredictionPool will not let me have an MLImage member of an input or output type #6936

Open tearlant opened 6 months ago

tearlant commented 6 months ago

System Information (please complete the following information):

Describe the bug There are a number of examples that use a PredictionEngine with an input type or output type that has a member of MLImage type, for example https://github.com/dotnet/machinelearning/issues/6886

However, my code is breaking when I call

var prePredictionEngine = mlContext.Model.CreatePredictionEngine<InputType, OutputType>(model);
    public class InputType
    {
        public string Label;

        [ColumnName("Image")]
        [ImageType(224, 224)]
        public MLImage Image { get; set; }
    }

This is the error that comes up image

Here's the console output

EXCEPTION
#########
System.NotImplementedException: Type 'Microsoft.ML.Data.MLImage' is not yet supported.
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.GenerateSetter(DataViewRow input, Int32 index, Column column, Delegate poke, Delegate peek)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase..ctor(TypedCursorable`1 parent, DataViewRow input, String channelMessage)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRow..ctor(TypedCursorable`1 parent, DataViewRow input)
   at Microsoft.ML.Data.TypedCursorable`1.GetRow(DataViewRow input)
   at Microsoft.ML.PredictionEngineBase`2.PredictionEngineCore(IHostEnvironment env, InputRow`1 inputRow, IRowToRowMapper mapper, Boolean ignoreMissingColumns, SchemaDefinition outputSchemaDefinition, Action& disposer, IRowReadableAs`1& outputRow)
   at Microsoft.ML.PredictionEngineBase`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.PredictionEngine`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at ImageClassification.ModelScorer.TFModelScorer.PredictDataUsingModelFromStructuredDirectory(IDataView testDataView) in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\ModelScorer\TFModelScorer.cs:line 181
   at ImageClassification.ModelScorer.TFModelScorer.ScoreUsingStructuredDirectory() in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\ModelScorer\TFModelScorer.cs:line 67
   at ImageClassification.Program.Main(String[] args) in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\Program.cs:line 32

I can make things work straight up by wrapping the data in a IDataView and calling .Transform(), but that defeats the point of having a PredictionEngine

tearlant commented 6 months ago

Update: I was able to create a workaround, by breaking things into two back to back PredictionEngines, the first of which is a pipeline with some data manipulation, then the first engine is just a ResizeImages and an ExtractImages transform, whose type is a VectorType instead of an ImageType.

But this feels like unnecessary gymnastics. I'm not sure what I'm doing wrong.