Microsoft.ML.Transforms.TensorFlow.TFException

[developervariety]

This is my class.

public interface IModelScorer
    {
        PredictionEngine<ImageNetData, ImageNetPrediction> CreatePredictionFunction();
        ImageNetLabelPrediction PredictLabelWithProbability(string imageFilePath);
    }

    public class ModelScorer : IModelScorer
    {
        private readonly string _dataLocation;
        private readonly string _imagesFolder;
        private readonly string _modelLocation;
        private readonly string _labelsLocation;
        private readonly MLContext _mlContext;
        private static string _imageReal = nameof(_imageReal);

        private readonly PredictionEngine<ImageNetData, ImageNetPrediction> _predictionFunction;
        public PredictionEngine<ImageNetData, ImageNetPrediction> PredictionFunction => _predictionFunction;

        public ModelScorer(string dataLocation, string imagesFolder, string modelLocation, string labelsLocation)
        {
            _dataLocation = dataLocation;
            _imagesFolder = imagesFolder;
            _modelLocation = modelLocation;
            _labelsLocation = labelsLocation;
            _mlContext = new MLContext();

            _predictionFunction = CreatePredictionFunction();
        }

        public struct ImageNetSettings
        {
            public const int ImageHeight = 224;
            public const int ImageWidth = 224;
            public const float Mean = 117;
            public const bool ChannelsLast = true;
        }

        public struct InceptionSettings
        {
            public const string InputTensorName = "Placeholder";
            public const string OutputTensorName = "loss";
        }

        public void Score(string imagePath)
        {

            ImageNetLabelPrediction prediction = PredictLabelWithProbability(imagePath);

            Console.WriteLine(prediction.PredictedLabel);
        }

        public PredictionEngine<ImageNetData, ImageNetPrediction> CreatePredictionFunction()
        {            
            Console.WriteLine($"Model location: {_modelLocation}");
            Console.WriteLine($"Images folder: {_imagesFolder}");
            Console.WriteLine($"Training file: {_dataLocation}");
            Console.WriteLine($"Default parameters: image size=({ImageNetSettings.ImageWidth},{ImageNetSettings.ImageHeight}), image mean: {ImageNetSettings.Mean}");

            IDataView dataView = CreateDataView();

            EstimatorChain<TensorFlowTransformer> pipeline = _mlContext.Transforms.LoadImages("Placeholder", _imagesFolder, nameof(ImageNetData.ImagePath))
                .Append(_mlContext.Transforms.ResizeImages("Placeholder", ImageNetSettings.ImageWidth, ImageNetSettings.ImageHeight, "Placeholder"))
                .Append(_mlContext.Transforms.ExtractPixels("Placeholder", interleavePixelColors: ImageNetSettings.ChannelsLast, offsetImage: ImageNetSettings.Mean))
                .Append(_mlContext.Model.LoadTensorFlowModel(_modelLocation).
                    ScoreTensorFlowModel(new[] { "loss" },
                        new[] { "Placeholder" }, true));

            ITransformer model = pipeline.Fit(dataView);

            return _mlContext.Model.CreatePredictionEngine<ImageNetData, ImageNetPrediction>(model);
        }

        ImageNetLabelPrediction IModelScorer.PredictLabelWithProbability(string imageFilePath)
        {
            return PredictLabelWithProbability(imageFilePath);
        }

        protected ImageNetLabelPrediction PredictLabelWithProbability(string imageFile)
        {
            try
            {
                string[] labels = ReadLabels(_labelsLocation);

                ImageNetData imageInputData = new ImageNetData { ImagePath = imageFile };
                float[] imageLabelPredictions;

                lock (_predictionFunction)
                {
                    imageLabelPredictions = _predictionFunction.Predict(imageInputData).PredictedLabels;
                }

                ImageNetLabelPrediction imageBestLabelPrediction = new ImageNetLabelPrediction
                {
                    ImagePath = imageInputData.ImagePath
                };

                (imageBestLabelPrediction.PredictedLabel, imageBestLabelPrediction.Probability) = GetBestLabel(labels, imageLabelPredictions);

                return imageBestLabelPrediction;

            }
            catch (Exception e)
            {
                throw e;
            } 
        }

        private IDataView CreateDataView()
        {
            List<ImageNetData> list = new List<ImageNetData>();
            IEnumerable<ImageNetData> enumerableData = list;

            IDataView dv = _mlContext.Data.LoadFromEnumerable(enumerableData);
            return dv;
        }
    }

Part of the exception

Unhandled Exception: Microsoft.ML.Transforms.TensorFlow.TFException: The first dimension of paddings must be the rank of inputs[4,2] [1,1,224,224,3]
         [[{{node conv0/pad_size}} = Pad[T=DT_FLOAT, Tpaddings=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](data_bn/data_bn/add_1, block2_0_conv3/pad_size/paddings)]]
   at Rcd.Core.ML.ModelScorer.PredictLabelWithProbability(String imageFile) in REDACTED:line 114
   at Core.ML.ModelScorer.Score(String imagePath) in REDACTED:line 57

[developervariety]

I should also include I am using a custom model exported from Custom Vision (Azure).

Package versions: all latest

[prathyusha12345]

Can you share the model location to try once from my end

[developervariety]

Yes, @prathyusha12345.

Here's a Google Drive link. https://drive.google.com/open?id=11Cvdku7IybhOA2vAqfcE86UcQeqhoEme

[prathyusha12345]

I have tried the custom model you have provided. While prediction, The model is not able to predict any label. The imageLabelPredictions value after executing below statement is null. Not sure why model is not able to predict.

imageLabelPredictions = _predictionFunction.Predict(imageInputData).PredictedLabels;

I have created an issue in MachinLearning Repo here https://github.com/dotnet/machinelearning/issues/3751

[developervariety]

iirc this works on ML v0.7

My project requires the latest version otherwise I would use v0.7, but thank you for your help.

[developervariety]

Switched to ML v0.7 for testing, and it's "working" but it has one issue. It continues giving me the same model.

Used the example from @CESARDELATORRE's web API, same model.

[CESARDELATORRE]

We’ll look into this. Btw, the model trained with Custom Vision from the WebAPI sample was able to just classify between two kinds of chairs. It was lacking a “null” or not applicable object, though, which should be implemented when training with Custom Vision.

[developervariety]

I used my own model with your example since it's the only example that helped and explained the most.

ML.NET is ONLY reporting the model on the second line of labels.txt with my custom model. I tested your custom model and it worked just fine..

I have about ~1k images for each tag in that model (3 tags). I'll make a new model and check results.

[developervariety]

Update with the new model

Same result, returns the same model when it shouldn't be. If anyone would like me to provide some images to test with that model just let me know.

The model works fine on Azure Prediction API, but when I export it to TensorFlow and use it on ML v0.7 it doesn't do the expected outcome.

GIF of it in action: https://i.imgur.com/Vm5zpQz.gifv

EDIT: I tested Cesar's Web API to check if it was my code and got the same result.. I believe Custom Vision updated something causing incompatibility with ML.NET

[zeahmed]

@developervariety, please set addBatchDimensionInput: false because this model has input with fully specified shape [?, 224, 224, 3] which is against inception model which does not have fully specified shape for the input.

[CESARDELATORRE]

@developervariety - Right, I tested the sample using the TensorFlow model generated with Cognitive Service Custom Vision and it works good by setting to addBatchDimensionInput: false

Closing this issue. Thanks for raising it up! 👍