Tensorflow prediction failure with MSTest

[nubcake94]

System information

• OS version/distro Windows 10:
• .NET Version (eg., dotnet --info) 4.8:

Issue

• What did you do?

Loaded Tensorflow model then called MakePrediction.

• What happened?

A Tensorflow.TensorflowException was raised

• What did you expect?

No exception, expected functionality

Source code

public float PredictOne(Bitmap image)
        {
            if(image == null)
            {
                throw new ArgumentNullException();
            }

            float[] floatArrayImage = BitmapToFloatArrayWithPreprocessing(image);
            ImageInputData[] input = new ImageInputData[] { new ImageInputData() { serving_default_input_1 = floatArrayImage } };
            var outScores = MakePrediction(input);

            float result = 0;

            foreach(var prediction in outScores)
            {
                result = prediction.StatefulPartitionedCall[0];
            }

            return result;
        }

Log

Test method HardhatDet.Tests.HardhatDetTest.PredictOneTestNoHardhat threw exception: 
System.InvalidOperationException: Splitter/consolidator worker encountered exception while consuming source data ---> Tensorflow.TensorflowException: Op type not registered 'FusedBatchNormV3' in binary running on GITLAB-RUNNER. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
     [[{{node StatefulPartitionedCall}}]]
     [[{{node StatefulPartitionedCall}}]]
  Stack Trace:
      at Microsoft.ML.TensorFlow.TensorFlowUtils.Runner.Run()
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.UpdateCacheIfNeeded(Int64 position, ITensorValueGetter[] srcTensorGetters, String[] activeOutputColNames, OutputCache outputCache)
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.<>c__DisplayClass9_0`1.<MakeGetter>b__4(VBuffer`1& dst)
   at Microsoft.ML.Data.DataViewUtils.Splitter.InPipe.Impl`1.Fill()
   at Microsoft.ML.Data.DataViewUtils.Splitter.<>c__DisplayClass7_1.<ConsolidateCore>b__2()
--- End of inner exception stack trace ---
    at Microsoft.ML.Data.DataViewUtils.Splitter.Batch.SetAll(OutPipe[] pipes)
   at Microsoft.ML.Data.DataViewUtils.Splitter.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.TypedCursorable`1.RowCursorImplementation.MoveNext()
   at Microsoft.ML.PipeEngine`1.<RunPipe>d__3.MoveNext()
   at HardhatDet.HardhatDetector.PredictOne(Bitmap image) in D:\Gitlabrunner\builds\gb6xNTch\0\ftf\door\HardhatDet\HardhatDetector.cs:line 55
   at HardhatDet.Tests.HardhatDetTest.PredictOneTestNoHardhat() in D:\Gitlabrunner\builds\gb6xNTch\0\ftf\door\GuardTests\HardhatDetTest.cs:line 123

[michaelgsharp]

What version of ML.NET are you running?

[nubcake94]

ML.NET version

latest - 1.5.4

Additional information

MakePrediction function

private IEnumerable<OutputScore> MakePrediction(ImageInputData[] input)
{
            var idv = _mlContext.Data.LoadFromEnumerable(input);

            var estimator = _pipeline.Fit(idv);
            var transformedValues = estimator.Transform(idv);

            var outScores = _mlContext.Data.CreateEnumerable<OutputScores>(
                transformedValues, reuseRowObject: false);

            return outScores;
}

Occurances

• It seems to always appear in CI/CD pipeline using dotnet test
• It seems to sometimes appear locally, when I run my tests from Visual Studio or cmd dotnet test, but after a while the exceptions are gone
• Making predictions in the main program seems to work without any error