[Image Classification API] metricsCallback NullReferenceException when using custom function

[luisquintanilla]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): .NET Core 2.2
• ML.NET Version: 1.4.0-preview

Issue

Tried to provide a custom function for the metricsCallback parameter. The documentation in IntelliSense says this method is only called during the Training phase. However, it does not appear to be the case. If the Bottleneck phase has to happen (training for first time or not using cached bottleneck values), a System.NullReferenceException is raised.

System.NullReferenceException: 'Object reference not set to an instance of an object.'

trainMetrics was null.

I suspect this is because the same callback is used for both the Bottleneck / Training phases. It would be good to either:

a) Have separate callbacks depending on whether it's Bottleneck / Training phase. b) Allow the user to check which phase is currently taking place so they can display the results accordingly in their metricsCallback method.

Note that once a model is trained and the Bottleneck phase no longer needs to happen since the cached values are being used, no Exceptions are raised and the application works as expected.

Source code / logs

Pipeline:

var trainingPipeline =
    mapLabelTransform
   .Append(mlContext.Model.ImageClassification(
       "ImagePath",
       "LabelAsKey",
       arch: ImageClassificationEstimator.Architecture.ResnetV2101,
       epoch: 100,
       batchSize: 20,
       metricsCallback: DisplayMetrics,
       validationSet: transformedTestData
       //reuseTrainSetBottleneckCachedValues: true,
       /*reuseValidationSetBottleneckCachedValues: true*/));

Metrics Callback Method:

public static void DisplayMetrics(ImageClassificationMetrics metrics)
{
    TrainMetrics trainMetrics = metrics.Train;
    Console.WriteLine($"Epoch: {trainMetrics.Epoch} | Accuracy {trainMetrics.Accuracy} | Loss: {trainMetrics.CrossEntropy}");
}

[luisquintanilla]

Here is a workaround for now to use a custom function.

public static void DisplayMetrics(ImageClassificationMetrics metrics)
{
    if(metrics.Train == null)
    {
        Console.WriteLine(metrics);
    } else
    {
        TrainMetrics trainMetrics = metrics.Train;
        Console.WriteLine($"Epoch: {trainMetrics.Epoch} | Accuracy {trainMetrics.Accuracy} | Loss: {trainMetrics.CrossEntropy}");
    }
}

[harshithapv]

metricsCallback parameter is called during training phase and bottleneck phase. I have fixed the documentation. Thanks.