Is it possible to define custom optimization metric for AutoML?

[michaelgsharp]

@LittleLittleCloud can you take a look a this?

[LittleLittleCloud]

@80LevelElf Yes you can. There're two ways to do that

Option 1: Create your own IEvaluationMetricManager

Firstly, you need to implement IEvaluationMetricManager

class LocalObjectDetectionMetricManager : IEvaluateMetricManager
    {
        public string LabelColumn { get; set; }

        // This tell AutoMLExperiment how to optimize this metric
        public bool IsMaximize => true; // Assume both metrics are maximize.

        public string MetricName { get; set; }

        public double Evaluate(MLContext context, IDataView eval)
        {
            var labelColumn = eval.Schema.First(s => s.Name == "Labels" && s.Type.RawType == typeof(VBuffer<ReadOnlyMemory<char>>));
            var predictedLabelColumn = eval.Schema.Last(s => s.Name == "PredictedLabel" && s.Type.RawType == typeof(VBuffer<ReadOnlyMemory<char>>));
            var boxColumn = eval.Schema["Box"];
            var predictedBoxColumn = eval.Schema["PredictedBoundingBoxes"];
            var scoreColumn = eval.Schema["score"];

            var metrics = context.MulticlassClassification.EvaluateObjectDetection(eval, labelColumn, boxColumn, predictedLabelColumn, predictedBoxColumn, scoreColumn);

            if (this.MetricName == TrainingConfigurationConstants.MAP50)
            {
                return metrics.MAP50;
            }
            else
            {
                return metrics.MAP50_95;
            }
        }
    }

Then inject it when creating AutoMLExperiment, you might need to use reflection to get internal service collection.

var metricManager = new LocalObjectDetectionMetricManager();
experiment.ServiceCollection.AddSingleton<IEvaluateMetricManager>(metricManager);

Option 2: Define your own ITrialRunner

Firstly, implements your own trial runner. You can take a reference at SweepableTrialRunner. Except that you can remove the IEvaluationMetricManager from its constructor. And calculate metric however way you want in ITrialRunner.Run. Make sure you always return loss (minimize object) in trial result.

Then inject your runner when creating AutoMLExperiment using one of SetTrialRunner API

experiment.SetTrialRunner<YourTrialRunner>();

[80LevelElf]

@LittleLittleCloud thank you for such a detail answer!!