Open exalate-issue-sync[bot] opened 1 year ago
Sebastien Poirier commented: [~accountid:557058:afd6e9a4-1891-4845-98ea-b5d34a2bc42c] This is a request from Sparkling Water, and I believe it would benefit to API consistency and maintainability.
I was about to restrict current {{sort_metric}} to metrics defined in {{ScoreKeeper.StoppingMetric}} until I realized that technically, for binomial classification problems, leaderboard could also support metrics like {{accuracy}}, {{f1}}, {{f2}}, {{precision}}, {{recall}}, and so on… defined in AUC2. This is not documented though, and the sorting direction is currently wrong for those AUC2 metrics (but this could be easily fixed). Is it something we want to support and mention in documentation?
If so, I would have to create another abstraction on top of the 2 existing enums{{ScoreKeeper.StoppingMetric}} and {{AUC2.ThresholdCriterion}} to be able to expose them as one single {{enum}} or {{interface}}.
Otherwise, I will simply expose {{sort_metric}} as a {{StoppingMetric}}.
Any thoughts/concerns?
JIRA Issue Migration Info
Jira Issue: PUBDEV-6701 Assignee: Sebastien Poirier Reporter: Sebastien Poirier State: Open Fix Version: N/A Attachments: N/A Development PRs: Available
Linked PRs from JIRA
In AutoML,
sorting_metric
andstopping_metric
are currently exposed as a string with values "unfortunately" slightly different from the metrics defined inScoreKeeper.StoppingMetric
. On its side and with good reasons, Sparkling Water would like to use anenum
to consume the API manipulating those metrics.We need to find a way to expose an enum but keeping backwards compatibility: it will probably necessary to have a new enum only at API level, but we should be able to use
ScoreKeeper.StoppingMetric
enum for all logic inLeaderboard
especially.