search

tidymodels / yardstick

Tidy methods for measuring model performance
https://yardstick.tidymodels.org/
Other
379 stars 54 forks source link

Feature request: custom `direction` in `new_metric()` #311

Open mikemahoney218 opened 2 years ago

mikemahoney218 commented 2 years ago

Feature

Apologies if this already exists and I've missed it.

It would be useful to allow custom direction options when creating new metrics. My current use-case for this behavior is implementing Geary's C in waywiser, which has an optimal value of 1; while this case is easy to engineer around (a metric C exists such that `C = C - 1, lettingdirection = "zero"`) I think there may be other metrics which would benefit from being able to provide their own direction arguments.

DavisVaughan commented 2 years ago

The biggest issue with a completely custom direction is that tune somehow has to know how to use it. In particular, tune_bayes() does different things depending on whether you are minimizing/maximizing your metric

https://github.com/tidymodels/tune/blob/e4b7485f6196c005d457f713c2bc5da5ffbfb63c/R/tune_bayes.R#L216

So does show_best() I believe

It wouldn't know how to handle a completely custom direction without some kind of constraints around it.