mlr-org / mlr3pipelines

Dataflow Programming for Machine Learning in R
https://mlr3pipelines.mlr-org.com/
GNU Lesser General Public License v3.0
140 stars 25 forks source link

PipeOpTuneThreshold does not respect maximization / minimization of measure #680

Closed sebffischer closed 2 months ago

sebffischer commented 2 years ago
library(mlr3verse)
#> Loading required package: mlr3
task = tsk("german_credit")
graph = po("learner_cv", learner = lrn("classif.rpart", predict_type = "prob")) %>>%
  po("tunethreshold", measure = msr("classif.acc"), optimizer = "random_search")

learner1 = as_learner(graph)
learner2 = lrn("classif.rpart")

design = benchmark_grid(
  task,
  list(learner1, learner2),
  rsmp("holdout")
)

bmr = benchmark(design)
#> INFO  [15:12:56.038] [mlr3] Running benchmark with 2 resampling iterations 
#> INFO  [15:12:56.225] [mlr3] Applying learner 'classif.rpart.tunethreshold' on task 'german_credit' (iter 1/1) 
#> INFO  [15:12:56.430] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 1/3) 
#> INFO  [15:12:56.645] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 2/3) 
#> INFO  [15:12:56.717] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 3/3) 
#> INFO  [15:12:58.398] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 1/1) 
#> INFO  [15:12:58.459] [mlr3] Finished benchmark
bmr$score(msr("classif.acc"))
#>                                   uhash nr              task       task_id
#> 1: 139dd95b-115b-4e5d-973a-51f43eaade6a  1 <TaskClassif[50]> german_credit
#> 2: f6830c7e-4df1-412b-89d0-9b76a8938c4e  2 <TaskClassif[50]> german_credit
#>                      learner                  learner_id
#> 1:        <GraphLearner[38]> classif.rpart.tunethreshold
#> 2: <LearnerClassifRpart[38]>               classif.rpart
#>                 resampling resampling_id iteration              prediction
#> 1: <ResamplingHoldout[20]>       holdout         1 <PredictionClassif[20]>
#> 2: <ResamplingHoldout[20]>       holdout         1 <PredictionClassif[20]>
#>    classif.acc
#> 1:   0.3093093
#> 2:   0.7477477

Created on 2022-08-02 by the reprex package (v2.0.1)

mb706 commented 2 months ago

holy cow