Measures can now return more than one scalar value. This is needed to make CIs work, which return a named vector with three elements (estimate, lower, upper)
Some measures like RMSE, use $obs_loss + trafo. In order to be able to create CIs for such measures, this PR adds the $trafo field, which in the case of RMSE is sqrt()
Some resampling methods in mlr3inference do not use all resampling iterations for point estimates (e.g. some are only used for variance estimation). We want to exclude these iterations when doing rr$aggrgate(msr("classif.acc")). Therefore, the Resampling class can now implement $primary_iters, which should be those iterations that should be used to obtain a point estimate. However, some measures with average = "custom" cannot handle this. Therefore I added the new "primary_iters" property to Measures, which should by default NOT be present in measures that have average = "custom".
Related work:
mlr3inferencepackage: https://github.com/mlr-org/mlr3inferenceSummary of changes:
Measures can now return more than one scalar value. This is needed to make CIs work, which return a named vector with three elements (estimate, lower, upper)$obs_loss+ trafo. In order to be able to create CIs for such measures, this PR adds the$trafofield, which in the case of RMSE issqrt()rr$aggrgate(msr("classif.acc")). Therefore, theResamplingclass can now implement$primary_iters, which should be those iterations that should be used to obtain a point estimate. However, some measures withaverage = "custom"cannot handle this. Therefore I added the new"primary_iters"property to Measures, which should by default NOT be present in measures that haveaverage = "custom".