mlr-org / mlr3hyperband

Successive Halving and Hyperband in the mlr3 ecosystem
https://mlr3hyperband.mlr-org.com
GNU Lesser General Public License v3.0
18 stars 5 forks source link

active_config indexing completely broken #41

Closed berndbischl closed 4 years ago

berndbischl commented 4 years ago

If i add the assert in this code, we see that produced indices are complety wrong

        # update active configurations
        assert_integer(best_indices, lower = 1, upper = nrow(active_configs))
        active_configs = active_configs[best_indices]

the problem is here

        # get performance of each active configuration
        configs_perf = instance$bmr$score(instance$measures)

        # select best mu_current indices
        if (length(msr_ids) < 2) {

          # single crit
          ordered_perf = order(
            configs_perf[[msr_ids]],
            decreasing = !to_minimize
          )

the indices are computed on the complete historical archive not the active_configs!

unfortunately this happens: R produces rows of NAs, for the "too large indices", then we later filter out the NAs with transpose from paradox. (we end up with an empty list, which runs the learner then in ints defaults)

how do we test against this: we actually should only solve this issue https://github.com/mlr-org/mlr3tuning/issues/228

then we would see it automatically

SebGGruber commented 4 years ago

Should be done now: I added a few more lines to only select the last few benchmark results of the size mu of the previous stage. To verify I used your assert from above. Commit: https://github.com/mlr-org/mlr3hyperband/commit/e3c75ff751b08e95a2a8359a205b68bf93e1b426