hyperparameters associated to`extract`ed objects are incorrect

[simonpcouch]

The hyperparameters attached to extract entries are incorrect for the models from the initial grid.

library(tidymodels)

set.seed(8825)
mt_knn_bo <-
  tune_bayes(
    workflow(mpg ~ ., nearest_neighbor("regression", neighbors = tune())),
    resamples = vfold_cv(mtcars, v = 5),
    iter = 3,
    control = control_bayes(extract = function(x) {list()})
  )

# same neighbors for all model configurations
mt_knn_bo %>%
  collect_extracts()
#> # A tibble: 30 × 5
#>    id    .iter neighbors .extracts  .config             
#>    <chr> <int>     <int> <list>     <chr>               
#>  1 Fold1     0        13 <list [0]> Preprocessor1_Model1
#>  2 Fold1     0        13 <list [0]> Preprocessor1_Model2
#>  3 Fold1     0        13 <list [0]> Preprocessor1_Model3
#>  4 Fold2     0        13 <list [0]> Preprocessor1_Model1
#>  5 Fold2     0        13 <list [0]> Preprocessor1_Model2
#>  6 Fold2     0        13 <list [0]> Preprocessor1_Model3
#>  7 Fold3     0        13 <list [0]> Preprocessor1_Model1
#>  8 Fold3     0        13 <list [0]> Preprocessor1_Model2
#>  9 Fold3     0        13 <list [0]> Preprocessor1_Model3
#> 10 Fold4     0        13 <list [0]> Preprocessor1_Model1
#> # ℹ 20 more rows

# though metrics shows neighbors did vary for these .configs
mt_knn_bo %>%
  collect_metrics()
#> # A tibble: 12 × 8
#>    neighbors .metric .estimator  mean     n std_err .config              .iter
#>        <int> <chr>   <chr>      <dbl> <int>   <dbl> <chr>                <int>
#>  1         4 rmse    standard   2.82      5  0.463  Preprocessor1_Model1     0
#>  2         4 rsq     standard   0.824     5  0.0331 Preprocessor1_Model1     0
#>  3         9 rmse    standard   2.80      5  0.517  Preprocessor1_Model2     0
#>  4         9 rsq     standard   0.823     5  0.0269 Preprocessor1_Model2     0
#>  5        13 rmse    standard   2.87      5  0.569  Preprocessor1_Model3     0
#>  6        13 rsq     standard   0.814     5  0.0377 Preprocessor1_Model3     0
#>  7         8 rmse    standard   2.80      5  0.510  Iter1                    1
#>  8         8 rsq     standard   0.826     5  0.0231 Iter1                    1
#>  9         7 rmse    standard   2.80      5  0.504  Iter2                    2
#> 10         7 rsq     standard   0.827     5  0.0203 Iter2                    2
#> 11         1 rmse    standard   3.13      5  0.374  Iter3                    3
#> 12         1 rsq     standard   0.775     5  0.0677 Iter3                    3

^{Created on 2023-09-27 with reprex v2.0.2}

Note that the issue shown for mt_knn_bo applies to the object in test_objects.RData by the same name.

[simonpcouch]

Ah, okay. There's a bit more going on here than I had wrapped my head around.👻

In this example, for each fold, we fit one model—with 13 neighbors—and then are able to predict across several different values of neighbors with that one model. When we extract, though, we're only extracting from that one fit with 13 neighbors. I'm not sure that it's incorrect for us to attach that same hyperparameter to all of the extracts associated with it, though it's probably confusing to then attach that .config to the hyperparameter combination that it's otherwise not associated with in our output.

I'm conflicted between

1) documenting the current behavior, or 2) revising the recycling here so that we only return extracts for the models that we've fitted (and documenting it)

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.