cimentadaj
commented
4 years ago Odd behavior and same problem but without a resample can be seen here. Trying to fit a bagged tree model with the pisa data. Without specifying a resample, the character VER_DAT gets expanded and fails in an error. I believe this is because bag_tree uses bootstraps behind the scenes, making this the same error as above.
However, this intuition doesn't fit the results since I can convert a column of mtcars to character and fit the same model and it first expanding the columns. Even more surprising, eliminating the VER_DAT column from pisa allows to run the model without a problem:
library(baguette)
#> Loading required package: parsnip
library(tidyflow)
library(rsample)
data_link <- "https://raw.githubusercontent.com/cimentadaj/ml_socsci/master/data/pisa_us_2018.csv"
pisa <- read.csv(data_link)
mod1 <- set_engine(bag_tree(mode = "regression"),
"rpart",
times = 3)
tflow <-
pisa %>%
tidyflow(seed = 23151) %>%
plug_split(initial_split) %>%
plug_formula(math_score ~ .) %>%
plug_model(mod1)
# The error is undefined columns. After inspecting what's happening,
# it is because the column VER_DAT gets expanded because it's a character
fit(tflow)
#> Error: All of the models failed. An example message was:
#> Error in `[.data.frame`(m, labs) : undefined columns selected
#> Timing stopped at: 0.844 0.02 0.864
# Works now
pisa$VER_DAT <- NULL
tflow %>%
replace_data(pisa) %>%
fit()
#> ══ Tidyflow [trained] ══════════════════════════════════════════════════════════
#> Data: 4.84K rows x 501 columns
#> Split: initial_split w/ default args
#> Formula: math_score ~ .
#> Resample: None
#> Grid: None
#> Model:
#> Bagged Decision Tree Model Specification (regression)
#>
#> Main Arguments:
#> cost_complexity = 0
#> min_n = 2
#>
#> Engine-Specific Arguments:
#> times = 3
#>
#> Computational engine: rpart
#>
#> ══ Results ═════════════════════════════════════════════════════════════════════
#>
#>
#> Fitted model:
#> Bagged CART (regression with 3 members)
#>
#> Variable importance scores include:
#>
#> # A tibble: 500 x 4
#>
#> ...
#> and 14 more lines.
# This is the same problem as the first comment in the issue.
# However, why doesn't this fail?
mtcars$gear <- as.character(mtcars$gear)
tflow %>%
replace_data(mtcars) %>%
replace_formula(mpg ~ .) %>%
fit() %>%
pull_tflow_fit()
#> parsnip model object
#>
#> Fit time: 507ms
#> Bagged CART (regression with 3 members)
#>
#> Variable importance scores include:
#>
#> # A tibble: 12 x 4
#> term value std.error used
#> <chr> <dbl> <dbl> <int>
#> 1 disp 748. 178. 3
#> 2 wt 672. 192. 3
#> 3 hp 636. 68.5 3
#> 4 drat 563. 121. 3
#> 5 cyl 218. 51.9 3
#> 6 qsec 122. 30.4 3
#> 7 carb 106. 99.1 2
#> 8 vs 65.3 23.0 3
#> 9 gear4 44.5 38.8 3
#> 10 gear3 22.9 21.6 3
#> 11 am 5.25 4.88 3
#> 12 gear5 0.016 0 1
# In fact, you can see that the column gear gets expanded and
# fitted correctly
When passing a formula to a
tidyflowthat has a character variables,hardhat::moldexpands character columns to haveNcolumns as the same number of categories in the variable. When this is passed tofit_resamples, it raises an error because these new columns weren't specified in the formula in the first place:Doesn't happen with a recipe because the recipe doesn't convert character columns to one hot encodings:
Reported at https://github.com/tidymodels/hardhat/issues/139. One solution to this would be to convert character to factors before passing it to
moldand then reconvert them back to characters from the result of mold. However, before doing it, I want to make suremoldis working correctly. Perhaps this is just an easy fix from their side.