juliasilge
commented
2 years ago You'll want to use extract_workflow() on your final fit object, and then you can pipe to the gtsummary functions:
library(tidymodels)
data(two_class_dat)
my_split <- initial_split(two_class_dat)
glm_fit <-
workflow(Class ~ ., logistic_reg()) %>%
last_fit(my_split)
glm_fit
#> # Resampling results
#> # Manual resampling
#> # A tibble: 1 × 6
#> splits id .metrics .notes .predictions .workflow
#> <list> <chr> <list> <list> <list> <list>
#> 1 <split [593/198]> train/test split <tibble> <tibble> <tibble> <workflow>
extract_workflow(glm_fit) %>%
gtsummary::tbl_regression(exponentiate = TRUE) %>%
gtsummary::as_kable()
#> To take full advantage of model formatting, e.g. grouping categorical variables,
#> please add the following argument to the `workflows::add_model()` call:
#> `blueprint = hardhat::default_formula_blueprint(indicators = 'none')`
#> Extracting {workflows} model fit with `workflows::extract_fit_parsnip(x) %>% tbl_regression(...)`
#> Extracting {parsnip} model fit with `tbl_regression(x = x$fit, ...)`| Characteristic | OR | 95% CI | p-value |
|---|---|---|---|
| A | 0.28 | 0.19, 0.42 | \<0.001 |
| B | 43.8 | 23.9, 85.4 | \<0.001 |
Created on 2022-06-17 by the reprex package (v2.0.1)
You can check out all the possibility for object extraction here: https://hardhat.tidymodels.org/reference/hardhat-extract.html
themichjam
github-actions[bot]
The problem
I'm having trouble with feeding a last_fit model to a gtsummary
tbl_regressionso that odds ratios, confidence intervals and p-values can be reported. I'm following thetidymodelscase study walkthrough here: https://www.tidymodels.org/start/case-study/. The error is:When rerun with
extract_fit_parsnip()error:Reproducible example
The Hotel Bookings Data
Let's read the hotel data into R and randomly select 30% of the rows in the data set to avoid long computation times later on.
Note that your results will differ from the original article, since you are only using 30% of the data.
Let's look at proportions of hotel stays that include children and/or babies:
A first model: penalized logistic regression
Do you recall Evaluate your model with resampling article for data splitting?
Let's reserve 25% of the
hotelsdata for the test set:Now let's reserve another 20% of the
hotel_otherfor our validation set.Build the model
Let's specify a penalized logistic regression model using the lasso method. Note that we define
penalty = tune()so we can tune it in the next steps, and since we are using lasso method, we setmixture = 1.For more details try typing
?logistic_regon the console.Create the recipe
Remember the second article Preprocess your data with recipes?
Let's preprocess the data by creating a recipe:
Create the workflow
Let's bundle the model and recipe into a single
workflow():Create the grid for tuning
We can now tune our model, similar to what is shown in the previous article Tune model parameters. Let's create a grid with 30 values for the hyperparameter we would like to tune:
Train and tune the model
Let's train all these logistic regression models with 30 different hyperparameter values. We provide the validation set
val_set, so model diagnostics computed onval_setwill be available after the fit.Now visualize the validation set metrics by plotting the area under the ROC curve against the range of penalty values:
Get the best values for this hyperparameter:
Let's pick candidate model 12 with a penalty value of
0.00137:Note that because you are using less data, your mean ROC AUC will be slightly lower than what's shown in the article.
And visualize the validation set ROC curve:
A second model: tree-based ensemble
Let's try to improve our prediction performance by using a random forest model (model type), which we also explored in the Evaluate your model with resampling article.
Check number of cores to work with:
Set model specification and provide number of cores for parallelization while tuning.
Create the recipe and workflow
Let's create the recipe for the model.
Then bundle it with the model specification:
Train and tune the model
When we set up our parsnip model, we chose two hyperparameters for tuning:
We will use a space-filling design to tune with 12 candidate models (instead of 25, to reduce computation time).
Be patient here! Computing these results will take several minutes to complete if you are using the default RStudio Cloud resources (1 GB memory, 1 CPU).
Here are our top 5 random forest models, out of the 12 candidates:
Note that your results will be different and your accuracy will take a small hit since you are using less data (only 25% of the whole data set) and setting up a smaller grid to tune model hyperparameters.
But we're already getting much better results than our penalized logistic regression!
Let's plot the results:
Let's select the best model according to the ROC AUC metric. Our final tuning parameter values are:
Collect predictions for the best model: Note that we simply provide our best model's parameter values
rf_bestto subset it from a whole list of tuned models.Now, we can compare the validation set ROC curves for our top penalized logistic regression model and random forest model:
The random forest is uniformly better across event probability thresholds.
The last fit
Let's evaluate the model performance one last time with the held-out test set. We'll start by building our parsnip model object again from scratch with our best hyperparameter values from our random forest model:
Note that we added a new argument
importance = "impurity"toset_engineto get variable importance scores. This is an optional, engine-specific argument. To see its documentation, you need to read the documentation for the underlyingranger()function. To see it and other options, type?rangerin console.Now let's collect the metrics:
Now let's
pluckthe workflow and pull out the fit and visualize the variable importance scores for the top 20 features:Let's generate our last ROC curve to visualize:
Not bad!
The error is