simonpcouch
commented
1 year ago Hey @SHo-JANG, thanks for the issue.
This is a really interesting use case, and I appreciate the helpful reprex! You’ve worked to integrate with the machinery in a very intuitive way.
I think the hitch you’re running into here is that tune expects the thing to be tuned to be an argument to the training function. This is where the Only one tunable value is currently allowed per argument error is coming from. This is why, for instance, we have a light wrapper xgboost::xgb.train() that “lifts” the params list arguments to be main arguments.
As such, you’ll need to specify the partials of the objective function as your grid argument. This will be your entry point for tuning many arguments as well, as you can manually specify any combination of hyperparameters that you’d like.
I realize this is sub-optimal, and means that you have to be careful in keeping track of which partial is which, though I don’t anticipate we’ll develop this interface further as this is an uncommon use case and would require some fundamental changes to tune. You can use the extracts, though, to be sure you’ve kept track of hyperparameters rigorously.
Mirroring your reprex:
library(tidymodels)
# focal_loss --------------------------------------------------------------
focal_loss <- function(preds, dtrain, alpha = 0.5,focal_gamma = 0) {
labels <- getinfo(dtrain, "label")
preds <- 1 / (1 + exp(-preds))
p<- preds
y <- labels
grad <- (y*alpha*(1-p)^focal_gamma*(focal_gamma*log(p)*p-(1-p))-
(1-y)*(1-alpha)*p^(focal_gamma)*(focal_gamma*log(1-p)*(1-p)-p))
du <- y*alpha*(1-p)^(focal_gamma-1)*(log(p)*(-focal_gamma^2 *p + (1-p)*focal_gamma)+ 2*focal_gamma*(1-p)+(1-p))
dv <- -(1-y)*(1-alpha)*p^(focal_gamma-1)*(log(1-p)*(focal_gamma^2*(1-p)-p*focal_gamma)-2*focal_gamma*p-p)
hess <- (du+dv)*p*(1-p)
return(list(grad = grad, hess = hess))
}
# data setup ----------------------------------------------------------------
data <- two_class_dat %>%
rename(y = Class)
set.seed(100)
splits <- initial_split(data,prop = 0.8, strata = y)
train_data <- training(splits)
test_data <- testing(splits)
resamples <- vfold_cv(data = train_data,v = 5,strata = y)
# specifications ----------------------------------------------------
xgb_spec <-
boost_tree(mode = "classification", tree_depth = tune(), trees = tune()) %>%
# note that i just set `objective = tune()` here
set_engine(object = ., engine = "xgboost", objective = tune())
xgb_recipe <- train_data %>%
recipe(y ~ .) |>
#step_mutate_at(all_numeric_predictors(), fn = list(orig = ~.)) %>%
step_normalize(all_predictors(), -all_outcomes())
xgb_workflow <-
workflow() %>%
add_recipe(xgb_recipe) %>%
add_model(xgb_spec)
# grid setup ------------------------------------------------------------------
partial_grid <-
expand_grid(
alpha = seq(0, 1, length.out = 3),
focal_gamma = seq(0, 5, length.out = 3)
)
partial_grid$partials <-
map2(
partial_grid$alpha,
partial_grid$focal_gamma,
~partial(focal_loss, alpha = !!.x, focal_gamma = !!.y)
)
xgb_grid <-
extract_parameter_set_dials(xgb_spec) %>%
filter(id != "objective") %>%
grid_latin_hypercube(size = 9) %>%
bind_cols(partial_grid %>% select(objective = partials))
# tune! -----------------------------------------------------------------------
res <- tune_grid(xgb_workflow, resamples = resamples, grid = xgb_grid)
#> → A | error: Error in xgb.iter.update(bst$handle, dtrain, ...
#> There were issues with some computations A: x5
#>
#> Warning: All models failed. Run `show_notes(.Last.tune.result)` for more
#> information.Created on 2023-03-09 with reprex v2.0.2
That said, as you can see, there's an issue here: supplying partials of the function as part of a tibble column means that we need to wrap that collection of functions as a list. So, tune passes each possible value of objective as a one-element list containing the function, rather than the function itself:
Browse[2]> finalize_workflow_spec(workflow, iter_grid_model)
══ Workflow ══════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: boost_tree()
── Preprocessor ──────────────────────────────────────────────────────────────
1 Recipe Step
• step_normalize()
── Model ─────────────────────────────────────────────────────────────────────
Boosted Tree Model Specification (classification)
Main Arguments:
trees = 1886
tree_depth = 2
Engine-Specific Arguments:
objective = list(structure(function (...) {focal_loss <- function (preds,<snip>
Computational engine: xgboost and xgboost thus trips up by saying that it doesn't know what to do with a list for an objective function.
This is where that change from objective = tune() to objective = list(...) happens:
but that change might actually need to happen in workflows.
So, for now, this does not work, but we're on it. :)
SHo-JANG
github-actions[bot]
I want to customize objective function.
For example, what I want to implement now is a loss function called Focal loss, which can be used when there is a class imbalance. Focal Loss
Because focal loss is a generalization of cross-entropy, setting certain hyperparameters will result in exactly the same result as CE.
I want to tuning alpha , focal_gamma
Created on 2023-02-28 with reprex v2.0.2
I checked that it works well on the existing xgb.train, but I don't know how to apply the tune function. From now on, the code below is the way I tried.
Created on 2023-02-28 with reprex v2.0.2
The
extract_parameter_set_dials()function does not recognize thefocal_gammaargument. How I can tuning objective function? Ultimately, I want to tune not only one parameter but also several parameters together.