Setting custom metric raises error in AutoML.fit

[hguidara]

I tried using a custom metric function as argument to AutoML.fit but it raises errors. I made sure to follow the required function signature as mentionned in the documentation :

metric –

A string of the metric name or a function, e.g., ‘accuracy’, ‘roc_auc’, ‘roc_auc_ovr’, ‘roc_auc_ovo’, ‘f1’, ‘micro_f1’, ‘macro_f1’, ‘log_loss’, ‘mae’, ‘mse’, ‘r2’, ‘mape’. If passing a customized metric function, the function needs to have the follwing signature:

def custom_metric( X_test, y_test, estimator, labels, X_train, y_train, weight_test=None, weight_train=None, config=None, groups_test=None, groups_train=None, ): return metric_to_minimize, metrics_to_log which returns a float number as the minimization objective, and a tuple of floats or a dictionary as the metrics to log.

This is the code I am using :

automl = AutoML()

def custom_metric_foo(
    X_test, y_test, estimator, labels,
    X_train, y_train, weight_test=None, weight_train=None,
    config=None, groups_test=None, groups_train=None,
):
    return 0.5, 0.5

automl_settings = {
    "time_budget": 60,
    "metric" : custom_metric_foo,
    "task": 'classification',
    "log_file_name": f"log_dir/exp_{EXP_NUM}.log",
    "log_training_metric" : True,
}

automl.fit(
    X_train = X_train, y_train = y_train,
    X_val = X_val, y_val = y_val,
    **automl_settings
)

As you can see, the _custom_metricfoo function I am using is very basic for testing purpose.

And this is the error I get:

---

UnboundLocalError Traceback (most recent call last) ~\AppData\Local\Temp/ipykernel_21888/1282311363.py in 23 } 24 ---> 25 automl.fit( 26 X_train = X_train, y_train = y_train, 27 X_val = X_val, y_val = y_val,

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\automl.py in fit(self, X_train, y_train, dataframe, label, metric, task, n_jobs, log_file_name, estimator_list, time_budget, max_iter, sample, ensemble, eval_method, log_type, model_history, split_ratio, n_splits, log_training_metric, mem_thres, pred_time_limit, train_time_limit, X_val, y_val, sample_weight_val, groups_val, groups, verbose, retrain_full, split_type, learner_selector, hpo_method, starting_points, seed, n_concurrent_trials, keep_search_state, **fit_kwargs) 1362 with training_log_writer(log_file_name) as save_helper: 1363 self._training_log = save_helper -> 1364 self._search() 1365 else: 1366 self._training_log = None

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\automl.py in _search(self) 1764 1765 if self._n_concurrent_trials == 1: -> 1766 self._search_sequential() 1767 else: 1768 self._search_parallel()

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\automl.py in _search_sequential(self) 1606 ) 1607 start_run_time = time.time() -> 1608 analysis = tune.run( 1609 search_state.training_function, 1610 search_alg=search_state.search_alg,

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\tune\tune.py in run(training_function, config, low_cost_partial_config, cat_hp_cost, metric, mode, time_budget_s, points_to_evaluate, evaluated_rewards, prune_attr, min_resource, max_resource, reduction_factor, report_intermediate_result, search_alg, verbose, local_dir, num_samples, resources_per_trial, config_constraints, metric_constraints, use_ray) 345 if verbose: 346 logger.info(f'trial {num_trials} config: {trial_to_run.config}') --> 347 result = training_function(trial_to_run.config) 348 if result is not None: 349 if isinstance(result, dict):

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\automl.py in _compute_with_config_base(self, estimator, config_w_resource) 189 190 trained_estimator, val_loss, metric_forlogging, , pred_time = \ --> 191 compute_estimator( 192 sampled_X_train, 193 sampled_y_train,

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\ml.py in compute_estimator(X_train, y_train, X_val, y_val, weight_val, groups_val, budget, kf, config_dic, task, estimator_name, eval_method, eval_metric, best_val_loss, n_jobs, estimator_class, log_training_metric, fit_kwargs) 318 **config_dic, task=task, n_jobs=n_jobs) 319 if 'holdout' in eval_method: --> 320 val_loss, metric_for_logging, train_time, pred_time = get_test_loss( 321 config_dic, estimator, X_train, y_train, X_val, y_val, weight_val, 322 groups_val, eval_metric, task, budget=budget,

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\ml.py in get_test_loss(config, estimator, X_train, y_train, X_test, y_test, weight_test, groups_test, eval_metric, obj, labels, budget, log_training_metric, fit_kwargs) 195 # fit_kwargs['y_val'] = y_test 196 estimator.fit(X_train, y_train, budget, **fit_kwargs) --> 197 test_loss, metric_for_logging, predtime, = _eval_estimator( 198 config, estimator, X_train, y_train, X_test, y_test, 199 weight_test, groups_test, eval_metric, obj,

c:\users\hguidara\appdata\local\programs\python\python38\lib\site-packages\flaml\ml.py in _eval_estimator(config, estimator, X_train, y_train, X_test, y_test, weight_test, groups_test, eval_metric, obj, labels, log_training_metric, fit_kwargs) 182 test_pred_y = None 183 # eval_metric may return test_pred_y but not necessarily. Setting None for now. --> 184 return test_loss, metric_for_logging, pred_time, test_pred_y 185 186

UnboundLocalError: local variable 'pred_time' referenced before assignment

N.B: When I set the "metric" argument to any predefined metric everything goes well. (Example "metric" = "accuracy")

[sonichi]

@hguidara Thanks for catching this. As a workaround, you can return a dict for metrics_to_log. It would be even better if you could make a PR and handle the case when metrics_to_log is not a dict in _eval_estimator() of ml.py.