Errors with Optuna search space and bohb search spaces

[RNarayan73]

Hello,

When running TuneSearchCV with the optuna search_optimization, it throws up the error below. Similar errors occur when I use the tune search apis- tune.randint, tune.uniform, tune.loguniform. "RuntimeError: Unknown Optuna search space: <ray.tune.sample.Integer object at 0x00000211AECCD220>"

This may be a recent development, because I was previously able to run the optuna version search fine. In fact, the Tune documentation for search with optuna here (https://docs.ray.io/en/master/tune/api_docs/suggestion.html) suggests that

Tune automatically converts search spaces to Optuna’s format.

Similarly, when I had run the bohb search optimization with the HyperBandForBOHB scheduler class previously, it had run fine, Now, I get an search space error: "TypeError: The method add_hyperparameter must be called with an instance of ConfigSpace.hyperparameters.Hyperparameter."

Is anyone aware of new issues around these cropping up recently?

Regards Narayan

Optuna error

RuntimeError Traceback (most recent call last) ~\AppData\Local\Temp/ipykernel_7060/1428500195.py in ----> 1 ray_pipe.fit(X_train, y_train)

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_basesearch.py in fit(self, X, y, groups, tune_params, fit_params) 629 ray_kwargs["local_mode"] = True 630 with ray_context(ray_kwargs): --> 631 return self._fit(X, y, groups, tune_params, **fit_params) 632 633 def score(self, X, y=None):

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_basesearch.py in _fit(self, X, y, groups, tune_params, **fit_params) 534 535 self._fill_config_hyperparam(config) --> 536 analysis = self._tune_run(config, resources_per_trial, tune_params) 537 538 self.cvresults = self._format_results(self.n_splits, analysis)

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_search.py in _tune_run(self, config, resources_per_trial, tune_params) 762 warnings.warn("'seed' is not implemented for Optuna.") 763 if override_search_space: --> 764 search_space = self._get_optuna_params() 765 search_algo = OptunaSearch(space=search_space, **search_kwargs) 766 run_args["search_alg"] = search_algo

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_search.py in _get_optuna_params(self) 566 config_space[param_name] = tune.choice(space) 567 else: --> 568 raise RuntimeError(f"Unknown Optuna search space: {space}") 569 return OptunaSearch.convert_search_space(config_space) 570

RuntimeError: Unknown Optuna search space: <ray.tune.sample.Integer object at 0x00000211AECCD220>

BOHB error

~\AppData\Local\Temp/ipykernel_7360/1428500195.py in ----> 1 ray_pipe.fit(X_train, y_train)

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_basesearch.py in fit(self, X, y, groups, tune_params, fit_params) 629 ray_kwargs["local_mode"] = True 630 with ray_context(ray_kwargs): --> 631 return self._fit(X, y, groups, tune_params, **fit_params) 632 633 def score(self, X, y=None):

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_basesearch.py in _fit(self, X, y, groups, tune_params, **fit_params) 534 535 self._fill_config_hyperparam(config) --> 536 analysis = self._tune_run(config, resources_per_trial, tune_params) 537 538 self.cvresults = self._format_results(self.n_splits, analysis)

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_search.py in _tune_run(self, config, resources_per_trial, tune_params) 750 elif self._search_optimization_lower == "bohb": 751 if override_search_space: --> 752 search_space = self._get_bohb_config_space() 753 search_algo = TuneBOHB( 754 space=search_space, seed=self.seed, **search_kwargs)

C:\Anaconda3\envs\Scikit-Learn\lib\site-packages\tune_sklearn\tune_search.py in _get_bohb_config_space(self) 535 name=param_name, choices=space)) 536 else: --> 537 config_space.add_hyperparameter(space) 538 return config_space 539

ConfigSpace\configuration_space.pyx in ConfigSpace.configuration_space.ConfigurationSpace.add_hyperparameter()

TypeError: The method add_hyperparameter must be called with an instance of ConfigSpace.hyperparameters.Hyperparameter.

[Yard1]

Thanks, will check

[Yard1]

@RNarayan73 Would it be possible for you to provide code to reproduce?

[RNarayan73]

@Yard1 When I tried to replicate the issue with some simple code, I hit upon the root cause. I had included a list of categorical hyperparameters in my search space. This list worked fine while using TuneSearchCV with the 'random', 'bayesian' and 'hyperopt' search_optimization options, but caused 'optuna' and 'bohb' to fail. When I used ray.tune.choice to add the list to the search space, 'optuna' and 'bohb' work fine as did the others.

Minor annoyance, but perhaps it would help to make the behaviour of all the search_optimization options consistent. Do you want me to still provide you code to fix this? Or should I just close the issue?

Narayan

[Yard1]

Ah, yeah, that would be the cause. We are hoping to refactor how we interact with search spaces eventually. I'll close this for now, thanks!