[Bug] ray.tune.run report KeyError: 'ml/domain' with Flaml search

[bladesaber]

Search before asking

• [X] I searched the issues and found no similar issues.

Ray Component

Ray Tune

What happened + What you expected to happen

Hello, I meet a strange Key Error. The detail is below:

Traceback (most recent call last): File "/home/bot/Desktop/person/AutoML_Pipeline/test.py", line 35, in verbose=False, File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/tune.py", line 597, in run runner.step() File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 696, in step next_trial = self._get_next_trial() # blocking File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 839, in _get_next_trial self._update_trial_queue(blocking=wait_for_trial) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 1308, in _update_trial_queue trial = self._search_alg.next_trial() File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/suggest/search_generator.py", line 90, in next_trial self._experiment.dir_name) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/suggest/search_generator.py", line 97, in create_trial_if_possible suggested_config = self.searcher.suggest(trial_id) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/ray/tune/suggest/suggestion.py", line 395, in suggest suggestion = self.searcher.suggest(trial_id) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/flaml/searcher/blendsearch.py", line 634, in suggest skip = self._should_skip(choice, trial_id, config, space) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/flaml/searcher/blendsearch.py", line 739, in _should_skip config_signature = self._ls.config_signature(config, space) File "/home/bot/anaconda3/envs/torch-environment/lib/python3.7/site-packages/flaml/searcher/flow2.py", line 527, in config_signature domain = space[key] KeyError: 'ml/domain'

Looking forward to your help as soon as possible, Thanks ^_^

Versions / Dependencies

the version of ray[tune] is 1.9.1

Reproduction script

import random from ray import tune from ray.tune.schedulers import AsyncHyperBandScheduler from ray.tune.suggest.flaml import BlendSearch

def train(config: dict): print(config) tune.report(loss=random.random())

if name == 'main':

algo = BlendSearch()
algo = tune.suggest.ConcurrencyLimiter(algo, max_concurrent=1)
scheduler = AsyncHyperBandScheduler()
choices = []
for i in range(3):
    choices.append({
        'name': 'id%d'%i,
        'domain': tune.choice(['a', 'b', 'c']),
        'test': tune.uniform(0.0, 10.0)
    })
space = {'ml': tune.choice(choices)}

analysis = tune.run(
    train,
    metric="loss",
    mode="min",
    search_alg=algo,
    scheduler=scheduler,
    num_samples=2,
    config=space,
)

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[xwjiang2010]

Was able to repro on master. @Yard1 Can you take a look?

[Yard1]

@bladesaber can you make sure your FLAML version is up to date? pip install -U flaml

[amogkam]

I can reproduce this with the latest flaml version (1.0.0). Seems like flaml does not support nested search spaces-- Tune should fail with a better error message in that case.

[Yard1]

cc @qingyun-wu

[EddyJens]

Same here, but with my base_learning_rate:

KeyError: 'base_learning_rate'

I'm using with the Repeater and everything went fine during the repeater, but when next iteration of new hyperparameters start to run it raises the KeyError (same code works fine with Optuna and other search algorithms)

[LinWencong]

I meet the same issue; I try to make FLAML version(1.0.13) up to date.
ray version == 1.13.0

[yxtay]

I think this is related. I also faced this issue when search_space contains a nested dict.

https://github.com/microsoft/FLAML/issues/1244