search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.3k stars 5.63k forks source link

[Tune] AxSearch not compatible with ax-platform >= 0.3.7 #45720

Open garymm opened 4 months ago

garymm commented 4 months ago

What happened + What you expected to happen

ax-platform 0.3.7 removed the objective_name agrument from AxClient.create_experiment

Versions / Dependencies

Ray: 2.23.0 ax-platform: 0.3.7

Reproduction script

I'm guessing running this test will fail with a new-enough ax-platform: https://github.com/ray-project/ray/blob/9f03bbcf9f9c837f832cc081d18f92ede86a3973/python/ray/tune/tests/test_searchers.py

Issue Severity

Medium: It is a significant difficulty but I can work around it.

KantiCodes commented 3 months ago

+1

mgrange1998 commented 3 months ago

Hi, is it possible to use the "objectives" field in place of the "objective_name" field? https://github.com/facebook/Ax/blob/0dce67c96289a31d35370fc6c74471898d9bc469/ax/service/ax_client.py#L251

KantiCodes commented 3 months ago

I confirm that editing the:

with the code below works

            self._ax.create_experiment(
                parameters=self._space,
                objectives={
                    self._metric: ObjectiveProperties(minimize=self._mode != "max",)
                },
                parameter_constraints=self._parameter_constraints,
                outcome_constraints=self._outcome_constraints,
            )

However now I am getting different error - probably mistake on my end

Traceback (most recent call last):
  File "/home/bartek/Projects/GNN-planning-paper/src/runner.py", line 260, in <module>
    runner = Runner(
  File "/home/bartek/Projects/GNN-planning-paper/src/runner.py", line 134, in __init__
    self.run_hyper()
  File "/home/bartek/Projects/GNN-planning-paper/src/runner.py", line 195, in run_hyper
    results = tuner.fit()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/tuner.py", line 377, in fit
    return self._local_tuner.fit()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/impl/tuner_internal.py", line 476, in fit
    analysis = self._fit_internal(trainable, param_space)
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/impl/tuner_internal.py", line 595, in _fit_internal
    analysis = run(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/tune.py", line 992, in run
    runner.step()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/execution/tune_controller.py", line 679, in step
    self._maybe_update_trial_queue()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/execution/tune_controller.py", line 779, in _maybe_update_trial_queue
    if not self._update_trial_queue(blocking=not dont_wait_for_trial):
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/execution/tune_controller.py", line 568, in _update_trial_queue
    trial = self._search_alg.next_trial()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/search/search_generator.py", line 99, in next_trial
    return self.create_trial_if_possible(self._experiment.spec)
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/search/search_generator.py", line 105, in create_trial_if_possible
    suggested_config = self.searcher.suggest(trial_id)
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ray/tune/search/ax/ax_search.py", line 303, in suggest
    parameters, trial_index = self._ax.get_next_trial()
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/utils/common/executils.py", line 163, in actual_wrapper
    return func(*args, **kwargs)
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/service/ax_client.py", line 539, in get_next_trial
    generator_run=self._gen_new_generator_run(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/service/ax_client.py", line 1790, in _gen_new_generator_run
    return not_none(self.generation_strategy).gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/generation_strategy.py", line 370, in gen
    return self._gen_multiple(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/generation_strategy.py", line 683, in _gen_multiple
    generator_run = self._curr.gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/generation_node.py", line 712, in gen
    gr = super().gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/generation_node.py", line 272, in gen
    generator_run = self._gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/generation_node.py", line 334, in _gen
    return model_spec.gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/model_spec.py", line 221, in gen
    return fitted_model.gen(**model_gen_kwargs)
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/base.py", line 786, in gen
    gen_results = self._gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/modelbridge/random.py", line 76, in _gen
    X, w = self.model.gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/models/random/sobol.py", line 112, in gen
    points, weights = super().gen(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/models/random/base.py", line 168, in gen
    raise e
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/models/random/base.py", line 128, in gen
    points, attempted_draws = rejection_sample(
  File "/home/bartek/Projects/GNN-planning-paper/.venv/lib/python3.10/site-packages/ax/models/model_utils.py", line 149, in rejection_sample
    raise SearchSpaceExhausted(
ax.exceptions.core.SearchSpaceExhausted: Rejection sampling error (specified maximum draws (10000) exhausted, without finding sufficiently many (1) candidates). This likely means that there are no new points left in the search space.
KantiCodes commented 3 months ago

Have mercy on my poor soul Im new to open source contributions

woshiyyya commented 3 months ago

Hi @KantiCodes, does that fix in ax_search.py works for you?

KantiCodes commented 3 months ago

Yes, it did