DEHB and non-async DE optimiser update design

[drpriscu]

Dear @Neeratyoy and @noorawad, according to the paper "DEHB: Evolutionary Hyperband for Scalable, Robust, and Efficient Hyperparameter Optimization," DEHB employs an immediate update design for DE by utilising the AsyncDE class. However, despite passing the async strategy argument in DEHB, it does not change the async strategy of the AsyncDE class, which by default is 'deferred.'

As I understand, the DE class currently only uses deferred updates. If one would like to compare DEHB and its DE component as separate optimisers for optimising a model's validation score, would the update design of deferred vs immediate have a significant impact on the final validation score or simply the wall-clock time? I'd like to compare DE and DEHB using the same update design, and the paper suggests that the immediate update design outperforms deferred DE.

Thank you.

[Neeratyoy]

Hi @drpriscu

To simply compare DEHB and DE, if you run DEHB with deferred then you should compare it with DE. If you run DEHB with immediate you should compare it with AsyncDe with immediate updates.

For reproducing experiment results from the paper, the experiments branch is the closest code version.

Closing this for now, as no code issues have been detected yet. Please feel free to reopen with the code snippet that needs to be looked into. Thanks!

[drpriscu]

Thank you for your response. I would like to run DEHB and DE with immediate, as according to the paper, DE was outperformed by Random Search because it used a deferred strategy, whereas DEHB performed better because it uses an immediate update design for DE:

"DEHB uses the immediate update design for DE, wherein it updates the population immediately after a DE selection, and not wait for the entire population to evolve. We posit that this feature, along with lower fidelity search, and performing grid search over population sizes with Hyperband, enables DEHB to be more practical than classical-DE.".

According to the experiments branch, the DEHB class stores the async_strategy with the value immediate in de_params, which is sent to the AsyncDE class. However, the code issue in the pip version is that it never stores the async_strategy in the DEHB class, so when the DEHB class calls AsyncDE and passes the de_params, AsyncDE always implements the default async_strategy value of deferred.

Could you please update DEHB to implement the async strategy value passed so that AsyncDE uses the desired value?

dehb.py: lines 41-57

# DE related variables
self.mutation_factor = mutation_factor
self.crossover_prob = crossover_prob
self.strategy = strategy
self.fix_type = boundary_fix_type
self.max_age = max_age
self.de_params = {
    "mutation_factor": self.mutation_factor,
    "crossover_prob": self.crossover_prob,
    "strategy": self.strategy,
    "configspace": self.configspace,
    "boundary_fix_type": self.fix_type,
    "max_age": self.max_age,
    "cs": self.cs,
    "dimensions": self.dimensions,
    "f": f
}

line 357:

self.de[b] = AsyncDE(**self.de_params, budget=b, pop_size=self._max_pop_size[b])

de.py: lines 548 - 551

class AsyncDE(DE):
    def __init__(self, cs=None, f=None, dimensions=None, pop_size=None, max_age=np.inf,
                 mutation_factor=None, crossover_prob=None, strategy='rand1_bin',
                 budget=None, async_strategy='deferred', **kwargs):                 

[Neeratyoy]

Hi @drpriscu

This issue is open again. Please feel free to link the issue and post a PR. I can take a look and merge it to master if all looks good.

Thanks!

[drpriscu]

class DEHBBase:
    def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=None,
                 crossover_prob=None, strategy=None, min_budget=None,
                 max_budget=None, eta=None, min_clip=None, max_clip=None,
                 boundary_fix_type='random', max_age=np.inf, async_strategy='deferred', **kwargs):
        # Benchmark related variables
        self.cs = cs
        self.configspace = True if isinstance(self.cs, ConfigSpace.ConfigurationSpace) else False
        if self.configspace:
            self.dimensions = len(self.cs.get_hyperparameters())
        elif dimensions is None or not isinstance(dimensions, (int, np.int)):
            assert "Need to specify `dimensions` as an int when `cs` is not available/specified!"
        else:
            self.dimensions = dimensions
        self.f = f

        # DE related variables
        self.mutation_factor = mutation_factor
        self.crossover_prob = crossover_prob
        self.strategy = strategy
        self.fix_type = boundary_fix_type
        self.max_age = max_age
        self.async_strategy = async_strategy
        self.de_params = {
            "mutation_factor": self.mutation_factor,
            "crossover_prob": self.crossover_prob,
            "strategy": self.strategy,
            "configspace": self.configspace,
            "boundary_fix_type": self.fix_type,
            "max_age": self.max_age,
            "cs": self.cs,
            "dimensions": self.dimensions,
            "f": f,
            "async_strategy": self.async_strategy
        }

class DEHB(DEHBBase):
    def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=0.5,
                 crossover_prob=0.5, strategy='rand1_bin', min_budget=None,
                 max_budget=None, eta=3, min_clip=None, max_clip=None, configspace=True,
                 boundary_fix_type='random', max_age=np.inf, n_workers=None, client=None, async_strategy='deferred', **kwargs):
        super().__init__(cs=cs, f=f, dimensions=dimensions, mutation_factor=mutation_factor,
                         crossover_prob=crossover_prob, strategy=strategy, min_budget=min_budget,
                         max_budget=max_budget, eta=eta, min_clip=min_clip, max_clip=max_clip,
                         configspace=configspace, boundary_fix_type=boundary_fix_type,
                         max_age=max_age, async_strategy=async_strategy, **kwargs)

[Neeratyoy]

Thanks for bringing the issue to notice!

It has been updated and the PyPi v0.0.4 should have this change included.