The random sampling is only for the initialization
Each SH bracket (divided by budget) is held as a population and the populations will evolve independently, but the point is that each population will be augmented by elites from a lower-budget population (Figs. 2, 3)
The experiments are performed with 50 random seeds for each
Performance over time
Random search
HB
TPE
BOHB
SMAC
RE
DE
DEHB
The benchmark datasets are:
Counting one (toy func)
surrogate bench by the BOHB paper (probably ParamNet?)
BNN by the BOHB paper
RL by the BOHB paper
NAS benches (NB101, NB201, NB1shot1, HPOlib)
Ablation study
some parameters
Scalability test
$n \in \{1,2,4,8,16,32,64\}$
Average rank over time
I am not sure if the expected runtime for each benchmark is more or less similar, but anyways they took the average over NB101, NB201, HPOlib, NB1shot1, OpenML surrogates, and the RL bench.
Confusing points
Section 4
The algorithm was really hard to understand from the text.
So basically, each population is inherited from (1) the previous SH bracket with exactly the same budget and the population is augmented from (2) the pruned low-budget population as well.
The first point is exactly what we do in evolution algorithms in general.
The second point is described in Fig. 2 (the extra individual is called parent pool) and Fig. 3 (the right figure shows that we additionally uses the parent pool on top of the inherited population).
DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization
Main points
Performance over time
The benchmark datasets are:
Ablation study
some parameters
Scalability test
$n \in \{1,2,4,8,16,32,64\}$
Average rank over time
I am not sure if the expected runtime for each benchmark is more or less similar, but anyways they took the average over NB101, NB201, HPOlib, NB1shot1, OpenML surrogates, and the RL bench.
Confusing points
Section 4
The algorithm was really hard to understand from the text. So basically, each population is inherited from (1) the previous SH bracket with exactly the same budget and the population is augmented from (2) the pruned low-budget population as well. The first point is exactly what we do in evolution algorithms in general. The second point is described in Fig. 2 (the extra individual is called parent pool) and Fig. 3 (the right figure shows that we additionally uses the parent pool on top of the inherited population).