When introducing a fidelity parameter with is_fidelity=True in RandomSearch, it will return configs where the value of the key, for example "epoch" will be set to None, rather than the maximum fidelity.
This is problematic as if you want to compare RS to some MF algorithm like SH, then you can't just switch out your optimizer with specific code to account for this
When introducing a fidelity parameter with
is_fidelity=TrueinRandomSearch, it will return configs where the value of the key, for example"epoch"will be set to None, rather than the maximum fidelity.This occurs in
SearchSpace.sample:https://github.com/automl/neps/blob/82b48903d0896db7d84ea2810f567a29d01688c3/neps/search_spaces/search_space.py#L335-L338
This is problematic as if you want to compare RS to some MF algorithm like SH, then you can't just switch out your optimizer with specific code to account for this