Ambiguity in the query

[nabenabe0928]

This is not exactly an issue but leads to misunderstandings, so I would like to mention it here.

1. Which will be taken into account, nepochs in the argument of Benchmark.__call__ or epoch in config dict? (I know the answer is nepochs, but it is ambiguous until we run)
2. We do not get any out of domain errors for numerical parameters (e.g. we can specify N=100) and it seems fidelity parameters are rounded somehow?
3. Continuous parameter also does not have out-of-domain errors and seems to be rounded?

I would really appreciate if we can choose a mode, either clipping automatically or raising errors anytime you get invalid items.

[eddiebergman]

I guess the surrogate is fine with out of domain values but i agree that an out of domain check would be helpful. I ended up creating my own configs with a validate() and manually specifying the bounds to counter this.

[NeoChaos12]

This is a very valid point, thank you for bringing it up! The way the number of epochs is currently being handled is definitely sub-optimal and will be revised in the upcoming updates.

Regarding the domain checks, this is something that requires some discussion.

• For the surrogate, there is technically no reason for forbidding out-of-domain values as long as a user is aware that our underlying model, XGBoost, is not very suitable for extrapolation. My personal preference would be to only raise a warning when out-of-domain values are encountered. Adding the option to enforce domain bounds and/or raise an error instead would then be a nice-to-have for users who want that.
• For the tabular dataset, domain checks actually make no sense, since the dataset can only be queried on the specific values that were originally sampled in any case. The thought behind the current design was that exposing the ConfigSpace implementation of the search space is enough to provide the requisite information to any algorithms that need the domain bounds, but maybe this can be improved.

What is your opinion?

[nabenabe0928]

Hi, thanks for the response:)

For the tabular, I don't think we need the check, but for the surrogate model, it would be grateful to have warning at least because XGBoost gives, by nature of the decision tree, the same prediction on each parameter which is rounded to the original range.