Conditions to summon bug (assume a blood offering has already been made):
Use BayesianOptPro
Use exclusively Categorical dimensions
(Kinda optional - See details below) At least one of the Categorical dimensions must be in FeatureEngineer
The above 3 conditions will cause BayesianOptPro to break outright. However, problems still arise when omitting FeatureEngineer, while searching an exclusively multi-Categorical space. The effect of this bug is that saved experiment results are not properly marked as similar_experiments. Tests demonstrating the behavior of this second bug are added in 4b60f1a. Tests for both variants of the bug are added in #152.
When the two xfail tests are marked as xpass, the bug is probably (hopefully) fixed.
Code to reproduce error:
from hyperparameter_hunter import Environment, BayesianOptPro, FeatureEngineer
from hyperparameter_hunter import Categorical, Integer
from hyperparameter_hunter.utils.learning_utils import get_boston_data
from sklearn.ensemble import AdaBoostRegressor
def nothing_transform(train_targets, non_train_targets):
return train_targets, non_train_targets, lambda _: _
env = Environment(
train_dataset=get_boston_data(),
results_path="HHAssets",
target_column="DIS",
metrics=["median_absolute_error"],
cv_params=dict(n_splits=3, random_state=1),
)
opt_1 = BayesianOptPro(iterations=5, random_state=32, n_initial_points=1)
opt_1.set_experiment_guidelines(
model_initializer=AdaBoostRegressor,
model_init_params=dict(), # FLAG: This breaks
# model_init_params=dict(n_estimators=Integer(10, 100)), # FLAG: This works
feature_engineer=FeatureEngineer([
Categorical([nothing_transform], optional=True),
]),
)
opt_1.go()
Notes
All other optimization protocols are fine with exclusively Categorical search spaces
Switching the two model_init_params in the code above demonstrates that the problem is bypassed by adding a non-Categorical dimension
Conditions to summon bug (assume a blood offering has already been made):
BayesianOptPro
Categorical
dimensionsCategorical
dimensions must be inFeatureEngineer
The above 3 conditions will cause
BayesianOptPro
to break outright. However, problems still arise when omittingFeatureEngineer
, while searching an exclusively multi-Categorical
space. The effect of this bug is that saved experiment results are not properly marked assimilar_experiments
. Tests demonstrating the behavior of this second bug are added in 4b60f1a. Tests for both variants of the bug are added in #152.When the two xfail tests are marked as xpass, the bug is probably (hopefully) fixed.
Code to reproduce error:
Notes
Categorical
search spacesmodel_init_params
in the code above demonstrates that the problem is bypassed by adding a non-Categorical
dimension