Louquinze
commented
2 years ago Hi, For beginning, I'm just a student assistant working on this project as well, so don't take this answer as final.
But as i see it. It is like this.
initial_configurations_via_metalearning: int = 25 is an integer value and sets the prior of hyperparameter optimization. Later in the code we than choose n values out off all possible metalearning configurations.
def suggest_via_metalearning(
meta_base, dataset_name, metric, task, sparse, num_initial_configurations, logger
):
if task == MULTILABEL_CLASSIFICATION:
task = MULTICLASS_CLASSIFICATION
task = TASK_TYPES_TO_STRING[task]
logger.info(task)
start = time.time()
ml = MetaLearningOptimizer(
dataset_name=dataset_name,
configuration_space=meta_base.configuration_space,
meta_base=meta_base,
distance="l1",
seed=1,
logger=logger,
)
logger.info("Reading meta-data took %5.2f seconds", time.time() - start)
runs = ml.metalearning_suggest_all(exclude_double_configurations=True)
return runs[:num_initial_configurations]This configurations are used as metalearning_configurations in smac (https://automl.github.io/SMAC3/master/). As i now smac is not limiting the given search space. So i think it is like your case 2.
autosklearn/smbo.py:526
smac_args = {
"scenario_dict": scenario_dict,
"seed": seed,
"ta": ta,
"ta_kwargs": ta_kwargs,
"metalearning_configurations": metalearning_configurations,
"n_jobs": self.n_jobs,
"dask_client": self.dask_client,
}
jmren168
eddiebergman
Hi,
After reading "initializing Bayesian Hyperparameter Optimization via Meta-Learning", I have a question about how initial_configurations_via_metalearning works in auto-sklearn, and hope someone could give me some hits. Many thanks.
When I enable initial_configurations_via_meta_learning to train a dataset D_n+1, and auto-sklearn found that the most similar dataset was D_J with a specific configuration theta_D_J, how this result is used to drive auto-sklearn to select an initial configuration for dataset D_n+1? case 1. theta_D_J is directly used as an initial configuration for dataset D_n+1 case 2. use dataset D_J to search another configuration (say theta_D_J_new) without limiting models, preprocessors, ..., and then theta_D_J_new is used as initial configuration for dataset D_n+1
If the answer is case 1, but the above paper mentioned only 3 classifiers (a linear SVM, an RBF SVM, and RF) are used in meta-learning, and I found that the type of model for dataset D_n+1 is mlp. This does not make sense. Any comments are highly appreciated.
10 | 48 | 0.00 | mlp | 0.236979 | 7.935055 | 9 | 0.218424 | 0 | 1.659342e+09 | 1.659342e+09 | 0.0 | StatusType.SUCCESS | [] | [feature_agglomeration] | none | Initial design -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 6 | 79 | 0.00 | mlp | 0.244792 | 53.374027 | 5 | 0.175456 | 0 | 1.659342e+09 | 1.659342e+09 | 0.0 | StatusType.SUCCESS | [] | [feature_agglomeration] | weighting | Initial design -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --