Add experiment framework

[Saethox]

We want to have the ability to do (arbitrary) experiments, which support

• comparisons of different models,
• and comparisons of different parameter configurations (possibly nested),
• each with parameter tuning (possibly nested),
• using different (but reproducible) random states,
• and as parallelizable as possible,
• while logging all necessary setup (with mlflow),
• on multiple datasets / test functions.

[Saethox]

The following code snippet is a first approach to how we could specify possibly nested experiment setups.

experiment = ParameterTuning(
    MetaParameter(
        NestedExperiment([
            ParameterTuning(
                ParameterList(
                    Evaluation(SupRB2()),
                    parameter_list={'individual_optimizer__population_size': range(128)}),
                parameter_space={'rule_optimizer__mutation__sigma': (0, 2)},
            ),
            ParameterTuning(
                ParameterList(
                    Evaluation(DecisionTreeRegressor()),
                    parameter_list={'max_depth': range(5)},
                ),
                parameter_space={'criterion': ['squared_error', 'absolute_error']},
            ),
        ]),
        parameters={
            'random_state': 42,
            'cv': 10,
            'X_train': X_train,
            'y_train': y_train,
            'X_test': X_test,
            'y_test': y_test,
        }
    ),
    tuner=SkoptTuner(),
    random_state=0,
)

Outer components can overwrite parameters of inner components, e.g., the cv parameter of MetaParameter overwrites the cv parameter of Evaluation, such that both SupRB2 and the DecisionTreeRegressor are evaluated using 10-fold cv. Similar approaches could be used for injecting different datasets.

Parameter tuning would be possible on multiple levels, too. For example, we may want to tune the sigma of the mutation of SupRB2, set this tuned value as fixed, and then evaluate what happens if we try all values between 0 and 128 for the population_size. Using nesting, we could then tune some other parameter, e.g., n_iter, depending on the fixed population_size.

[Saethox]

A first concept implementation was pushed to master. It is very hacky though, so I'm not sure if I'm satisfied with this. I'm still torn over if we should just hard-code our experiments, because there are currently more urgent things to worry about.

I still like the concept, but it may be trickier than I thought to implement this correctly.