Transforming Tikhonov Regularization into Hyperparameter

[eliseoe]

Would it make sense to make the Tikhonov weighing of the ridge regression a parameter to validate during parameter tuning? We would have to add it to the ESN dictionary instead of giving it separately.

[eliseoe]

• removed the option to change Tikhonov parameter during training in esn.py
• added Tikhonov regularization as a parameter in the ESN dictionary (init), changed in esn.py
• allowed for validation over Tikhonov parameter in validate.py

[defneozan]

I found out that in skopt we can also pass discrete values, in case we want to search over discrete values of tikhonov instead of continuous. We can code it like this,

def create_search_space(n_param, grid_range, param_names):
    search_space = [None] * n_param
    for param_idx in range(n_param):
        if param_names[param_idx] == 'tikhonov':
            search_space[param_idx] = Integer(
            *grid_range[param_idx], name=param_names[param_idx]
        )
        else:
            search_space[param_idx] = Real(
                *grid_range[param_idx], name=param_names[param_idx]
            )

    return search_space