[FEATURE] Feature selection and optimization simultaneously

[doodledood]

It seems to me that the best approach to optimizing an estimator would be to run both feature selection AND hyperparameter optimization simultaneously within the same evolution process. It would be complex but probably yield better results instead of using one after the other.

Is this something I can do within the current framework, or does this require new code?

Also, do you think it is even a good idea in the first place?

[rodrigo-arenas]

Hi @doodledood This is technically possible, although I'd not recommend doing it together, since you have to create a multi-objective function and there will be trade-offs between the two objectives, it also increases the space complexity of the search since now a point in the space is not just one combination of hyperparameters but those hyperparameters + a set of features

This could deviate the optimizer from good solutions, let's say you found good hyperparameters but the feature selection part only selected one bad feature, the optimizer will "discard" this solution since the overall score will probably be bad and it might avoid exploring those good hyperparameters with another set of features

I'd suggest first running the feature selection and then the hyperparameters optimization

For now, I'd not implement this in the package

Let me know if you have more questions about it

[rodrigo-arenas]

I'm closing this issue for now, let me know if you have more questions regarding this subject