Open aldder opened 2 years ago
Thanks for the PR! I agree that overfitting can become an issue. Currently, there is the option
k_features="parsimonious"
which will select the smallest feature set within 1 standard error of the best feature set, which helps with this.
I like adding an early_stop
option, but I have a few suggestions / concerns regarding the API:
I think that the two parameters
early_stop
and early_stop_rounds
can be consolidated into a single one. E.g.,
if self.early_stop and k != k_to_select:
if k_score <= best_score:
could be
if self.early_stop_rounds and k != k_to_select:
if k_score <= best_score:
What I mean is instead of having
early_stop : bool (default: False)
early_stop_rounds : int (default 3)
this could be simplified to
early_stop_rounds : int (default 0)
The second concern I have is that if a user selects e.g., k_features=(1, 3), early_stop_rounds=3
, it's not necessarily guaranteed that there will be 1 to 3 features, which can be confusing.
I wonder if it makes sense to allow early_stopping_rounds
only for k_features='best'
and k_features='parsimonious'
, which both explore the whole feature subset size space.
E.g.,
if k_features='best'
is selected with early_stop_rounds=0
, it will evaluate and select the best feature subset in the range 1 to m.
if k_features='best'
is selected with early_stop_rounds=2
, it will evaluate and select the best feature subset in the range 1 to m but it may stop early, which means for forward selection, it may not explore the whole feature subset sizes up to m.
If a user selects e.g., k_features=3 and early_stop_rounds=2
we could
a) raise an error saying that f"Early stopping is not supported with fixed feature subset sizes."
b) raise a warning saying f"Early stopping may lead to feature subset sizes that are different from k_features={k_features}."
What are your thoughts?
Thanks for your suggestions, I agree with your points. I will edit this PR with the followings:
early_stop_rounds
: int >= 0early_stop_rounds
> 0 and k_features
in ['best', 'parsimonious']Sounds great! Looking forward to it!
Hello @aldder! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:
What is the status on this ?
I use the k_features="parsimonious" on my model. But it continues to add more and more features even after it is obvious the model will not improve, and after that it will select one of the very early model anywys.
I think this PR could get my runtime from 10 days into hours ;-)
Thanks for the ping, and I need to look into this some time -- sorry, haven't had a chance recently due to too many other commitments. Unfortunately, I currently don't have a timeline for this.
Description
According to some studies reported on papers (like this: https://www.researchgate.net/publication/220804144_Overfitting_in_Wrapper-Based_Feature_Subset_Selection_The_Harder_You_Try_the_Worse_it_Gets), the feature selection methodologies known as Wrapper suffer from overfitting as the number of explored states increases. A method to reduce this overfitting is to use automatic stop criteria (early stop, as the one most known for neural networks). In this PR I have implemented the criterion of early stopping for the class SequentialFeatureSelector.
One parameter has been added in during instantiation of the object:
Code Example:
Pull Request Checklist
./docs/sources/CHANGELOG.md
file (if applicable)./mlxtend/*/tests
directories (if applicable)mlxtend/docs/sources/
(if applicable)PYTHONPATH='.' pytest ./mlxtend -sv
and make sure that all unit tests pass (for small modifications, it might be sufficient to only run the specific test file, e.g.,PYTHONPATH='.' pytest ./mlxtend/classifier/tests/test_stacking_cv_classifier.py -sv
)flake8 ./mlxtend