search

TeamHG-Memex / eli5

A library for debugging/inspecting machine learning classifiers and explaining their predictions
http://eli5.readthedocs.io
MIT License
2.75k stars 331 forks source link

Use out-of-bag score to get the permutation Importance #375

Open Yanjiayork opened 4 years ago

Yanjiayork commented 4 years ago

Hi, I have tried to use out-of-bag score to get the permutation importance, the code is shown as below:

from sklearn.ensemble import RandomForestClassifier import eli5 from eli5.sklearn import PermutationImportance model = RandomForestClassifier(n_estimators=100, n_jobs=-1, oob_score = True, bootstrap = True, random_state=42,) model.fit(x,y) perm = PermutationImportance(model, cv = None, refit = False, scoring = 'oob_score', n_iter = 50).fit(x, y)

It shows that ValueError: 'oob_score' is not a valid scoring value.

Is there a possible way to use out-of-bag score to calculate the feature importance?

Many thanks Yan

InterferencePattern commented 3 years ago

Out-of-bag scores are produced during the training process, while Permutation Importance relies on being provided a pretrained model.

PermutationImportance is asking for a method of the model object to use for scoring. In your case, the RandomForestClassifier object doesn't have an 'oob_score' method, and so I don't think this would be possible.