Comparing RF feature importance to PPS

[MattConflitti]

Has anyone compared RF feature importance to PPS?

My one concern with PPS is that it is a univariate calculation. Even in the matrix, the numbers correspond to a single feature's predictive power on the target.

My inkling is that feature importance can cut across multiple variables when determining which variables provide the most power in that sense.

Just curious. Thanks!

[FlorianWetschoreck]

Hi Matt,

I am also looking forward to some formal comparisons between feature importance and the PPS. If you created some for your personal studies, it would be much appreciated if you share them.

Some general thoughts:

• you are right that PPS only corresponds to a single feature's predictive power on the target and thus, it should be accompanied by looking at the feature importances.
• Feature Importance is sometimes misleading depending on the calculation method. Therefore, I would suggest to always use Permutation Feature Importance https://scikit-learn.org/stable/modules/permutation_importance.html
• It might be interesting to subtract the PPS from the Permutation Feature Importance in order to estimate the amount of predictive power that resulted from interactions with other features

Best, Florian

[MattConflitti]

I am actually exploring a few options to compare: distance correlation, feature importance (taking into account the shortcomings you mentioned), PPS, and Pearson's correlation. I want to see how they each stack up.

I'll followup when I am done. Thanks!

On Mon, Jun 8, 2020, 11:14 AM Florian Wetschoreck notifications@github.com wrote:

Hi Matt,

I am also looking forward to some formal comparisons between feature importance and the PPS. If you created some for your personal studies, it would be much appreciated if you share them.

Some general thoughts:

• you are right that PPS only corresponds to a single feature's predictive power on the target and thus, it should be accompanied by looking at the feature importances.
• Feature Importance is sometimes misleading depending on the calculation method. Therefore, I would suggest to always use Permutation Feature Importance https://scikit-learn.org/stable/modules/permutation_importance.html
• It might be interesting to subtract the PPS from the Permutation Feature Importance in order to estimate the amount of predictive power that resulted from interactions with other features

Best, Florian

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/8080labs/ppscore/issues/22#issuecomment-640693514, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADBVIG2HYTX6RHS2NUHZUHDRVT55NANCNFSM4NTDSDQQ .

[FlorianWetschoreck]

Great, happy to hear that!

You might also want to add the Maximal Information Coefficient into the mix: https://medium.com/@rhondenewint93/on-maximal-information-coefficient-a-modern-approach-for-finding-associations-in-large-data-sets-ba8c36ebb96b

[FlorianWetschoreck]

Any updates from your side or did your priorities change?

[UmarIgan]

@FlorianWetschoreck I just demonstrated for selecting features for a model with xgboost vs ppscore. you can check it here.