search

ustunb / risk-slim

simple customizable risk scores in python
BSD 3-Clause "New" or "Revised" License
130 stars 33 forks source link

Nested 5-fold Cross-validation and performance metric? #18

Closed juanhu96 closed 2 years ago

juanhu96 commented 2 years ago

Hi Berk,

I am trying to run the RiskSLIM model for my own dataset and am trying to do a nested k-fold cross-validation (same as what you've done in the paper). However, I wasn't able to find the code where nested cross-validation is being handled. The only thing I found related is the fold_csv_file in the utils.py, but to me it seems like it is just regular k-fold cross-validation instead of nested validation. I am also thinking of doing the cross-validation based on different performance metrics (accuracy, AUC, etc.) and am not sure where could I change those things.

Thanks!

ustunb commented 2 years ago

Hi.

The code here is not really meant to support nested-CV since there are too many variations and since it's duplicating functionality that's already implemented in other packages.

The code that's already here for CV is mainly meant to help you reproduce the results in the paper. In that case, I didn't have to do run nested-CV for risk-slim since there is no "parameter tuning" once the model size is fixed.

I recommend that you implement a script. You can use the functions in this repository to fit a risk slim model for a given X, y.

Hope this helps!

On Mon, Jun 20, 2022 at 1:54 PM Jingyuan Hu @.***> wrote:

Hi Berk,

I am trying to run the RiskSLIM model for my own dataset and am trying to do a nested k-fold cross-validation (same as what you've done in the paper). However, I wasn't able to find the code where nested cross-validation is being handled. The only thing I found related is the fold_csv_file in the utils.py, but to me it seems like it is just regular k-fold cross-validation instead of nested validation. I am also thinking of doing the cross-validation based on different performance metrics (accuracy, AUC, etc.) and am not sure where could I change those things.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/ustunb/risk-slim/issues/18, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM4UCJTVNUMFMHJJS4W4PDVQDLBJANCNFSM5ZKCGXUA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

juanhu96 commented 2 years ago

Thank you so much!