search

jaredhuling / oem

Penalized least squares estimation using the Orthogonalizing EM (OEM) algorithm
http://jaredhuling.org/oem
27 stars 6 forks source link

AUPRC as a loss function #23

Open pedro-ribeirosantiago opened 3 years ago

pedro-ribeirosantiago commented 3 years ago

Hi all,

Thanks for the excellent package! Is there any way to use the Area Under the Precision-Recall Curve (AUPRC) as the measure to evaluate cross-validation? This would be useful in cases of an imbalanced dataset.

Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3), e0118432.

Cheers, Pedro

jaredhuling commented 3 years ago

I would be happy to look at a pull request if you want to code it up

pedro-ribeirosantiago commented 3 years ago

Hello Jared,

I believe I figured out the code to use the AUPRC as the measure to evaluate cross-validation. However, it will not be possible to account for observation weights (as in the case of the AUC using the function glmnet:::auc.mat) since there is no R function to compute the Weighted AUPRC. Should I go ahead and open a pull request?

Cheers, Pedro

glycerine commented 1 year ago

I would be interested in AUPRC too. @pedro-ribeirosantiago could you describe how far you got, or post a pull request here? Thanks!

glycerine commented 1 year ago

Ah! I see you did submit a PR.

https://github.com/jaredhuling/oem/pull/24

@jaredhuling Could you comment? Would this be okay to merge in? Thanks guys.

pedro-ribeirosantiago commented 1 year ago

Hey @glycerine and @jaredhuling,

I did the pull request last year. The build had a failure only in the 'R_VERSION=devel, R_ARCH=x64'. I'm experienced with R programming but, unfortunately, I was unable to identify the problem since the modifications were built correctly in the "stable" and "patched" versions. I'm happy to provide further support and go ahead. Please let me know how I can help!

Cheers, Pedro

glycerine commented 1 year ago

Thanks @pedro-ribeirosantiago ! I was able to merge and load the patch just fine on my machine. I'm using linux, amd64, and R 4.1.0.

I tried to look at why the resulting oem package would install fine on stable and patched but not on devel. However the artifact log gets deleted after a month, and so AppVeyor does not have it any more.

@pedro-ribeirosantiago Could you make a small tiny change (to a comment perhaps) in the pull request and push it that so that AppVeyor will re-run the check and we can inspect the C:/projects/oem/oem.Rcheck/00install.out file which should contain the error.

pedro-ribeirosantiago commented 1 year ago

Dear @glycerine and @jaredhuling,

I just made a new pull request and now it passed all tests, so it would be ready for merging. Please let me know if you need anything from my side and I'm happy to keep making changes if needed!

Cheers, Pedro