search

clemsgrs / hipt

Re-implementation of HIPT
18 stars 6 forks source link

Evaluation missing? #14

Open vildesboe opened 6 months ago

vildesboe commented 6 months ago

Hi! Thank you for sharing your re-implementation! I cannot seem to find any of your evaluation results, and I was wondering if you were able to achieve the same results (AUC) as the original HIPT paper on all datasets and experiments? Or which datasets/experiments you tested on, and which results you got? Thank you

clemsgrs commented 6 months ago

hi, indeed i haven't added evaluation results to the README yet. i only tried to reproduce the TCGA BRCA subtyping results, here's a comparison between the numbers reported in the paper and the numbers i got (using same 10 cross-fold validation setup):

model paper mine
ViT-16PF, ViT-256PF, ViT-4096 0.874 ± 0.060 0.887 ± 0.064
ViT-16PF, ViT-256P, ViT-4096 0.827 ± 0.069 0.894 ± 0.049
ViT-16PF, ViT-256, ViT-4096 0.823 ± 0.071 0.864 ± 0.056

the main difference (actually more a discrepancy) is that i get better results with (ViT-16PF, ViT-256P, ViT-4096) than with (ViT-16PF, ViT-256PF, ViT-4096), which actually makes sense given (ViT-16PF, ViT-256P, ViT-4096) has more trainable parameters, hence can better fit the data.

vildesboe commented 2 months ago

@clemsgrs Thank you for your response and for sharing your results! I see that you only have results for subtyping, but that you have some code for survival prediction. Have you tried to work with some of the survival predictions and have any insights there? I am currently working on the survival task, but struggling to get results significantly higher than a c-index of 0.5 even on "easier datasets". Best regards