Open yuhui-zh15 opened 3 years ago
Firstly, thanks to the authors for the easy-to-use codebase, it's deeply appreciated!
I can confirm that I have the same issue as @yuhui-zh15 for BBBP, in my case I get
3-fold cross validation
Seed 0 ==> test auc = 0.901969
Seed 1 ==> test auc = 0.903515
Seed 2 ==> test auc = 0.876906
overall_scaffold_balanced_test_auc=0.894130
std=0.012196
I have done multiple runs and played around with a few things including activating and deactivating args.dense
as well as changing the split type to random
just in case, but I can't get an AUC close to the one stated in the paper. Some lucky random folds get to 0.95 AUC but this disappears in the averaging.
I will experiment with the large model and some of the other endpoints to see if I have any luck reproducing any of the results.
p.s. the downloadable fine-tuned models seem to be much larger than the base model and of varying sizes, so perhaps different hyperparameters were used for each endpoint and even ensembles in some cases? Unfortunately I am having difficulties downloading them to verify.
I tried to finetune the large model, but it seems it is even worse than the base model.
Fold 0
Model 0 best val loss = 0.486441 on epoch 7
Model 0 test auc = 0.893492
Ensemble test auc = 0.893492
Fold 1
Model 0 best val loss = 0.479239 on epoch 8
Model 0 test auc = 0.888364
Ensemble test auc = 0.888364
Fold 2
Model 0 best val loss = 0.490516 on epoch 0
Model 0 test auc = 0.892271
Ensemble test auc = 0.892271
3-fold cross validation
Seed 0 ==> test auc = 0.893492
Seed 1 ==> test auc = 0.888364
Seed 2 ==> test auc = 0.892271
overall_scaffold_balanced_test_auc=0.891375
std=0.002187
Thanks to the author for providing the source code. Unfortunately, I get the same results as @yuhui-zh15 and @TWRogers on BBBP, and the results are as follows
Model 0 test auc = 0.895133
Ensemble test auc = 0.895133
1-fold cross validation
Seed 0 ==> test auc = 0.895133
overall_scaffold_balanced_test_auc=0.895133
std=0.000000
The test auc (0.895133) is lower than the value reported in the paper (0.936). Are there any special tricks that need to be considered?
Hello, I also encountered the same problem as you. Have you solved it? @yuhui-zh15 @TWRogers @WenjinW
Hi, thanks for your great work and clear documentation! I'm trying to reproduce your result on BBBP. However, while I followed the exact setting in the README, there seems to be a huge gap between my result (89.4) and the reported number (93.6). I listed all the steps that are fully reproducible. Could you check if there is anything wrong with my side? Thanks a lot for your help in advance!
The training log (
quiet.log
) is: