search

yangnianzu0515 / MoleOOD

Official implementation for the paper "Learning Substructure Invariance for Out-of-Distribution Molecular Representations" (NeurIPS 2022).
MIT License
54 stars 6 forks source link

High variance of the result on DrugOOD. #3

Closed panmianzhi closed 8 months ago

panmianzhi commented 8 months ago

I run your code on DrugOOD for 5 times, but the result has very high variance. For example, the AUC-ROC on data_assay_ec50 are 71.41, 72.39, 61.93, 64.70, 67.24. I can't understand why this happen. Maybe I make mistakes about the hyper-parameters, and they are as following:

屏幕截图 2024-01-27 205012

I saw the similar question in the close issue. Can you tell me the detailed value of the hyper-parameters on each dataset (except the random seed)?

yangnianzu0515 commented 8 months ago

Thank you very much for your interest in our work.

We think the issue might be related to your selection of hyperparameters. We notice in a recent paper accepted by NeurIPS23, titled "Learning Invariant Molecular Representation in Latent Discrete Space", the variance on the EC50 dataset was not as significant as what you have experienced. image However, it's been about a year and a half since that work was published, and we cannot recall the specific parameters. We suspect that part of the reason for the discrepancy might be due to your setting of the 'environment number' hyperparameter to 20, which seems somewhat high.

We hope our response is of some assistance to you.

panmianzhi commented 8 months ago

Thank you very much! PS: actually I also run the code of Learning Invariant Molecular Representation in Latent Discrete Space on DrugOOD, but the result is also not as good as their paper😂😂 Anyway, thank you for your reply!

mxqmxqmxq commented 2 months ago

@panmianzhi May I ask if you have read the article published in KDD 2024? It is of the same type as this one！