yangnianzu0515
commented
7 months ago Thank you very much for your interest in our work!
There may be some differences between our official implementation and the one in PyHealth. Recently, we have also noticed some discrepancies in the performance between the two versions of the code. However, due to our busy schedule, we haven't had the opportunity to address this issue yet. We suspect that the problem may be related to the multi-label hinge loss, as PyHealth's methods, apart from our own, do not implement multi-label hinge loss. Nevertheless, this loss is used in SafeDrug and GAMENet as well. We have implemented this loss ourselves in PyHealth, but there may be some issues with its implementation, as we found that setting the weight of this loss to 0 (the 'multiloss_weight' parameter in the 'MoleRecLayer' class) resulted in some performance improvement. However, we are not certain if the issue lies here or if there are other problems. We plan to address these potential bugs when we have the time.
Another point to mention is that the data preprocessing in PyHealth differs from both SafeDrug and MoleRec, which makes it challenging to align the results in PyHealth with those in the respective papers. Additionally, the default parameters for MoleRec in PyHealth may not be the optimal ones, so you may need to perform some hyperparameter tuning. We submitted a pull request after running a basic test without waiting for the optimal parameters to be selected.
Our MoleRec implementation in our own repository should be free from these issues. In fact, we found that our MoleRec implementation still performs the best among baselines in a recently accepted paper at NeurIPS 2023. We believe the authors of that paper used our repository's code.
If you identify any issues with PyHealth, please feel free to let us know.
We hope this response addresses your concerns. If you have any further questions, please don't hesitate to reach out at any time.
He-Yichen
zhandand
Hello. I noticed that when running the MoleRec algorithms from PyHealth, their performance falls far short of what is claimed in your papers and benchmarks.(jaccard: 0.44382; F1: 0.6025; PRAUC: 0.7243; ddi: 0.0662) In Pyhealth, MoleRec performs even lower than the SafeDrug. Why is there such a big gap? Is there a difference between the MoleRec in Pyhealth and this branch? Thank you for your response.