search

zjupgx / modig

MODIG: Integrating Multi-Omics and Multi-Dimensional Gene Network for Cancer Driver Gene Identification based on Graph Attention Network Model
9 stars 2 forks source link

Reproduced performance metric not the same as the paper #4

Open Mugariya opened 1 year ago

Mugariya commented 1 year ago

Hello,

I tried to reproduce the results as mentioned in the paper by following the same set of steps but I'm getting a lower score. Is there something wrong with the repo?

zjupgx commented 1 year ago

We found a small error in the utils.py file at line 106 and have made the change tpr, fpr, _ = roc_curve(y_t, pred) to fpr, tpr, _ = roc_curve(y_t, pred). Not sure if your question is related to this. If not, please describe your problem in detail.

Mugariya commented 1 year ago

Thank you for your response. When I run the experiments, the mean AUC for (ppi as string) that I get is 0.879 and according to the paper it should be 0.90

zjupgx commented 1 year ago

After checking the code, I think the problem lies in the two hyperparameters thr_seq and thr_path. In the paper we used 0.5, in github it is 0.6, which has been modified. Thanks!

Mugariya commented 1 year ago

Thank you . I'll try to use that and check again.

There's one more thing in modig_graph.py on line 136 os.path.join('./Data/simmatrix/seqsim_matrix.csv'). in your data folder it is not named the same. I used the csv given in your data folder named seqsim_rbbs_matrix.csv. Is that okay ?

Mugariya commented 1 year ago

Hello, I tried with these hyperparameters but the result produced is not the same as in the paper

Mugariya commented 1 year ago

Hello,

Is it possible to see what is wrong? The reproduced metric are not the same as paper and MTGCN performs better.

Thanks