search

PetarV- / GAT

Graph Attention Networks (https://arxiv.org/abs/1710.10903)
https://petar-v.com/GAT/
MIT License
3.18k stars 642 forks source link

Early Stopping Strategy for PPI dataset #26

Open aviczhl2 opened 5 years ago

aviczhl2 commented 5 years ago

I implemented GAT on PPI according to the paper. (256,256 dim/ 4,4,6 heads) I followed the early stopping strategy of Pubmed, and get 96.9\pm 0.4 f1-score. How can I get the reported 97.3 \pm 0.2? Thanks very much.

chaojieji commented 5 years ago

@aviczhl2 Hi, I have a doubt about these results. I used function, micro_f1, as evaluation metric to train model. At last, the result of micro f1 is only about 0.5. So how to get 96.9\pm or 97.3 \pm mentioned in paper ? Thanks.

aviczhl2 commented 5 years ago

@aviczhl2 Hi, I have a doubt about these results. I used function, micro_f1, as evaluation metric to train model. At last, the result of micro f1 is only about 0.5. So how to get 96.9\pm or 97.3 \pm mentioned in paper ? Thanks.

You shall change the loss function to masked_sigmoid instead of masked_softmax I guess. Since it is multi-label classification.

chaojieji commented 5 years ago

@aviczhl2 I've changed. From another hand, what does "96.9\pm 0.4 f1-score" mean? I ran about 50 epochs, and final f1 score is about 0.5. What is about "96.9\pm" ? Thanks.

aviczhl2 commented 5 years ago

@aviczhl2 I've changed. From another hand, what does "96.9\pm 0.4 f1-score" mean? I ran about 50 epochs, and final f1 score is about 0.5. What is about "96.9\pm" ? Thanks.

Another tricky thing here(I suppose you have followed everything in the paper about structure and loss/evaluate function) is that you shall set dropout=0.0 and residual=True.

aviczhl2 commented 5 years ago

@aviczhl2 I've changed. From another hand, what does "96.9\pm 0.4 f1-score" mean? I ran about 50 epochs, and final f1 score is about 0.5. What is about "96.9\pm" ? Thanks.

Another tricky thing here(I suppose you have followed everything else in the paper about structure and loss/evaluate function) is that you shall set dropout=0.0 and residual=True. (Since I don't have your code so I cannot tell exactly which mistake you're making, just listing some possibilities I've experienced...)

BTW , /pm is the latex representation of "plus and minus" (standard deviation)

chaojieji commented 5 years ago

@aviczhl2 Thanks for your strong reminders ! I checked these parameter again and again, but ignored the setting of dropout rate in code. When I correct it, it seems work well. :)

PetarV- commented 5 years ago

Hello,

Thank you both for the interest in GAT and for performing evaluations on PPI!

Could you try using the same early stopping strategy as for Cora? While our paper reads a little unclear on this, I think this is the one we used for PPI.

Otherwise, it could just be library versions you're using -- the result you're getting already overlaps with the reported result in terms of standard deviations.

Thanks, Petar

linzhongping commented 5 years ago

Hello, do you still remember the final value of BCEloss when training inductively the GAT on PPI? I performed my own model on PPI, getting a loss value of 0.55 as well as a f1 score of 0.5.

LeeKangjie commented 4 years ago

@aviczhl2 Hi! I am wondering where did you download the PPI dataset? could you send it to me? Thank you very much!

caizhenhui commented 4 years ago

I implemented GAT on PPI according to the paper. (256,256 dim/ 4,4,6 heads) I followed the early stopping strategy of Pubmed, and get 96.9\pm 0.4 f1-score. How can I get the reported 97.3 \pm 0.2? Thanks very much.

Hi！I also change the code according to the paper(dropout、residual...),but i only get 92 f1-score.Can you share your code?