search

PetarV- / GAT

Graph Attention Networks (https://arxiv.org/abs/1710.10903)
https://petar-v.com/GAT/
MIT License
3.15k stars 643 forks source link

The attention value of all edges of a node is almost equal #64

Open Yaoyuanzhou opened 3 years ago

Yaoyuanzhou commented 3 years ago

hello.

I pulled out one of the attention value matrices and observed, but I found that all the edges of a node are almost the same. And the value is 1/n, n is the number of edges on that node. This indicates that GAT has not learned the importance of the corresponding edge, which is almost similar to GCN. The figure below shows the corresponding attention values for adjacent edges of some nodes in the Cora dataset (the same is true for the Citeseer dataset).

Why does this happen

image

PetarV- commented 3 years ago

Hi, and thank you for your issue!

The effect here usually occurs on homophilous datasets (of which Cora and Citeseer are definite cases). By definition, in such datasets, the edges often merely indicate that classes should be shared, so most of the performance can be recovered by something that resembles simple averaging. It is therefore in GATs' interest to learn a distribution that is close to uniform.

This effect does not happen on PPI. Check out this blog-post from the Deep Graph Library team, which explores this effect detailedly:

https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html

Hope that helps!

Thanks, Petar

Yaoyuanzhou commented 3 years ago

Thank you for your reply, which was very helpful to me