search

PetarV- / GAT

Graph Attention Networks (https://arxiv.org/abs/1710.10903)
https://petar-v.com/GAT/
MIT License
3.18k stars 642 forks source link

Code error in attention part #23

Closed weiwongg closed 5 years ago

weiwongg commented 5 years ago

In my opinion, the coefficients part in your layer.py file should be coded in this way:

coefs = tf.exp(tf.nn.leaky_relu(logits)) coefs = coefs * tf.expand_dims(tf.sparse.to_dense(self.A),0) coefs = coefs / tf.reduce_sum(coefs, -1,keepdims=True) coefs = tf.nn.dropout(coefs, self.keep_prob)

Maybe your implementation is wrong, because exp(0) = 1.

PetarV- commented 5 years ago

Hello,

Thanks for your comment! However, I believe that the code should be fine as-is, for both the sparse and dense layer.

If your comment concerned the dense layer, note that bias_mat is added to the matrix, making the non-existing entries have a coefficient which is a large negative number, causing their normalised coefficient to tend to zero.

If it regards the sparse layer, check the documentation for tf.sparse_softmax at:

https://www.tensorflow.org/api_docs/python/tf/sparse/softmax

as can be seen, it ignores the masked entries when computing the softmax.

(Also, it should be noted that, in this case, tf.sparse.to_dense could create a potentially massive matrix and cause an OOM issue.)

weiwongg commented 5 years ago

Thanks for your explanation!

Best regard!

Wei

From: Petar Veličković notifications@github.com Sent: Wednesday, April 17, 2019 7:16:56 AM To: PetarV-/GAT Cc: Huang Wei; Author Subject: Re: [PetarV-/GAT] Code error in attention part (#23)

Hello,

Thanks for your comment! However, I believe that the code should be fine as-is, for both the sparse and dense layer.

If your comment concerned the dense layer, note that bias_mat is added to the matrix, making the non-existing entries have a coefficient which is a large negative number, causing their normalised coefficient to tend to zero.

If it regards the sparse layer, check the documentation for tf.sparse_softmax at:

https://www.tensorflow.org/api_docs/python/tf/sparse/softmax

as can be seen, it ignores the masked entries when computing the softmax.

(Also, it should be noted that, in this case, tf.sparse.to_dense could create a potentially massive matrix and cause an OOM issue.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/PetarV-/GAT/issues/23#issuecomment-484134764, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Agaz4SLYFi0yCypkhI5gwkIAVSLyyplkks5vhzrogaJpZM4c1Nk2.