Closed weiwongg closed 5 years ago
Hello,
Thanks for your comment! However, I believe that the code should be fine as-is, for both the sparse and dense layer.
If your comment concerned the dense layer, note that bias_mat
is added to the matrix, making the non-existing entries have a coefficient which is a large negative number, causing their normalised coefficient to tend to zero.
If it regards the sparse layer, check the documentation for tf.sparse_softmax
at:
https://www.tensorflow.org/api_docs/python/tf/sparse/softmax
as can be seen, it ignores the masked entries when computing the softmax.
(Also, it should be noted that, in this case, tf.sparse.to_dense
could create a potentially massive matrix and cause an OOM issue.)
Thanks for your explanation!
Best regard!
Wei
From: Petar Veličković notifications@github.com Sent: Wednesday, April 17, 2019 7:16:56 AM To: PetarV-/GAT Cc: Huang Wei; Author Subject: Re: [PetarV-/GAT] Code error in attention part (#23)
Hello,
Thanks for your comment! However, I believe that the code should be fine as-is, for both the sparse and dense layer.
If your comment concerned the dense layer, note that bias_mat is added to the matrix, making the non-existing entries have a coefficient which is a large negative number, causing their normalised coefficient to tend to zero.
If it regards the sparse layer, check the documentation for tf.sparse_softmax at:
https://www.tensorflow.org/api_docs/python/tf/sparse/softmax
as can be seen, it ignores the masked entries when computing the softmax.
(Also, it should be noted that, in this case, tf.sparse.to_dense could create a potentially massive matrix and cause an OOM issue.)
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/PetarV-/GAT/issues/23#issuecomment-484134764, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Agaz4SLYFi0yCypkhI5gwkIAVSLyyplkks5vhzrogaJpZM4c1Nk2.
In my opinion, the coefficients part in your layer.py file should be coded in this way:
coefs = tf.exp(tf.nn.leaky_relu(logits)) coefs = coefs * tf.expand_dims(tf.sparse.to_dense(self.A),0) coefs = coefs / tf.reduce_sum(coefs, -1,keepdims=True) coefs = tf.nn.dropout(coefs, self.keep_prob)
Maybe your implementation is wrong, because exp(0) = 1.