Closed jermainewang closed 6 years ago
Hi Minjie,
Thank you for the kind words and your interest in GAT!
I believe what you are probably missing is adding the identity matrix to the adjacency. The raw sparse matrix, as extracted by Thomas Kipf's code, does not have ones on the diagonal (they only add it in post-processing).
Disclaimer: We have played around with the sparse layer only for the purposes of Pubmed, so cannot guarantee exact reproducibility compared to the dense layer. However, I just now executed the sparse model on Cora once and managed to get 83.7% on the test set.
P.S. If you manage to get a good-looking version of the sparse code running, could you consider submitting a pull request with a models/sp_gat.py
and sp_execute_cora.py
? All the versions I have implemented seem too hacky for inclusion in this repository. :)
Thanks, Petar
Hi Petar,
Thank you for your reply! I figured out what I missed. When I fed the sparse adj matrix to tensorflow, I incorrectly packed the row index and column index. Basically, I used following codes to convert the adj matrix to tf.SparseTensor
:
def process_adj_sp(adj):
num_nodes = adj.shape[0]
adj = adj + sp.eye(num_nodes) # self-loop
adj[adj > 0.0] = 1.0
if not sp.isspmatrix_coo(adj):
adj = adj.tocoo()
adj = adj.astype(np.float32)
indices = np.vstack((adj.col, adj.row)).transpose() # This is where I made a mistake, I used (adj.row, adj.col) instead
return tf.SparseTensor(indices=indices, values=adj.data, dense_shape=adj.shape)
It worked perfectly now. I think you implementation looks fine for me. I did make some changes, but the main sp_attn_head
function followed your implementation. I could try make a pull request later.
Best regards, Minjie
Hi, @jermainewang and @PetarV- , I just have a quick simple question. Why would you suggest a sparse adjacency bias matrix? To deal with what scenarios? I tried, it seems not to work for speedup.
Hi GAT folks,
Awesome works! I tried out your sparse implementation. What I did are: (1) Replace the
attn_head
withsp_attn_head
function. (2) Use sparse adj matrix. I removed theadj.todense()
andprocess.adj_to_bias
function calls and replace it with sparse version using scipy.I found I cannot achieve the expected accuracy after using the sparse implementation for Cora. Here is the log:
You could see the accuracy dropped suddenly after achieving ~60% for validation set. Have you guys met similar problems? Did I miss anything?
Thank you, Minjie