Reproduce for comparison

[flyingdoog]

Dear Authors,

Thanks for sharing the codes. I am trying to reproduce the results for comparison. However, some settings are not clear. For example, what is the architecture of GNN? The same for all synthetic datasets? Which nodes are explained in quantitative evaluation? only range(400,700,5), or all nodes inside motifs?

[pindapuj]

^upvote. -- I'm also not sure how this is working.

I played around with explain_nodes_gnn_stats and dataset=syn1 (the house motifs datasets). Manually tried explaining nodes that were not included in range(400,700,5), and received errors. IIUC, evaluation is only being done using a specific house node only in red. Haven't looked at other datasets yet. Bit curious why this specific house node was selected, and why other house-nodes were not also included in the evaluation.

Any clarifications would be appreciated as well!

[spindro]

Hi, Like you, I was trying to replicate the results in the paper. For syn1 and syn2 evaluating the set of nodes in range(400,700,5) almost matches the AUCs provided.

For syn4, looking into make_pred_real(self, adj, start) the range that matches the proposed mask nodes should be range(511, 871, 6). This refers to the node "starting" the cycle from the binary tree graph.

Hoewer I could not match the performances cited in the paper. I got only an AUC of 0.61 instead of the declared 0.948.

Any clarifications on the evaluation procedure would be helpful.

[louiswng]

Hi, Like you, I was trying to replicate the results in the paper. For syn1 and syn2 evaluating the set of nodes in range(400,700,5) almost matches the AUCs provided.

For syn4, looking into make_pred_real(self, adj, start) the range that matches the proposed mask nodes should be range(511, 871, 6). This refers to the node "starting" the cycle from the binary tree graph.

Hoewer I could not match the performances cited in the paper. I got only an AUC of 0.61 instead of the declared 0.948.

Any clarifications on the evaluation procedure would be helpful.

What about the syn5? The source code did not provide make_pred_real on syn5. And I find bug when I am trying to reproduce the results on syn4 like this:

Saved adjacency matrix to masked_adj_syn4_base_h20_o20_explainnode_idx_695graph_idx_-1.npy Traceback (most recent call last): File "/home/wrj/anaconda3/envs/ge/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/wrj/anaconda3/envs/ge/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/wrj/wtl/gnn-model-explainer-master/explainer_main.py", line 317, in <module> main() File "/home/wrj/wtl/gnn-model-explainer-master/explainer_main.py", line 312, in main range(400, 700, 5), prog_args File "/home/wrj/wtl/gnn-model-explainer-master/explainer/explain.py", line 323, in explain_nodes_gnn_stats pred, real = self.make_pred_real(masked_adjs[i], new_idx) File "/home/wrj/wtl/gnn-model-explainer-master/explainer/explain.py", line 585, in make_pred_real if real[start + 1][start + 2] > 0: IndexError: index 6 is out of bounds for axis 0 with size 6