Missed param/filtered directory

[ZhaoningYu1996]

Hi,

It looks like that you missed the param/filtered folder. Could you upload those indices pt files?

Thank you, Zhaoning

[Wuyxin]

Hi Zhaoning,

As indicated in README: the filtered indices are generated from train/refine_train.py

Since the indices would be different if you use different GNNs and data splits, I think it would be better to re-generate them.

[Wuyxin]

I will create another file for the dataset filtering, though. I guess you encountered this problem when using the local explainers (not involves global training) first. Note if you just want to do the explaining, you may not filter the dataset.

[ZhaoningYu1996]

Thank you for your reply. I regenerate the indices using refine_train.py. But I can only get ACC-AUC accuracy around 0.781 with the setting in run.sh ( python refine_train.py --dataset mutag --hid 100 --epoch 100 --ratio 0.4 --lr 1e-3 ) Could you tell me the setting for reproducing the accuracy in your paper?

Thank you, Zhaoning

[Wuyxin]

By "ACC-AUC accuracy", did you mean the output of refine_train.py or evaluate.py?

It would be better for you to specify what GNN did you use (the one provided or retrained). Also, did you try different random seeds?

[ZhaoningYu1996]

I retrained gnns by running gnns/run.sh. I mean the output of both refine_train.py and example.py. I have read the code. The ACC-AUC is the mean of all accuracy with different select ratios, right? I did not try different random seeds, because the result I get is way less than the accuracy reported in the paper. ( 0.781 and 0.955). I think it may be caused by the wrong setting I am using. The steps I am doing:

1. python mutag_gnn.py
2. python refine_train.py --dataset mutag --hid 100 --epoch 100 --ratio 0.4 --lr 1e-3
3. python example.py ( I changed codes in example.py for mutag dataset) Also, I am using all default settings in those files.

Thank you, Zhaoning

[Wuyxin]

example.py is for evaluating baseline methods, which requires to specify an explanatory method. You shall use evaluate.py instead to evaluate ReFine.

[ZhaoningYu1996]

I run evaluate.py and get acc-auc 0.7884 with R-1.00. I changed the default dataset from ba3 to mutag in the evaluate.py. Do I need to change other parameters?

[ZhaoningYu1996]

Also, what is the select ratio for refine_ft? The default is 0.4, do I need to change it?

[Wuyxin]

pls try the default GNN first to see if this decline is caused by the GNN.

[ZhaoningYu1996]

I have tried the default GNN and can get ACC-AUC around 0.946. And ReFine-FT and ReFine have almost no difference. Could you tell me how to reproduce the default GNN and how to achieve the 0.978 ReFine accuracy in your paper? Also, if I want to explain other GNNs, how can I change the settings?

[Wuyxin]

In the paper, we reported 0.945 and 0.955 for ReFine-FT and ReFine, not 0.978. Note this two results are close in this dataset.

[Wuyxin]

I have fixed the random seeds in the code. My settings follow exactly the same routines as yours, and the outputs are included in the log folder.

I guess the this problem might be caused by pytorch or the difference of the machines (I am not an expert of this). So my suggestion is that you could try different seeds in training the GNNs/ReFine in case you use some really bad seeds, and compare the relative improvements of ReFine over the baselines.

I will close this since what we discuss doesn't fit the title. You can open another if you need further discussion. Thanks.