The accuracy of prediction on MUTAG dataset

[ChisamXz]

Hi, I ran the code by its default setting which is training on MUTAG, the learning rate and other parameters are remained as the default value. However, I can only achieve 72.22% accuracy after 300 epochs of training, which is different from the reported 85.83% in the paper. Could you give any suggestions to fix this?

[ChisamXz]

When I adjust the learning rate to 0.001, the accuracy increases to 83.3%

[muhanzhang]

Hi, MUTAG is a rather small dataset that is prone to overfitting and sensitive to parameters/data splitting. I suggest you do 10-fold cross validation and calculate the average accuracy, or more stably average out 10 series of cross validation accuracies. Or perhaps, moving to another larger dataset. Thanks!

[ChisamXz]

@muhanzhang Thank you for your quick reply and suggestions. Your suggestion really works! I should've paid more attention to the experimental settings in the paper.

[ChisamXz]

Hi, after using the 10-fold cross validation, my accuracy for the datasets are respectively: D&D 77.86%; PROTEINS 73.06%; MUTAG 83.33%. Would you give me some suggestions of how to improve the results to the reported values in the paper. Thank you in advance~

[ChisamXz]

I wa using the default hype-parameters. I ran the code with "./run_DGCNN.sh Dataset 0" every time to get the results. The above accuracies were the best I could achieve after repeated execution.

[muhanzhang]

Hi, the original paper uses a batch size 1. By changing bsize=1, I ran on the default permutation indices provided, I got average accuracies: MUTAG: 0.8611 PROTEINS: 0.797673 DD: 0.773504 on my machine. The results of the paper were got from the original torch version by running 10 series of 10-fold cross validation exps (each using a different permutation order) and reporting the average of the 10 average accuracies.

[ChisamXz]

Yes, I tried it on weekends, now the accuracy seems corresponding now. Thank you so much for the answering, it helped me a lot!