Hyperparameter search?

[jackd]

I'm trying to reproduce the results reported in the paper and I'm struggling with the hyperparameter search. The paper states the search was conducted over the weight_decay parameter with fixed lr=0.2, but the examples in run_citations.sh all use different lr values. It's also unclear how the T parameter was chosen if not the result of a hyperparameter search.

I've conducted my own hyperopt optimizations on cora over both weight_decay and weight_decay + T (other parameters from run_citations.sh), and the only way I can find hyperparameters similar to yours is if I minimize -test_acc. The searches I run minimizing -val_acc all yield hyperparameters that generate high val_acc but lower test_acc than the hyperparameters in run_citations.sh.

Could you provide the code (or a more complete description of the procedure) for the hyperparameter search? My apologies in advance if this already exists somewhere.

[yifeiwang77]

Thanks for your interest in our work! As for your first question, the fixed lr=0.2 is a typo that we forgot to update in the new version. We will fix it in the next update of the paper.

Meanwhile, we note that DGC can also attain comparable performance with a fixed lr=0.2 and even a fixed K=500, with the following configuration. Therefore, DGC only requires to tune one more hyper-parameter T.

Cora	Citeseer	Pubmed
82.8 ± 0.0	73.4 ± 0.0	80.1 ± 0.0

python main.py --dataset cora --T 4.8 --weight_decay 1.6e-5 --lr 0.2 --K 500
python main.py --dataset citeseer --T 3.9 --weight_decay 2e-4 --lr 0.2 --K 500
python main.py --dataset pubmed --T 5.35 --weight_decay 2.7e-5 --lr 0.2 --K 500

As for hyperparameter tuning, few state-of-the-art GCNs utilize the validation set on the citation datasets, as can be noticed from the "Validation" column in the PaperWithCode benchmark. For comparison, we also follow this setting.

[jackd]

@yifeiwang77 thanks for your response. That clears things up mostly, though could you please elaborate on your statement

few state-of-the-art GCNs utilize the validation set on the citation datasets, as can be noticed from the "Validation" column in the PaperWithCode benchmark. For comparison, we also follow this setting.

Specifically, what quantity do you optimize during your hyperparameter search?