Reproduce the results in the paper

[lei-liu1]

Hello,

Thanks for sharing the codes. I try to reproduce the results reported in the paper using the provided parameters: python3 sslmain.py --pre_train_datasets arxiv sportsfit products --method graphmae --num_heads 4 --num_out_heads 1 --num_layers 3 --num_hidden 1024 --residual --in_drop 0.5 --attn_drop 0.5 --norm 'batchnorm' --lr 0.01 --weight_decay 1e-5 --activation 'prelu' --mask_rate 0.75 --drop_edge_rate 0 --replace_rate 0.2 --scheduler --lrtype 'cosine' --save_model --max_epoch 5 --subgraph_size 1024 --cpuinf

After running the code above, I also try to load the pretrained model to test it on the downstream datasets like this: python3 sslmain.py --pre_train_datasets arxiv sportsfit products --downstream_datasets cora bookhis amazonratings --method graphmae --num_heads 4 --num_out_heads 1 --num_layers 3 --num_hidden 1024 --residual --in_drop 0.5 --attn_drop 0.5 --norm 'batchnorm' --lr 0.01 --weight_decay 1e-5 --activation 'prelu' --mask_rate 0.75 --drop_edge_rate 0 --replace_rate 0.2 --scheduler --lrtype 'cosine' --max_epoch 10 --subgraph_size 1024 --cpuinf --load_model "graphmae-gat-gat-['arxiv', 'sportsfit', 'products'].pt" --eval_only --not_same_pretrain_downstream However, the results are much lower than that in the paper. I wonder if there is something I missed. Could you please provide some more details about reproducing the results? Thanks very much!

Best Regards, Lei

[CurryTang]

Thanks for reminding, there should be --warmup option in the command. May you have another try?

[lei-liu1]

Hello,

Thanks for your reply. It works. The results have been normal.

But there still seem to be some issues. After pretraining the model and trying to apply it to some different downstream datasets, I got IndexError: list index out of range.

What I did is, removing --save_model and adding

--downstream_datasets cora bookhis --load_model "graphmae-gat-gat-['arxiv', 'sportsfit', 'products'].pt" --eval_only --not_same_pretrain_downstream

I checked the codes and found the problem may come from lines 268-269 in sslmain.py:

for i in range(len(data)):
    logging.info("Epoch {} Dataset {} Val-{} {} Test-{} {}".format(e, args.downstream_datasets[i], DATASET[args.downstream_datasets[i]]['metric'], eval_val[i], DATASET[args.downstream_datasets[i]]['metric'], eval_test[I]))

which requires the pretraining and downstream datasets to be the same number. It may be the co-training scenario described in the paper.

So if I want to reproduce the results of Case 3 and Case 4 in the paper, how should I set the parameters, especially for zero-shot, few-shot, and fine-tune settings?

Many thanks!