search

HKUDS / GraphGPT

[SIGIR'2024] "GraphGPT: Graph Instruction Tuning for Large Language Models"
https://arxiv.org/abs/2310.13023
Apache License 2.0
493 stars 36 forks source link

GNN Baseline Too Low #65

Closed W-rudder closed 3 months ago

W-rudder commented 3 months ago

Hello! I think this is a very interesting work! However, is the GNN result given in the paper too low? Following the proposal in Section 4.2 of the paper, image I trained a GNN and a single-layer linear classifier using the arXiv training dataset and. When testing on the PubMed/Cora dataset, I trained a new classifier using the training set of the target dataset, while keeping the parameters of the GNN fixed. In the end, I obtained the following results, which differ significantly from those presented in the paper. Could it be due to differences in my experimental setup compared to that of the paper? What should be the correct experimental setup? The paper does not seem to provide detailed descriptions. Looking forward to your reply! Wishing you smooth work and a pleasant life! image

tjb-tech commented 3 months ago

Hello! I think this is a very interesting work! However, is the GNN result given in the paper too low? Following the proposal in Section 4.2 of the paper, image I trained a GNN and a single-layer linear classifier using the arXiv training dataset and. When testing on the PubMed/Cora dataset, I trained a new classifier using the training set of the target dataset, while keeping the parameters of the GNN fixed. In the end, I obtained the following results, which differ significantly from those presented in the paper. Could it be due to differences in my experimental setup compared to that of the paper? What should be the correct experimental setup? The paper does not seem to provide detailed descriptions. Looking forward to your reply! Wishing you smooth work and a pleasant life! image

Thanks for your interests! Actually, the zero-shot settings in our experiments are not required to be trained. Taking arxiv to pubmed setting as an example, we use the GNN backbone trained on arxiv, and the classifier head in the the same GNN backbone+head model trained on pubmed to conduct training-free zero-shot setting. We think that your operation training classifier head in new dataset is more like domain adaption rather than zero-shot learning.

W-rudder commented 3 months ago

Hello! I think this is a very interesting work! However, is the GNN result given in the paper too low? Following the proposal in Section 4.2 of the paper, image I trained a GNN and a single-layer linear classifier using the arXiv training dataset and. When testing on the PubMed/Cora dataset, I trained a new classifier using the training set of the target dataset, while keeping the parameters of the GNN fixed. In the end, I obtained the following results, which differ significantly from those presented in the paper. Could it be due to differences in my experimental setup compared to that of the paper? What should be the correct experimental setup? The paper does not seem to provide detailed descriptions. Looking forward to your reply! Wishing you smooth work and a pleasant life! image

Thanks for your interests! Actually, the zero-shot settings in our experiments are not required to be trained. Taking arxiv to pubmed setting as an example, we use the GNN backbone trained on arxiv, and the classifier head in the the same GNN backbone+head model trained on pubmed to conduct training-free zero-shot setting. We think that your operation training classifier head in new dataset is more like domain adaption rather than zero-shot learning.

Thank you for your prompt reply! But I'm sorry I may be a little uncertain about what you mean. Are you saying that we use the classifier from GNN+classifier trained on PubMed and the GNN from GNN+classifier trained on arXiv for zero-shot learning, right?

tjb-tech commented 3 months ago

Hello! I think this is a very interesting work! However, is the GNN result given in the paper too low? Following the proposal in Section 4.2 of the paper, image I trained a GNN and a single-layer linear classifier using the arXiv training dataset and. When testing on the PubMed/Cora dataset, I trained a new classifier using the training set of the target dataset, while keeping the parameters of the GNN fixed. In the end, I obtained the following results, which differ significantly from those presented in the paper. Could it be due to differences in my experimental setup compared to that of the paper? What should be the correct experimental setup? The paper does not seem to provide detailed descriptions. Looking forward to your reply! Wishing you smooth work and a pleasant life! image

Thanks for your interests! Actually, the zero-shot settings in our experiments are not required to be trained. Taking arxiv to pubmed setting as an example, we use the GNN backbone trained on arxiv, and the classifier head in the the same GNN backbone+head model trained on pubmed to conduct training-free zero-shot setting. We think that your operation training classifier head in new dataset is more like domain adaption rather than zero-shot learning.

Thank you for your prompt reply! But I'm sorry I may be a little uncertain about what you mean. Are you saying that we use the classifier from GNN+classifier trained on PubMed and the GNN from GNN+classifier trained on arXiv for zero-shot learning, right?

Yes, you are right.