search

uma-pi1 / kgt5-context

11 stars 3 forks source link

Reproducing results of KGT5 with this repository code #4

Closed phucdoitoan closed 2 months ago

phucdoitoan commented 3 months ago

Hi, I tried to reproduce the result of KGT5 with the command: python main.py dataset.name=wikidata5m_v3 dataset.v1=True run on 2 GPU, with original config.yaml file. But the result was MRR ~ 0.254, which is far lower than that in the KGT5 paper ( MRR ~0.300)

Can you tell me how to reproduce the KGT5's resutls?

Thank you a lot!

AdrianKs commented 3 months ago

Hi,

I am sorry for the late reply. As this is not the repository from the original KGT5 paper, results are expected to be slightly different. However, results should definitely be better than the 0.254 you report. Typically, with a larger effective batch size KGT5 performs considerably better. I would propose to use a batch size of at least 128 or 256 per GPU with more GPUs.

python main.py dataset.name=wikidata5m_v3 dataset.v1=True train.batch_size=128

In case you still have reproduction problems, you can use the pretrained checkpoint, which was trained with the original repo.

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/apoorvumang/t5-small-wd5mv3-adafactor_82ep

python eval_legacy_huggingface.py -m t5-small-wd5mv3-adafactor_82ep --v1 --is_legacy

In case you are comparing the performance to other models, I would propose to additionally use the descriptions, as they are also used with SimKGC.

python main.py dataset.name=wikidata5m_v3 dataset.v1=True train.batch_size=128 descriptions.use=True
phucdoitoan commented 2 months ago

Hi Adrian,

Thank you a lot for your detailed reply. It helps a lot.