search

DM2-ND / MoKGE

Author: Wenhao Yu (wyu1@nd.edu). ACL 2022. Commonsense Reasoning on Knowledge Graph for Text Generation
69 stars 10 forks source link

Low Bleu Score #8

Closed eujhwang closed 1 year ago

eujhwang commented 1 year ago

Hello, I ran your code with the same hyper-parameter you provided in the paper and code and I'm not able to get the same bleu score as you put in the paper. The model I ran is KGMixtureOfExpertCho.sh with eg dataset. The output score is "topk_bleu_4": 0.1459059145584853, "topk_rouge_l": 0.3882581646863371, which are much lower than the numbers in the paper. (The metrics (distinct_2, self_bleu,..etc) look similar to the one with your paper )

Could you guide me on what things I'm possibly missing?

Thank you,

wyu97 commented 1 year ago

Hi,

The data pre-processing code is added after paper submission because previously I directly used their code (https://github.com/cdjhz/multigen).

I did not realize a small error when I added the code to my repo. But now it has been fixed up.

Best regards, Wenhao