search

clear-nus / edc

MIT License
69 stars 9 forks source link

Question about WebNLG (v3.0) Test Data Size Discrepancy #15

Open Qing1Zhong opened 1 month ago

Qing1Zhong commented 1 month ago

Thank you for sharing your work. I have a question regarding the use of the WebNLG (v3.0) dataset in your work. In your paper, it is mentioned that the test data contains 1,165 text-triplet pairs. However, when I checked the official WebNLG repository in the file semantic-parsing-test-data-without-refs-en.txt, I found 2,155 text entries.

Could you please clarify the reason for this difference? Did you apply any specific filtering or processing to the dataset that resulted in the smaller test set? Thank you for your time and help.

bzhangj13zzz commented 4 weeks ago

Thank you for raising the issue and sorry for the late reply.

The 2155-entry file as you mentioned, if I remember correctly, is the split without references. So I was not able to evaluate with that set. The one with references should have a size of 1165. I can double check it later.

Qing1Zhong commented 3 weeks ago

Thank you for your reply @bzhangj13zzz , but I still don't understand your explanation. I checked both files (semantic-parsing-test-data-with-refs-en.xml and semantic-parsing-test-data-without-refs-en.xml), and I found 2155 entries in both files. Could you please confirm if there might be other factors contributing to this discrepancy? Or could you verify whether there are different versions of the test set? Thank you for your assistance!