search

WebNLG / GenerationEval

WebNLG+ Challenge 2020: Scripts to evaluate the RDF-to-text task with automatic metrics (BLEU, METEOR, chrF++, TER and BERT-Score)
MIT License
16 stars 5 forks source link

Multiple references #7

Open senjed opened 2 years ago

senjed commented 2 years ago

Hi, from the ReadMe file it is not clear what file format the script needs for multiple references .... in case some examples have k references but some have less than k, how should the files be created?

Shimorina commented 2 years ago

If there are k references, then you need to create k files named reference0, reference1, ..., reference{k}

Please have a look here: https://github.com/WebNLG/GenerationEval/tree/master/data/en/references

Example:

There are two instances a-text and b-text with 2 and 3 references respectively.

a-text: {a-ref1, a-ref2}
b-text: {b-ref1, b-ref2, b-ref3}

Then you need to create 3 files: reference0, reference1, reference2 with the following content.

reference0:

a-ref1
b-ref1

reference1:

a-ref2
b-ref2

reference2 (the first line is an empty line):


b-ref3

You can generate references for WebNLG using generate_references.py from this repo: https://gitlab.com/webnlg/corpus-reader