Support for multiple references

[elliottd]

README.md suggests that MT-ComparEval currently supports only one reference translation.

reference.txt - a plain text file with reference translations (in target language).

It would be really useful if MT-ComparEval could support multiple reference translations!

[martinpopel]

I agree it would be useful. (@chrhad also asked about multi-reference BLEU.) So we will give it a higher priority, but @choko cannot start sooner than July 22 (SLT deadline), so any help (pull requests) is welcome. Here is a task list to coordinate the effort:

• [ ] uploading (and storing) more references
• [ ] computing 4-reference BLEU: a) as an external script in a similar way as Hjerson is implemented - this may be easier to to start with b) natively in PHP (taking advantage of precomputed matching ngrams) - this is the preferred way
• [ ] the default way of adapting other metrics for more references is for each sentence to choose the reference which gives the best sentence score.
• [ ] Visualization: Showing more references should be simple. But MT-ComparEval also highlights the matching/improving n-grams and underlines diffs. This would be more difficult to adapt (I am not sure what the diff with "the reference" should show - but this is the least important feature, we can show diff with the first reference).

[Gldkslfmsd]

I'd also appreciate multiple sources. This sounds weird generally, but in my research it makes sense. I compare impact of different preprocessing strategies.

[martinpopel]

It's not weird, we all agree multiple references would be useful. Just there is no one to work on it. So as noted above, PRs are welcome.