List of appropriate metrics to test

[jack89roberts]

Starting pitch:

• BLEU (baseline)
• ROUGE-S / any other skip-gram or conventional metric of interest that may improve on BLEU (e.g. METEOR, CHRF)
• BLASER 2.0
• CometKiwi (reference-based variant) / other model-based/text translation-oriented metric of choice (e.g. critical error detection?)

and

• the above with different variants of pre-processing etc.

[klh5]

It doesn't include BLASER as it's from 2023 but this paper also has a nice taxonomy of metrics which we could re-use.

[jack89roberts]

Also sentence similarity metrics e.g. these models: https://huggingface.co/models?pipeline_tag=sentence-similarity&sort=trending

SPICE is using one of these as part of: https://arxiv.org/pdf/2405.13845