Usage example?

[JohnGiorgi]

Hi!

Is there an example of how to use this metric for evaluation? I would be interested in instantiating some objects and computing the DisoScore score between a reference and generated summary. Ideally a usage something like BERTScore or BARTScore

[andyweizhao]

@JohnGiorgi thanks for your interest! I will add a running example in 1-2 days :))

[JohnGiorgi]

Awesome! Thanks a lot

[andyweizhao]

@JohnGiorgi,

I have added an example in a way as in BARTScore. The code supports 6 discourse metrics, including DiscoScore. The details of these metrics are provided in Appendix A.1 in the paper.

Note that if system and reference texts do not contain coherence phenomena (e.g., no word repetition), then the discourse metrics would return 0.

[JohnGiorgi]

Awesome! Thank you @andyweizhao. A couple of questions if that's okay!

• Is model_name="bert-base-uncased" the currently recommended pre-trained model to use to get best results?
• Does DiscoScore work in both directions like BARTScore? E.g. disco_scorer.DS_SENT_NN(s, refs) is different (but valid) from disco_scorer.DS_SENT_NN(refs, s)
• Have you tried it out in the reference-free setting? E.g. disco_scorer.DS_SENT_NN(s, source_doc). It looks like the source document text is retrieved in your code but never used.

[andyweizhao]

@JohnGiorgi

1. The recommended models are Conpono (the discourse variant of BERT) for DS-Focus and BERT-NLI for DS_SENT. I have slightly adjusted the text in README.md.
2. DiscoScore uses F1, and thus takes into account both directions. But the input arguments of m(x, y) cannot be swapped, b/c the current code is written in a way that assumes the 2nd argument is a set of multi-references.
3. If I remember correctly, the results of DiscoScore in reference-free settings are not strong.

[andyweizhao]

About F1, DiscoScore uses Precision by default, but one can enable Recall and F1 by slightly adjusted the code.