search

nouhadziri / DialogEntailment

The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"
https://arxiv.org/abs/1904.03371
MIT License
74 stars 5 forks source link

How to test with other metrics ? #4

Closed wizare closed 2 years ago

wizare commented 3 years ago

Hi. I have already trained my BERT model and tested my responses with the entailment model.

But I didn't know how to test with other metrics, i.e., Semantic Similarity, Word-level metrics, Consistency by textual entailment.

Can you show a script as an example? Thanks

ehsk commented 3 years ago

Sorry for the late reply! This somehow slipped my mind.

If it still helps, you can find semantic similarity and consistency metrics in our repo: here and here. For both metrics, we implemented two methods: compute_metric (for one sample) and compute_metric_for_file (for an entire file).

For word-overlap metrics, you can use this repo.

Hope this helps!