search

AIPHES / ACL20-Reference-Free-MT-Evaluation

Reference-free MT Evaluation Metrics
Apache License 2.0
20 stars 5 forks source link

Applying xmoverscore to novel dataset #5

Open billray0259 opened 2 years ago

billray0259 commented 2 years ago

Hello, I'm attempting to apply the xmoverscore metric to a novel dataset.

I ran main.py, and it generated the following files, which I organized into results directories.

image

Are the person correlation coefficients reported in these files? I seem to be unable to find them. Also, are the sample-level xmoverscore values reported in the HUMAN columns of DA-seglevel.csv file?

image

I calculated sample-level xmoverscores on the novel dataset I'm working with and the scores seem to be roughly in the range [-0.1, -0.2]

image

Does that seem like a valid range for the scores, or is it likely there is an error in the way I am calculating the scores? Thank you.

andyweizhao commented 2 years ago

Hello @billray0259

Are the Pearson correlation coefficients reported in these files?

No, these would be reported in console.

are the sample-level xmoverscore values reported in the HUMAN columns of DA-seglevel.csv file?

No, the HUMAN column shows human judgments of translation quality normalized by z-score, done by WMT workshops.

does that seem like a valid range for the scores?

The xmoverscore metric produces scores less than (or equal to) 1. Below are the details:

xmoverscore = 1 - EMD (earth mover distance). EMD produces positive scores or zeros. I have normalized the metric scores into the interval of [0, 1], with 1 as a perfect score. See https://github.com/AIPHES/ACL20-Reference-Free-MT-Evaluation/commit/2026e18c8874e5659ea6c03f82ec3e600d0feac1

Hope these can help!