search

ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.
8 stars 3 forks source link

Need a usage example of SLTev and elitr-testset working together #27

Closed bhaddow closed 3 years ago

bhaddow commented 3 years ago

Hi

So I have installed SLTev and checked that the demo runs. Now (say) I would like to evaluate my own system on one of the elitr test sets. It's not that clear how to do this - navigating test set indexes. translating and collecting output files etc. I'm not sure where this belongs - is it in SLTev or elitr-testsets?

Barry

pyRis commented 3 years ago

Hi, Barry,

I am responsible for this task thus assigning it to myself. The evaluation part is done but not yet completely documented. I will finish it by theAItre event and will commit it in its respective directory on elitr-testset-evaluations.

mzilinec commented 3 years ago

Hi, I believe we need some actual documentation that explains all the hidden details. For example, what indexes are available, what prefixes you need to use in your translation outputs, how do you properly name the files etc.

I will just write how I tried to do MT evaluation today for reference.

# 1. Install SLTev
pip install -e .

# 2. Generate a directory with input and reference files
# The following will generate a directory with both public & private en-cs data (only if run on CUNI cluster?)
SLTev -g auto-mt-en2cs --outdir eval-run-4-feb-2021/

# 3. Run model for all input files and save with the suffix .slt, prefixing lines with "C 0 0 0 /"
tgt="cs"  # target language
for infile in $(find eval-run-4-feb-2021/ -name '*.OSt'); do
  prefix=${infile%.OSt}
  cat $infile | marian-decoder > $prefix.$tgt.slt
  sed -i 's/^/C 0 0 0 /' $prefix.$tgt.slt
done

# 4. Run evaluation
SLTev -e eval-run-4-feb-2021/
obo commented 3 years ago

Following the README.md today, I think that sample outputs exemplifying what your systems should produce are there. Reopen if more explanation is needed.