search

ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.
8 stars 3 forks source link

Demo does not run out of the box #37

Closed obo closed 3 years ago

obo commented 3 years ago

Please keep regularly testing the 'demo', described in README.md. Always test is with a fresh checkout.

I tried:

(venv) 13:29 tap test-sltev$SLTev -g SLTev-sample --outdir my-evaluation-run-1
ELITR_CONFIDENTIAL_PASSWORD=... was not used
elitr-testset pulled
elitr-testset commit id is:  elitrtestset_d9475f2
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.OSt copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.OStt copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.cs.OSt copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.de.OSt copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.cs.align copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.de.align copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.en.asr copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.de.slt copied to my-evaluation-run-1
/tmp/test-sltev/elitr-testset/documents/SLTev-sample/kaccNlwi6lUCEM.en.cs.slt copied to my-evaluation-run-1

I guess README.md should clearly say that the demo already contains sample system outputs, namely kaccNlwi6lUCEM.en.en.asr, kaccNlwi6lUCEM.en.de.slt and kaccNlwi6lUCEM.en.cs.slt.

Or am I still expected to create some files myself?

Anyway, the evaluation crashed:

(venv) 13:30 tap test-sltev$SLTev -e my-evaluation-run-1
working directory is  my-evaluation-run-1
Evaluating the file  my-evaluation-run-1/kaccNlwi6lUCEM.en.en.asr  in terms of  WER score against  my-evaluation-run-1/kaccNlwi6lUCEM.en.OSt
Traceback (most recent call last):
  File "/tmp/test-sltev/venv/bin/SLTev", line 187, in <module>
    ASRev(ost=tt[0], asr=submission_file, SLTev_home=sltev_home, simple=args.simple)
  File "/tmp/test-sltev/venv/lib/python3.6/site-packages/SLTev/ASRev.py", line 244, in ASRev
    ostt = read_ostt(ostt_file)
  File "/tmp/test-sltev/venv/lib/python3.6/site-packages/SLTev/ASRev.py", line 77, in read_ostt
    if line[0] != 'P' and line[0] != 'C':
IndexError: list index out of range
mohammad2928 commented 3 years ago

Did you manipulate "my-evaluation-run-1/kaccNlwi6lUCEM.en.OSt"? Because of the demo run without any problems.
The problem is that the input file contains blank lines. I will update the script to ignore blank lines. Thanks,

obo commented 3 years ago

No, no manipulation whatsovever, just -g and then -e.

mohammad2928 commented 3 years ago

Please check the new version. (version 1.0.6)

obo commented 3 years ago

Tested today:

obo commented 3 years ago

Also elitr-testset run works.