ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.
8 stars 3 forks source link

ASRev should be *nothing more than a symlink* to SLTev. #10

Closed obo closed 3 years ago

obo commented 3 years ago

I was trying to explain a while ago that ASRev should be just a different behaviour of SLTev main. ASRev itself then should be just a symlink to SLTev.

To summarize:

This is the correct top-level strategy for SLTev:

#!/usr/bin/env python3
import sys
import os

def do_eval(whatever):
  print("Evaluating", whatever)
  return whatever

if __name__ == "__main__":
  if os.path.basename(sys.argv[0]) == "ASRev": mode="ASRev"
  else: mode="SLTev"
  print("Running in mode:", mode)
  do_eval("standalone run of "+mode)

Put that into SLTevTest and then try this:

chmod +x SLTevTest
./SLTevTest
ln -s SLTevTest ASRev
./ASRev

And directly in python:

python3
import SLTevTest
SLTevTest.do_eval("directly in python")
obo commented 3 years ago

(This is still not tackled.)

mohammad2928 commented 3 years ago

I converted SLTev evaluator and ASRev to the functions and I just use them in the main (SLTev/SLTev file).

obo commented 3 years ago

Do I get you correctly that there is no longer any plan to have a separate ASRev? I fully support that. Just close this issue to confirm.

obo commented 3 years ago

I still see ASRev.py in the repo. If it is outdated, delete it. Please provide an example of ASR-only evaluation, as a separate section to the README.

mohammad2928 commented 3 years ago

@obo ASRev.py just is a module (a set of functions) in the SLTev and it imported in the SLTev. if you think it is confusing we can rename it (e.g. asr-modules.py).

Also, for each submission file that has "asr" suffix, the WER score calculated for it.

mzilinec commented 3 years ago

Also see #34 - executables such as SLTev and ASRev should only be aliases linked to functions in setup.py and pip install will take care of it.

mohammad2928 commented 3 years ago

ASRev does not work independently (ASRev is a function that would be called from SLTev), but I have used entry_points for SLTev. Thanks.

eebism commented 3 years ago

As I see, we don't have ASRev as a separate file anymore, and it is currently a function inside the SLTev code.