Custom order of arguments (SLTeval -i ... -f ...) is order-dependent

[obo]

I have three files called slt (the hypothesis of an SLT system), ref (the plain text reference translation) and ostt (timed source transcript). Yes, they are somewhat problematic, but still, SLTeval should behave identically regardless the order in which I specify them.

What I am getting is this:

===== SLTeval -f ostt ref slt -i ostt ref slt =====
evaluation for  slt  failed, the number of Complete lines (C) in  ostt  and  ref  are not equal
The number of C segment (complete) in ostt is 1 and number of lines in ref is 25
===== SLTeval -f ostt slt ref -i ostt slt ref =====
evaluation failed, the reference file does not exist for  slt
===== SLTeval -f ref ostt slt -i ref ostt slt =====
evaluation for  slt  failed, the number of Complete lines (C) in  ostt  and  ref  are not equal
The number of C segment (complete) in ostt is 1 and number of lines in ref is 25
===== SLTeval -f ref slt ostt -i ref slt ostt =====
evaluation failed, the OStt file does not exist for  slt
===== SLTeval -f slt ostt ref -i slt ostt ref =====
evaluation for  slt  failed, the number of Complete lines (C) in  ostt  and  ref  are not equal
The number of C segment (complete) in ostt is 1 and number of lines in ref is 25
===== SLTeval -f slt ref ostt -i slt ref ostt =====
evaluation for  slt  failed, the number of Complete lines (C) in  ostt  and  ref  are not equal
The number of C segment (complete) in ostt is 1 and number of lines in ref is 25

[obo]

Just tested it also with files that work and got the same problem: some ordering of arguments fails:

===== SLTeval -f ostt ref slt -i ostt ref slt =====
Evaluating the file  slt  in terms of translation quality against  ref
P ... considering Partial segments in delay and quality calculation(in addition to Complete segments)
...good output
===== SLTeval -f ostt slt ref -i ostt slt ref =====
evaluation failed, the reference file does not exist for  slt
===== SLTeval -f ref ostt slt -i ref ostt slt =====
Evaluating the file  slt  in terms of translation quality against  ref
P ... considering Partial segments in delay and quality calculation(in addition to Complete segments)
...good output
===== SLTeval -f ref slt ostt -i ref slt ostt =====
evaluation failed, the OStt file does not exist for  slt
===== SLTeval -f slt ostt ref -i slt ostt ref =====
Evaluating the file  slt  in terms of translation quality against  ref
P ... considering Partial segments in delay and quality calculation(in addition to Complete segments)
...good output
===== SLTeval -f slt ref ostt -i slt ref ostt =====
Evaluating the file  slt  in terms of translation quality against  ref
P ... considering Partial segments in delay and quality calculation(in addition to Complete segments)
...good output

[mohammad2928]

Hi, The order of files is important because SLTev needs to support multiple documents at once. So, candidates (such as SLT or ASR) must be before or after input files (Please see issue #58).

e.g. Correct format:

1. SLTeval -i ref ostt slt ref ostt slt -f ref ostt slt ref ostt slt
2. SLTeval -i slt ref ostt slt ref ostt -f slt ref ostt slt ref ostt

Wrong format:

1. SLTeval -i ref slt ostt ref slt ostt -f ref slt ostt ref slt ostt

Idea:

1. We can ignore orders if there is just one candidate file.

[obo]

Hi, Good point. Then the order can remain relevant but the documenation and error message have to be clear. Perhaps the calling convention with -f and -i, each with two lists, is not optimal. It invites for such a calling error. O.

On June 23, 2021 12:29:21 AM GMT+02:00, Mohammad Mahmoudi @.***> wrote:

Hi, The order of files is important because SLTev needs to support multiple documents at once. So, candidates (such as SLT or ASR) must be before or after input files (Please see issue #58).

e.g. Correct format:

1. SLTeval -i ref ostt slt ref ostt slt -f ref ostt slt ref ostt slt
2. SLTeval -i slt ref ostt slt ref ostt -f slt ref ostt slt ref ostt

Wrong format:

1. SLTeval -i ref slt ostt ref slt ostt -f ref slt ostt ref slt ostt

Idea:

1. We can ignore orders if there is just one candidate file.

-- Ondrej Bojar @. / @.) http://www.cuni.cz/~obo

[mohammad2928]

I have added an error message to the SLTev for handling the order and update the README file.