calamari-eval: skip missing pairs

[bertsky]

There seems to be a mismatch between documentation and implementation: --non_existing_file_handling_mode is ignored completely.

Also, IIUC that leaves me with no option on the CLI to skip missing file pairs – I can only use --gt.non_existing_as_empty and --skip_empty_gt and --evaluator.non_existing_pred_as_empty, which is obviously not the same.

(My use-case: I have a subset of manually edited files and lots more of predictions, and want to restrict calculation to that subset.)

[andbue]

That's weird, you're right. Does --skip_empty_gt do what you need?

[bertsky]

Kind of. But I'd have to check against existing lines that are actually empty first...

[andbue]

Ah, I get it, skip_empty_gt would not count any wrong predictions in actually empty lines as errors.