Looking for a simple command and its documentation for evaluating EAL output

[simon-clematide]

Hi there This repo is linked as the official 2017 evaluation repo, but we do have severe problems to understand how any of these software pieces can be used for simply evaluating the output of our EAL system using this official scorer.

Ideally it would be some shell command with its relevant arguments and options, or at least a working example for a configuration file.

Thanks a lot.

[jayded]

I think you're looking for https://github.com/BBN-E/tac-kbp-eal/blob/master/Evaluation.md and https://github.com/BBN-E/tac-kbp-eal/blob/master/bin/evaluate2016.sh

The class that runs scoring is https://github.com/BBN-E/tac-kbp-eal/blob/master/tac-kbp-eal-scorer/src/main/java/com/bbn/kbp/events/ScoreKBPAgainstERE.java

[gabbard]

To clarify, the only people who can currently run scoring for 2017 is NIST, because they are the only ones with access to the gold standard data to score against. After the official scoring is complete, a 2017 shell script like evaluate2016.sh will be added to the repository. You are correct that we should link to these from the main README file.

[jayded]

No new traffic on this issue and the evaluation finished. Closing. Please re-open with a specific action item if something in particular needs to be done.