EPE 2017

[datquocnguyen]

Hi, I am using the development branch of TEES to evaluate a parser with respect to the EPE 2017 shared task. In https://github.com/jbjorne/TEES/wiki/EPE-2017 , I only find the information about getting F1 scores on the "devel" set. Here the command line I used: python EvaluateEPE.py -i [INPUT] -o [OUTPUT] -n 5 -t True --subTask 1 --pos pos Could you please provide instructions (or command line) to extract the results on the "test" set ? Thank you very much. Best regards, Dat.

[datquocnguyen]

I think I could use the information from https://github.com/jbjorne/TEES/wiki/Classifying to get the results on test set. Please correct me if I was wrong. Thanks a lot.

[jbjorne]

The EPE 2017 shared task used (for the biomedical event extraction downstream task) the 2009 BioNLP Shared Task corpus. The downloadable corpus and the official evaluator program can be used (in the context of the EPE task) according to the instructions at https://github.com/jbjorne/TEES/wiki/Classifying.

However, for evaluating the BioNLP 2009 corpus test set, the official online evaluation server must be used. Please see the 2009 BioNLP Shared Task home page for more information regarding this.

[datquocnguyen]

Thanks for your quick response. So, in the final results in Table 1 (page 24) in "EPE 2017: The Biomedical Event Extraction Downstream Application", scores on the development set are calculated using TEES while scores on the evaluation set are reported using the online evaluation server?

I find when using TEES, the classification-test output folder contains the following files:

Which file should I use to upload to the online evaluation server ?

Thanks a lot. Best, Dat.

[jbjorne]

The development set results are calculated using TEES, but only in the sense that TEES provides a wrapper for the downloadable BioNLP 2009 Shared Task evaluator program. This evaluator can also be run on its own.

In any case, the BioNLP evaluators (program or online service) use the BioNLP Shared Task format (a1/a2/txt files), which in the TEES output files means the *-events.tar.gz archives. In the classification-test output directory, the test-events.tar.gz file contains the results in the BioNLP Shared Task format (it is generated from the test-pred.xml.gz file) and it is the file that can be used with the BioNLP Shared Task online evaluation server.

[datquocnguyen]

Thanks a lot for your clarification.

[datquocnguyen]

I find the online evaluation server is currently not working. Can I get any support from you to get the scores on the test set? Thank you very much.

[jbjorne]

Regarding issues with the BioNLP 2009 Shared Task corpus and its online services please contact NaCTeM.

[datquocnguyen]

Thanks for your information.