search

nusnlp / mlconvgec2018

Code and model files for the paper: "A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction" (AAAI-18).
GNU General Public License v3.0
185 stars 73 forks source link

Files in outputs Dir #6

Closed xixiddd closed 6 years ago

xixiddd commented 6 years ago

Is the file(best_with-spellcheck.conll14st-test.tok.out) in outputs dir corresponding to the best results in paper? when I run the m2score with this file and gold truth, I only got F0.5 < 30.

xixiddd commented 6 years ago

and, it takes 5 hours to run m2score on Conll14 testset, is that normal?

shamilcm commented 6 years ago

I have verified again, the output file is ok. It takes almost half an hour for evaluation with the M2Scorer.

Can you also verify if you used the correct M2Scorer (http://www.comp.nus.edu.sg/~nlp/sw/m2scorer.tar.gz) and the CoNLL-2014 test set M2 reference file (mlconvgec2018/data/test/conll14st-test/conll14st-test.m2 after doing ./prepare_test_data.sh in data/ directory) .

$echo start time `date` ; $M2SCORER outputs/best_with-spellcheck.conll14st-test.tok.out data/test/conll14st-test/conll14st-test.m2 ; echo end time `date`
start time Sat Jun 2 18:49:25 +08 2018
Precision   : 0.6549
Recall      : 0.3314
F_0.5       : 0.5479
end time Sat Jun 2 19:26:37 +08 2018
xixiddd commented 6 years ago

Yes, you are right, thank you, but there is a typo, recall should be 0.3314

shamilcm commented 6 years ago

Yes, you’re right. Corrected in the comment above.