search

nusnlp / crosentgec

Code for cross-sentence grammatical error correction using multilayer convolutional seq2seq models (ACL 2019)
GNU General Public License v3.0
50 stars 12 forks source link

How to prepare a new testing dataset? #7

Closed pidugusundeep closed 4 years ago

pidugusundeep commented 4 years ago
shamilcm commented 4 years ago

The scripts to prepare test sets from CoNLL-2013 and CoNLL-2014 are given here: https://github.com/nusnlp/crosentgec/blob/master/prepare_test.sh#L26

Format for test set includes two files: (1) context file: contains the document-level context, i.e., two previous sentences (of the source sentence to be corrected) per line. For the first source sentence in an essay, context line is empty, for the second source sentence, context line will be a single previous sentence. (2) source file: one source sentence, i.e., the sentence to be corrected per line. Note that you should pass tokenized and subword segmented sentences to the model. Refer decode.sh to understand how the test data should be preprocessed.

Before you create your own test sets, try to follow the steps to decode and replicate the results.

shamilcm commented 4 years ago

You could use any of the four CroSent models first and you should get a similar result as in the paper. Each model is simply trained with a different random initialization (random seed) without any other significant differences.

pidugusundeep commented 4 years ago

The scripts to prepare test sets from CoNLL-2013 and CoNLL-2014 are given here: https://github.com/nusnlp/crosentgec/blob/master/prepare_test.sh#L26

Format for test set includes two files: (1) context file: contains the document-level context, i.e., two previous sentences (of the source sentence to be corrected) per line. For the first source sentence in an essay, context line is empty, for the second source sentence, context line will be a single previous sentence. (2) source file: one source sentence, i.e., the sentence to be corrected per line. Note that you should pass tokenized and subword segmented sentences to the model. Refer decode.sh to understand how the test data should be preprocessed.

Before you create your own test sets, try to follow the steps to decode and replicate the results.

I ran the decode for the CoNLL-2013 and CoNLL-2014 but where are the original results to compare with and to check if I am able to replicate the same results.

shamilcm commented 4 years ago

Testing is following the official CoNLL-2014 shared task. Use M2Scorer to evaluate using the annotated test data (https://www.comp.nus.edu.sg/~nlp/conll14st.html)