Closed pidugusundeep closed 4 years ago
The scripts to prepare test sets from CoNLL-2013 and CoNLL-2014 are given here: https://github.com/nusnlp/crosentgec/blob/master/prepare_test.sh#L26
Format for test set includes two files: (1) context file: contains the document-level context, i.e., two previous sentences (of the source sentence to be corrected) per line. For the first source sentence in an essay, context line is empty, for the second source sentence, context line will be a single previous sentence. (2) source file: one source sentence, i.e., the sentence to be corrected per line. Note that you should pass tokenized and subword segmented sentences to the model. Refer decode.sh to understand how the test data should be preprocessed.
Before you create your own test sets, try to follow the steps to decode and replicate the results.
You could use any of the four CroSent models first and you should get a similar result as in the paper. Each model is simply trained with a different random initialization (random seed) without any other significant differences.
The scripts to prepare test sets from CoNLL-2013 and CoNLL-2014 are given here: https://github.com/nusnlp/crosentgec/blob/master/prepare_test.sh#L26
Format for test set includes two files: (1) context file: contains the document-level context, i.e., two previous sentences (of the source sentence to be corrected) per line. For the first source sentence in an essay, context line is empty, for the second source sentence, context line will be a single previous sentence. (2) source file: one source sentence, i.e., the sentence to be corrected per line. Note that you should pass tokenized and subword segmented sentences to the model. Refer decode.sh to understand how the test data should be preprocessed.
Before you create your own test sets, try to follow the steps to decode and replicate the results.
I ran the decode for the CoNLL-2013 and CoNLL-2014
but where are the original results to compare with and to check if I am able to replicate the same results.
Testing is following the official CoNLL-2014 shared task. Use M2Scorer to evaluate using the annotated test data (https://www.comp.nus.edu.sg/~nlp/conll14st.html)