search

grammarly / gector

Official implementation of the papers "GECToR – Grammatical Error Correction: Tag, Not Rewrite" (BEA-20) and "Text Simplification by Tagging" (BEA-21)
Apache License 2.0
891 stars 216 forks source link

data preprocessing #177

Open Lj4040 opened 1 year ago

Lj4040 commented 1 year ago

What do SOURCE and TARGER stand for in data preprocessing? Could you explain them? Thank you for your reply

skurzhanskyi commented 1 year ago

As it was mentioned , source is original text, target is corrected text

Lj4040 commented 1 year ago

For example, what I downloaded is the FCE data set, which contains M2 file and json file. In this file, there is no distinction between correct and incorrect sentences. How should I pass the data processing file.I would like to ask for your guidance, for which I greatly appreciate it

Lj4040 commented 1 year ago

Only the downloaded synthetic data set has correct and incorrect sentences, do we have to use the synthetic data to pass in?

skurzhanskyi commented 1 year ago

You can take a look at the M2scorer repository and specifically the edit_creator.py script. To get original/source sentences, you can simply run cat myfile.m2 | grep "^S " | cut -c3- > myfile.src

Lj4040 commented 1 year ago

Sincerely thank you for your answer, I will try

Lj4040 commented 1 year ago

Dear author, after data processing, the data set has become the following picture. This format file is quite different from the M2 file, so I'm not sure if it's correct.May I ask if the data set in this format is correct? 图片1 Because I am a beginner of GEC, some questions may be a little naive. I hope you can understand. Thank you for your reply

skurzhanskyi commented 1 year ago

Yes, this is a specific format for training to save only input tokens and corresponding tags