TobiasLee
commented
4 years ago I did the exact preprocessing process as described in the original paper. In more detail, the process includes:
- Segmentation, Chinese input needs segmentation, jieba is used in this repo.
- Build word vocabulary, note that the word not in the vocab is converted to UNK.
- Max length truncation and padding, remove the examples whose length is greater than the threshold(e.g. 60 word tokens) and pad shorter ones with 0.
Dear author, thank you for your repo. May I please ask for a data preprocessing method you used in this repo? Are there any samples? Thank you