About Chinese corpus

[BillKiller]

How to transfer a Chinese sentence to conllx format?

[allanj]

I will suggest you use some tools for Chinese word segmentation and then need to do dependency parsing. I think it shouldn’t be difficult to convert those information into conll format. You can simply write them to a file.

[BillKiller]

In fact , I asked same question weeks ago and I found that it is not easy to transform a Chinese sentence to conll-x format. Could you provide some examples that how to transform Chinese sentences to conll-x format? Appreciate.

[allanj]

I'm not sure what difficulties you are facing now. Would you like to tell me that? Basically, you are going to do the following steps.

1. Chinese word segmentation. (Some tools can help you do that, such as Jieba segmentation or Stanza by Stanford)
2. Dependency parsing (You can also use Stanza by Stanford)

Would you like to let me know which step you are in trouble?

[BillKiller]

Dependency parse need to tokenize first and tokenizer result may be wrong. for example 请:O 帮:O 我:O 打开:O 推特:S-App（means open the twitter for me）and slot is the correct result is 请\帮\我\打开\推特 Maybe the tokenizer result is 请\帮\我\打\开推特 and the Dependency parse will be wrong. If you use the correct word segmentation（you can get it by slot）, Dependency parse can not use custom tokenizer. Although stanza claims that you can use pretagged document but upos, xpos, feats, lemma need tokenize first.

[allanj]

Yeah. I guess you are right and this is the painful part we need to deal with. Given a Chinese sentence, I think the model might not be able to get the "correct" segmentation unless we annotate them. Thus, using an external tokenizer/segmenter should be the only choice. I would suggest you use the Jieba segmentation tool. Then pass the sentence to Stanza for dependency parsing.

[BillKiller]

Thx.