search

fxsjy / jieba

结巴中文分词
MIT License
33.13k stars 6.72k forks source link

可以处理纯英文的文章么？ #447

Open wanghai1988 opened 7 years ago

Artoria2e5 commented 7 years ago

我只能告诉你这玩意看得懂空格和标点……（

gumblex commented 7 years ago

用 https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl