By-pass the tokenizer

[wongtaksum]

Thank you for creating the tool for public use! I found that the tokenizer does not work well in some occasions. Is there any way to give a delimited input to your POS and dependency parser directly and by-pass your tokenizer?

[KoichiYasuoka]

By-pass the tokenizer... Well, you can do that, just using ufal.udpipe module directly:

>>> import os,ufal.udpipe,udchinese.udchinese
>>> m=ufal.udpipe.Model.load(os.path.join(udchinese.udchinese.PACKAGE_DIR,"ud-chinese.udpipe"))
>>> udpipe=ufal.udpipe.Pipeline(m,"conllu","","","")
>>> nlp=lambda x:udpipe.process("\n".join("\t".join([str(i+1),j]+["_"]*8) for i,j in enumerate(x.split()))+"\n\n")
>>> doc=nlp("不 入 虎穴 不 得 虎子")
>>> print(doc)
1   不   不   ADV v,副詞,否定,無界  Polarity=Neg    2   advmod  _   _
2   入   入   VERB    v,動詞,行為,移動  _   0   root    _   _
3   虎穴  虎穴  NOUN    n,名詞,固定物,地形 _   2   obj _   _
4   不   不   ADV v,副詞,否定,無界  Polarity=Neg    5   advmod  _   _
5   得   得   VERB    v,動詞,行為,得失  _   2   parataxis   _   _
6   虎子  虎子  NOUN    n,名詞,人,人    _   5   obj _   _

But it seems to cause several bad effects...