Open luvensaitory opened 6 years ago
I have the same problem with Otomí (mexican language). My Traceback looks like this
'ascii' codec can't encode character '\xe9' in position 8: ordinal not in range(128)
And the first three elements of xseq
list looks like this:
[[b'bias', b'letterLowercase=d', b'postag=unkwn', b'BOS', b'nxtpostag=cnj', b'BOW', b'nxtletter=<i', b'nxt2letters=<ig', b'nxt3letters=<ige', b'nxt4letters=<igeh'], [b'bias', b'letterLowercase=i', b'postag=unkwn', b'BOS', b'nxtpostag=cnj', b'letterposition=-7', b'prevletter=d>', b'nxtletter=<g', b'nxt2letters=<ge', b'nxt3letters=<geh', b'nxt4letters=<geh\xc3\xb1'], [b'bias', b'letterLowercase=g', b'postag=unkwn', b'BOS', b'nxtpostag=cnj', b'letterposition=-6', b'prev2letters=di>', b'prevletter=i>', b'nxtletter=<e', b'nxt2letters=<eh', b'nxt3letters=<eh\xc3\xb1', b'nxt4letters=<eh\xc3\xb1a']]
In previous step i try to do this for encoding but seems not works property:
featurelist.append([f.encode('utf-8') for f in features])
Is this problem solved? I have the same problem with same error trying to train the NER model with Chinese too...
Details![image](https://user-images.githubusercontent.com/17849431/44319506-e6c79f00-a46e-11e8-884d-2aa00ca35add.png)
My data is about Chinese Math questions : 小蓉吃了8顆水餃,小宇吃了10顆水餃,誰吃的水餃比較多? ( )吃的多
And the training data is : 小 小蓉 人名 B-人名 蓉 小蓉 人名 E-人名 吃 吃 VC S 了 了 Di S 8 8 Neu S 顆 顆 Nf S 水 水餃 Na S 餃 水餃 Na S , , COMMACATEGORY S 小 小宇 人名 B-人名 宇 小宇 人名 E-人名 吃 吃 VC S 了 了 Di S 1 10 Neu S 0 10 Neu S 顆 顆 Nf S 水 水餃 Na S 餃 水餃 Na S , , COMMACATEGORY S 誰 誰 Nh S 吃 吃 VC S 的 的 DE S 水 水餃 Na S 餃 水餃 Na S 比 比較 Dfa S 較 比較 Dfa S 多 多 VH S ? ? QUESTIONCATEGORY S ( ( PARENTHESISCATEGORY S ) ) PARENTHESISCATEGORY S 吃 吃 VC S 的 的 DE S 多 多 VH S