search

inukshuk / wapiti-ruby

Wicked fast Conditional Random Fields for Ruby
Other
37 stars 7 forks source link

Error when tokenizing some Chinese characters #12

Open inukshuk opened 9 months ago

inukshuk commented 9 months ago

If string contains "著":

Wapiti::NativeError (missing tokens at 0: 19/19, cannot apply pattern)

See https://github.com/inukshuk/anystyle/issues/223