search

mhulden / foma

Automatically exported from code.google.com/p/foma
115 stars 90 forks source link

fix tokenized apply under python 3.10 #149

Open moocow-the-bovine opened 1 year ago

moocow-the-bovine commented 1 year ago

Dodge TypeErrors from FST.apply() with tokenize=True.

Reproduce:

import foma
next(foma.FST('foo').words(tokenize=True))

Traceback:

  File "/home/moocow/.local/lib/python3.10/site-packages/foma/__init__.py", line 329, in _apply
    foma_apply_set_space_symbol(c_void_p(applyerhandle), c_char_p(toksym))
TypeError: bytes or integer address expected instead of str instance