search

WorksApplications / SudachiPy

Python version of Sudachi, a Japanese tokenizer.
Apache License 2.0
392 stars 50 forks source link

`Lattice.dump` error: 'sudachipy.latticenode.LatticeNode' object has no attribute 'is_defined' #128

Closed adrianeboyd closed 4 years ago

adrianeboyd commented 4 years ago

When logging is enabled, I noticed a bug in Lattice.dump, which might be related to the recent cython-related modifications? The error looks like this, calling sudachipy from spacy:

  File "/tmp/venv/lib/python3.7/site-packages/spacy/lang/ja/__init__.py", line 116, in get_dtokens
    tokens = tokenizer.tokenize(text)
  File "sudachipy/tokenizer.pyx", line 149, in sudachipy.tokenizer.Tokenizer.tokenize
  File "sudachipy/lattice.pyx", line 134, in sudachipy.lattice.Lattice.dump
AttributeError: 'sudachipy.latticenode.LatticeNode' object has no attribute 'is_defined'
sorami commented 4 years ago

@adrianeboyd Thank you for the notice, let me have a look.

sorami commented 4 years ago

I confirmed that bug does not happen with v0.4.5 which is pre-Cython.

sorami commented 4 years ago

The fix released in v0.4.8

adrianeboyd commented 4 years ago

Thanks for the quick fix!