WorksApplications / SudachiPy

Python version of Sudachi, a Japanese tokenizer.
Apache License 2.0
392 stars 50 forks source link

"BufferError" happens when calling create method of dictionary object #99

Closed araiman closed 4 years ago

araiman commented 5 years ago

Dear maintainers.

On creating the dictionary object, the following error has happened. This error got in version 0.4.0 but it doesn't happen in 0.3.8. Would you mind to check is this error?

Code

import SudachiPy

dictionary_obj = dictionary.Dictionary().create()

Error

BufferError: memoryview: underlying buffer is not writable

/aaa/python/lib/python3.5/site-packages/sudachipy/dictionary.py in __init__(self, config_path, resource_dir)
     35         self.dictionaries = []
     36         self.header = None
---> 37         self._read_system_dictionary(config.settings.system_dict_path())
     38 
     39         # self.edit_connection_plugin = [InhibitConnectionPlugin()]

/aaa/python/lib/python3.5/site-packages/sudachipy/dictionary.py in _read_system_dictionary(self, filename)
     64         if filename is None:
     65             raise AttributeError("system dictionary is not specified")
---> 66         dict_ = BinaryDictionary.from_system_dictionary(filename)
     67         self.dictionaries.append(dict_)
     68         self.grammar = dict_.grammar

/aaa/python/lib/python3.5/site-packages/sudachipy/dictionarylib/binarydictionary.py in from_system_dictionary(cls, filename)
     48     @classmethod
     49     def from_system_dictionary(cls, filename):
---> 50         args = cls._read_dictionary(filename)
     51         version = args[2].version
     52         if version != SYSTEM_DICT_VERSION:

/aaa/python/lib/python3.5/site-packages/sudachipy/dictionarylib/binarydictionary.py in _read_dictionary(filename, access)
     43             offset += grammar.get_storage_size()
     44 
---> 45         lexicon = DoubleArrayLexicon(bytes_, offset)
     46         return bytes_, grammar, header, lexicon
     47 

/aaa/python/lib/python3.5/site-packages/sudachipy/dictionarylib/doublearraylexicon.py in __init__(self, bytes_, offset)
     40 
     41         array = memoryview(bytes_)[offset:offset + size * 4]
---> 42         self.trie.set_array(array, size)
     43         offset += self.trie.total_size()
     44 

/aaa/python/lib/python3.5/site-packages/dartsclone/_dartsclone.cpython-35m-x86_64-linux-gnu.so in dartsclone._dartsclone.DoubleArray.set_array (dartsclone/_dartsclone.cpp:1898)()

/aaa/python/lib/python3.5/site-packages/dartsclone/_dartsclone.cpython-35m-x86_64-linux-gnu.so in View.MemoryView.memoryview_cwrapper (dartsclone/_dartsclone.cpp:10995)()

/aaa/python/lib/python3.5/site-packages/dartsclone/_dartsclone.cpython-35m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__cinit__ (dartsclone/_dartsclone.cpp:7270)()

Environment

Python: 3.5
SudachiPy: 0.4.0
Dict_Core: https://object-storage.tyo2.conoha.io/v1/nc_2520839e1f9641b08211a5c85243124a/sudachi/SudachiDict_core-20190718.tar.gz
izziiyt commented 5 years ago

I'm very glad to get FB about latest SudachiPy. I can't figure out the reason from just a glance. Please use v0.3.13 before this problem will have been resolved. I'll ask you something about this issue later probably. Thanks.

araiman commented 5 years ago

Thanks for your reply. I'll use v0.3.13 before for a while. Please feel free to ask me. Thanks.