pytries / datrie

Fast, efficiently stored Trie for Python. Uses libdatrie.
http://pypi.python.org/pypi/datrie/
GNU Lesser General Public License v2.1
530 stars 88 forks source link

Memory corruption with slighthly bigger alphabets. #74

Open c4f3a0ce opened 4 years ago

c4f3a0ce commented 4 years ago

When I try to use Trie with larger alphabet I start to experience different memory related issue. Details differ from Python version to Python version, and environment, but in general following:

import datrie

xs = [chr(i) for i in range(500)]

datrie.Trie(xs[:250])  # Works fine
datrie.Trie(xs[250:])  # Works fine

datrie.Trie(xs)  # Fails miserably 

fails when executed as script with:

malloc(): invalid size (unsorted)
[1]    19260 abort      python foo.py

In IPython I got:

corrupted double-linked list                                                                                                                                                    
[1]    19270 abort      ipython

or

malloc(): invalid size (unsorted)
[1]    19364 abort      ipython

and some other, less common failures.

This specific outcomes tested with

Is this something expected?

c4f3a0ce commented 4 years ago

After further investigation it seems it might be related to https://github.com/tlwg/libdatrie/issues/6.

I've updated and rebuild the package from fa3cfc3f78be4d27d1366ae8c1b39d0a91be0dca by updating libdatrie submodule to https://github.com/tlwg/libdatrie/commit/6ef4485474890606946ed208ff453231b581cdf1 and the problem disappears.