dpwe
commented
7 years ago The system allocates a fixed array of numhashes bucketsize 4 bytes. With nhashbits==20 (the default), a bucketsize of 512 pushes the array size to a number larger than can be stored in a 32 bit int.
I verify that I get the same problem on my 64 bit MacOS system. The error is actually in saving the large data structure:
dpwe@dpwe-macbookpro2:~/projects/audfprint$ ~/homebrew/bin/python
audfprint.py new --bucketsize 512 --dbase fpdbase.pkl Nine_Lives/01*.mp3
Mon May 1 09:16:23 2017 ingesting #0: Nine_Lives/01-Nine_Lives.mp3 ...
Added 194 hashes (19.4 hashes/sec)
Processed 1 files (10.0 s total dur) in 0.0 s sec = 0.005 x RT
Traceback (most recent call last):
File "audfprint.py", line 482, in <module>
main(sys.argv)
File "audfprint.py", line 477, in main
hash_tab.save(dbasename)
File "/Users/dpwe/projects/audfprint/hash_table.py", line 168, in save
pickle.dump(self, f, pickle.HIGHEST_PROTOCOL)
SystemError: error return without exception setSome quick googling suggests the problem is in the underlying MacOS implementation (I'm using Python 2.7.9 from Homebrew):
http://stackoverflow.com/questions/31468117/python-3-can-pickle-handle-byte-objects-larger-than-4gb
.. which suggests a workaround by replacing the pickle.dump command with writing in smaller pieces, but I don't know if it's worth putting a work-around like that into the main code base; such large databases are perhaps not the ideal use case.
DAn.
On Mon, May 1, 2017 at 5:18 AM, altec404 notifications@github.com wrote:
Hello dpwe, The error was occurred when I set --bucketsize 512 or more. Why max bucket num is less than 511? Sorry my poor English. Thanks.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
altec404
Hello dpwe, The error was occurred when I set --bucketsize 512 or more. Why max bucket num is less than 511? Sorry my poor English. Thanks.