search

iemejia / catho

A file catalog utility inspired by the awesome Robert Vasicek's Cathy project. Or my excuse to hack something that I really need.
GNU Lesser General Public License v3.0
6 stars 1 forks source link

Memory Error #31

Closed iemejia closed 11 years ago

iemejia commented 11 years ago

When indexing really long volumes

iemejia commented 11 years ago

an alternative is to measure the size of the files collection and add it in batches to the database probably using generators would be also a nice trick.

iemejia commented 11 years ago

or fix a number for batches also

iemejia commented 11 years ago

Maybe a more elegant solution exists but this works for the moment.

iemejia commented 11 years ago

the exception is still going on:

Python(15742) malloc: * mmap(size=140645843243008) failed (error code=12) * error: can't allocate region *\ set a breakpoint in malloc_error_break to debug Traceback (most recent call last): File "catho/catho.py", line 303, in for files in filesubsets: File "catho/catho.py", line 77, in file_get_filelist hash = file_hash(fullpath, hash_type) File "catho/catho.py", line 50, in file_hash sha1.update(f.read()) MemoryError

iemejia commented 11 years ago

Actually this is a different issue, see issue #32