iemejia / catho

A file catalog utility inspired by the awesome Robert Vasicek's Cathy project. Or my excuse to hack something that I really need.
GNU Lesser General Public License v3.0
6 stars 1 forks source link

Error when calculating hash of BIG files #32

Closed iemejia closed 11 years ago

iemejia commented 11 years ago

I found it trying to index a VM that's bigger than the memory in my local machine:

Python(15742) malloc: * mmap(size=140645843243008) failed (error code=12) * error: can't allocate region *\ set a breakpoint in malloc_error_break to debug Traceback (most recent call last): File "catho/catho.py", line 303, in for files in filesubsets: File "catho/catho.py", line 77, in file_get_filelist hash = file_hash(fullpath, hash_type) File "catho/catho.py", line 50, in file_hash sha1.update(f.read()) MemoryError