verterok / tritcask

Yet another python, bitcask inspired, fast, log based key/value store
GNU General Public License v3.0
18 stars 8 forks source link

Tritcask fails to load data file bigger than address space. #10

Open luciotorre opened 10 months ago

luciotorre commented 10 months ago

When a huge data file is created in a session, Tritcask fails to load it in the next sesssion.

This is caused by the use of mmap on a file bigger than the address space, in this case 9GB.

Here is the traceback:

  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/ubuntuone-client/ubuntuone/syncdaemon/tritcask.py", line 636, in __init__
    self._build_keydir()
  File "/usr/lib/python2.7/dist-packages/ubuntuone-client/ubuntuone/syncdaemon/tritcask.py", line 791, in _build_keydir
    self._load_from_data(self.live_file)
  File "/usr/lib/python2.7/dist-packages/ubuntuone-client/ubuntuone/syncdaemon/tritcask.py", line 819, in _load_from_data
    for entry in data_file.iter_entries():
  File "/usr/lib/python2.7/dist-packages/ubuntuone-client/ubuntuone/syncdaemon/tritcask.py", line 254, in iter_entries
    fmmap = mmap.mmap(self.fd.fileno(), 0, access=mmap.ACCESS_READ)
ValueError: mmap length is too large
luciotorre commented 10 months ago

https://bugs.launchpad.net/ubuntuone-client/+bug/987376