Merge multiple Updates into single file

[jankotek]

IODB right now stored each Update in separate file. That leads to problem where file handles are exhausted and JVM crashes. Default maximal number of opened files is around 64K. It can be modified at OS level, but IODB should not require that.

This also reads to huge memory overhead, since each update consumes about 400 bytes of heap space for file handle, versionID etc.

So we need to change format, so that multiple Updates are stored within single file (and single file handle). Use linked-list of updates as outlined here: http://www.mapdb.org/blog/lsm_store_and_updates/

Changes:

• drop log as a component

• find efficient way to represent single update

  • use primitive collections, so memory consumption should 8 bytes per update, file# and offset?
  • versionID is byte[] so perhaps use hashcode multimap to identify Updates?

• append-only files can not be written with mmap files

  • JVM crash with sparse files if disk space runs out
  • use FileOutputStream to write data
  • readonly mmap huge region of file, which was not yet written. Data should become readable by mmap, once they are written by FileOutputStream

• reopen code needs to traverse all updates and assemble linked tree of Updates

  • multithreaded?

[jankotek]

Fixed in 0.2 release