MemTable WAL implementation

[velvia]

This could be as simple as a one liner to enable it in the current MapDBMemTable, plus recovery logic. However, need to benchmark as the MapDBMemTable is already not fast.

[velvia]

Now that we've switched to FiloMemTable, need a fast in-memory off-heap storage that can also be persisted. Some options:

Chronicle-java

https://github.com/xerial/larray

[velvia]

@parekuti here are some guidelines for the write-ahead log implementation for the memtable.

Requirements

• Must be able to save the state of FiloAppendStore and FiloMemTable such that it could be restored if a crash happens
• Must be able to save new Filo chunks as appended by FiloAppendStore to disk
• If FiloAppendStore decides to rewrite the most current chunk, this must be handled (instead of appending new chunks, it replaces most recently appended chunks)

If a crash happens, the on disk file must restore all the state of the FiloAppendStore as well as the partSegKeyMap in the FiloMemTable. However, the thought is that the partSegKeyMap does not need to be preserved on disk because the partition and segment keys for each row could be recovered from the chunks themselves.

At a higher level, we must be able to restore the state of all the active NodeCoordinatorActors. Thus, the active and flushing memtables; for each NodeCoordinatorActor, the dataset, version, and ingestion schema / columns. This needs to be persisted somewhere.

Write-Ahead Log File Format

While the FiloMemTable already uses binary Filo chunks, we still need some file format for containing the chunks. So this is a proposal for the format.

File Header

The file header consists of the following bytes. The + signifies an offset in hex. Everything is written little endian.

• +0000: 8 bytes: The UTF8 for the string "FiloWAL" followed by 0x00
• +0008: 2 bytes: 0x0001 - signifying a header with column definitions
• +000a: 2 bytes: the little-endian number of columns
• +000c: 2 bytes: the number of bytes of column definitions (NN)
• +000e: NN bytes: The output of DataColumn.toString for each column, UTF8-encoded / written using DataWriter.write(string)
• +000e + NN: 2 bytes: 0x0002 - signifying a section holding Filo chunks, should have number of chunks corresponding to the number of columns
• +0010 + NN: 4 bytes: number of bytes of first Filo columnar chunk
• +0014 + NN: first Filo columnar chunk The above pattern repeats for each columnar chunk.

[velvia]

directory structure:

${memtable-wal-dir} / $dataset_$version / $timestamp.wal

Need to store datasets being written somewhere

[velvia]

@parekuti is working on this issue, but for some reason cannot assign this issue to her.

[velvia]

The PR for this has been merged.