Perf tradeoff reads vs writes

[alekcz]

I've changed the store to split layout. "/meta" and "/data" is added to keys and the we put and delete using transactions. We can the get-meta inexpensively.

The trade offs:

• -update now perform two reads one for meta and one for data.
• -keys now reads the entire db

This means we're prioritising read performance which seems reasonable to me, especially because being able to only read the meta is critical for the GC.

[whilo]

Do you know whether it is actually twice as slow? I think putting both puts in one transaction could have similar performance than doing one large put, but that is just speculation.

[alekcz]

@whilo I'm not sure about the actual performance, my thinking is that two different reads means you touch disk twice. LevelDB probably does some optimisation under the hood, but I imagine if the data isn't cached we would have to go to disk.

[whilo]

I think if both keys land in the same index fragment of leveldb it could fetch them at once. So we should use the same UUID key prefix and distinguish the value by some suffix like -value for instance.

[whilo]

Here we did it the other way around, but that is not good for cache locality: https://github.com/FerdiKuehne/konserve-rocksdb/blob/feature_meta_data_support/src/konserve_rocksdb/core.clj#L126

[alekcz]

That makes sense. I went with "/" instead of "-". That way leveldb, rocksdb, couchdb and firebase will all use the same format, so to speak.

[alekcz]

I'll make this change to Rocksdb and push them both to clojars in the morning.