man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.51k stars 93 forks source link

Introduce the block version ref key #1969

Open poodlewars opened 3 weeks ago

poodlewars commented 3 weeks ago

Useful for background jobs.

See design doc shared internally - https://manwiki.maninvestments.com/x/n4CsFg .

test_snapshot.cpp has rotted and wasn't included in any cmake targets (so hasn't compiled for months or ever). Since it didn't check anything very useful, I've just removed it here.

An alternative design would be to store the segment sorted by stream ID, like we do for snapshots, and work with it directly rather than transforming it in to an unordered_map. This would save us the cost of transforming the segment to the unordered_map and back again. I plan to see how the current implementation performs in replication and we can swap out to the other idea at any time until we release this (the point of no return is once unordered block ref keys are serialized in production).