Can Collection object be serialized?

kelindar / column

High-performance, columnar, in-memory store with bitmap indexing in Go

MIT License

1.44k stars 57 forks source link

Can Collection object be serialized? #33

Closed didip closed 2 years ago

didip commented 2 years ago

I am thinking of serializing it into BadgerDB. Is there a safe mechanism to do so?

kelindar commented 2 years ago

Snapshots aren't supported right now, but I've been thinking of implementing it. Only thing supported is CDC via a transaction commit log, which can be used for durability purposes and replayed to create a read replica.

Do keep in mind tho, the API of this project isn't stable yet and will change in the future (guaranteed) - hence it's not yet versioned.

kelindar commented 2 years ago

Fyi, I merged the snapshotting today (see https://github.com/kelindar/column/pull/39), there's still some work to be done to merge/reconcile incomplete transactions in case the snapshot happens concurrently with new writes, but most of the stuff should work.