lucid-kv / lucid

High performance and distributed KV store w/ REST API. 🦀
https://clintnetwork.gitbook.io/lucid/
MIT License
373 stars 31 forks source link

[Proposition] Dump / Export #55

Open imclint21 opened 4 years ago

imclint21 commented 4 years ago

Hi,

While I was eating a cake I think about making exporting/dump function.

What do you think @CephalonRho?

Best regards

shuni64 commented 4 years ago

Shouldn't be too hard to do. We would have to use dashmap instead of chashmap and maybe put an RwLock around it to make sure that it doesn't change during iteration. We might also want to wait until there's progress on persistence since it could impact this significantly.

There's also the question of how we want to expose this feature. We could provide a list of all key names over http and make the client retrieve each value independently (but values could get deleted while that's happening). Once persistence is done there could just be another subcommand instead that dumps all elements to a single file in a binary format and another subcommand to load this file.

imclint21 commented 4 years ago

Shouldn't be too hard to do. We would have to use dashmap instead of chashmap and maybe put an RwLock around it to make sure that it doesn't change during iteration. We might also want to wait until there's progress on persistence since it could impact this significantly.

About dashmap, I already spoken with @xacrimon, but I think it was only promotion, not real benefits, but apparently it's not the case.

There's also the question of how we want to expose this feature. We could provide a list of all key names over http and make the client retrieve each value independently (but values could get deleted while that's happening). Once persistence is done there could just be another subcommand instead that dumps all elements to a single file in a binary format and another subcommand to load this file.

For me there are two things, for an endpoint go get all keys, it's surely a good idea, I don't really know how we can do that and be REST compliant, but anyway, a backup/restore command should to be implemented, for DevOps usage!