search

Netflix / dynomite-manager

A sidecar to manage Dynomite clusters
https://github.com/Netflix/dynomite
Apache License 2.0
91 stars 59 forks source link

Cold Bootstraping for ARDB RocksDB as a persistent backend? #90

Closed deegoy2 closed 6 years ago

deegoy2 commented 6 years ago

Hi, I love the work you guys have done with Dynomite, Dyno and DM. Huge fan. :)

Does Cold Bootstrapping work inherently with RocksDB as a persistent backend or is it designed only around Redis? If yes, is there a reference implementation? If no, what can be done to achieve this? Cold bootstrapping is an essential feature to have towards a fully scalable distributed persistent DB.

Also, I wanted to understand what drove the design decisions towards not implementing an auto-scaling consistent hash ring (and instead using a dual-write approach to achieve horizontal scalability). Was it related to the requirements at Netflix or lack of support from OSS or pluggable backends lacking full support or something else?

deegoy2 commented 6 years ago

Gentle ping...

ipapapa commented 6 years ago

Yes, it does work for RocksDB. You can use something like ARDB that embeds RocksDB and provides Redis API. Hence cold bootstrap is leveraging the master-slave replication.

We have captured a few of the issues and solutions of the auto-scaling in the Scale up and down project. I have captured some of the notes https://github.com/Netflix/dynomite/issues/342 Dual-writer can be used if the data has TTL. It is a convenient way of having a simple client functionality over a more complicated server feature. Currently, we are looking for folks assisting to build the scale up and scale down feature. If you are using Dynomite as a cache the steps that I describe should be enough.

deegoy2 commented 6 years ago

Thank you @ipapapa :D