search

valkey-io / valkey

A flexible distributed key-value datastore that is optimized for caching and other realtime workloads.
https://valkey.io
Other
17.58k stars 663 forks source link

[NEW] Cluster support without special client bindings #748

Open robd003 opened 4 months ago

robd003 commented 4 months ago

The problem/use-case that the feature addresses

I'd love it if Valkey could automatically scale horizontally by adding new nodes to a cluster. I'd also like to be able to query any node and get a response rather than having to have a client-side map of what data is on what node.

Using raft as a consensus algorithm would be fantastic

hpatro commented 4 months ago

@robd003 Could you share the inconvenience faced due to the current limitation?

I'd love it if Valkey could automatically scale horizontally by adding new nodes to a cluster.

Isn't this an operator action? Valkey can only emit out metrics to signal the usage of the engine.

I'd also like to be able to query any node and get a response rather than having to have a client-side map of what data is on what node.

Are the existing cluster-enabled client not viable for your needs? They maintain the map and deal with redirection(s). That should be agnostic to the application design.

Using raft as a consensus algorithm would be fantastic

Would love to know more, what are the concerns with the current design?

robd003 commented 4 months ago

@robd003 Could you share the inconvenience faced due to the current limitation?

I'd love it if Valkey could automatically scale horizontally by adding new nodes to a cluster.

Isn't this an operator action? Valkey can only emit out metrics to signal the usage of the engine.

What I mean by automatic scaling is the ability to add nodes without needing to refresh any written configuration on existing nodes. Best example would be adding another node to an ElasticSearch cluster. It just works.

I'd also like to be able to query any node and get a response rather than having to have a client-side map of what data is on what node.

Are the existing cluster-enabled client not viable for your needs? They maintain the map and deal with redirection(s). That should be agnostic to the application design.

I had issues in the past with Redis cluster changes needing to be manually set in the clients configuration. (A node would fail, then be replaced and all of the slot mappings would be inaccurate in the client) Being able to have everything "automagically" work would be ideal.

Using raft as a consensus algorithm would be fantastic

Would love to know more, what are the concerns with the current design?

Looks like this was covered here as the second item on the list: https://github.com/valkey-io/valkey/issues/17