search

taoensso / carmine

Redis client + message queue for Clojure
https://www.taoensso.com/carmine
Eclipse Public License 1.0
1.16k stars 132 forks source link

Add Redis Cluster support #76

Closed ptaoussanis closed 2 years ago

ptaoussanis commented 10 years ago

@bpoweski Hi Ben, just pinging you to reopen this - didn't realise GitHub had automatically closed the last PR (I wish they wouldn't close issues automatically).

Salvatore's just posted on multi-key Cluster support: https://groups.google.com/forum/#!msg/redis-db/R7hKYR5iBuo/8owvFFhm84sJ

Shouldn't require much (anything?) from our end.

If you're still interested in finishing up the implementation, just let me know if there's anything I can assist with.

Cheers! :-)

bpoweski commented 10 years ago

Definitely interested, just fighting the time constraints.

I'm tracking and rebasing against dev on my fork. So don't feel like you need to wait for me if want to make progress on it.

ptaoussanis commented 10 years ago

Absolutely no rush or stress - please take your time!

I'm tracking and rebasing against dev on my fork.

Sure. You'll notice the old cluster branch is gone and merged into dev. It should be stable from here - all the protocol stuff that needed reworking for Cluster is done. The taoensso.carmine.cluster namespace is basically a skeleton with some notes. A lot of your code should be transplantable there.

So don't feel like you need to wait for me if want to make progress on it.

No, no - not at all. My attention's elsewhere atm and I'm not using Cluster myself so this isn't something I'll be hacking on. The initial burst in activity was just to get the protocol stuff in the right state for supporting Cluster pipelines.

ptaoussanis commented 10 years ago

Quick update: discovered a minor bug with the new protocol, just pushed a fix to dev branch. The change is purely an implementation detail: won't interfere with the Cluster API at all.

ptaoussanis commented 10 years ago

Update: going to be cutting a new stable release soon with all the changes in the dev branch so far.

bpoweski commented 10 years ago

Sounds great. I'll pull in the changes.

shmish111 commented 9 years ago

@bpoweski did you get anywhere with this?

bpoweski commented 9 years ago

@shmish111 I was able to get a proof of concept working but then we went a different route for the project where I planned to use it.

Coincidently, we're currently building out a twemproxy + docker + consul environment as it fits some of our goals better.

shmish111 commented 9 years ago

@bpoweski how are you doing the sharding, your own consistent hashing?

bpoweski commented 9 years ago

@shmish111 We plan to use the hashing mechanism provided in twemproxy along with pre-sharding our Redis instances. One of the biggest disadvantages to this approach is the management cost of the numerous shards and how to manage the migration to different hosts as you need more RAM. We think docker, with its ecosystem, along with consul provide much of what's needed to manage it without too much pain. Granted, this is all conjecture at this point.

shmish111 commented 9 years ago

@bpoweski I have started to look into this again recently, how did it go with twemproxy and consul? I am investigating how we can get some sort of redis cluster going that is fast and has some failover capabilities. Using Redis Cluster is one option and I would be grateful if you could release your proof of concept somewhere? Twemproxy is the other major contender I think, could you tell me why you chose it over redis cluster and if you feel you made the right decision?

bbsbb commented 8 years ago

Is there any plan on shipping this or is it looking for external help at this point?

ptaoussanis commented 8 years ago

Fully-baked PRs welcome, otherwise happy to implement if someone wants to fund my time on this. Otherwise don't need Cluster myself so unfortunately no other plans atm, sorry.

lemonteaa commented 7 years ago

Hey, just to let people know that I'm working on this in my forked repo. Since it's been so long (last updates on the order of year), with some non-trivially intertwined developments, I have first merged from the main branch into the dev branch for cluster, as well as combine codes from two different attempts. (Using rebase/cherrypick) (to take advantage of stuffs in main branch since then)

Things are actually already 70-80% completed before I started, from my first glance of the codes. I will mostly follow the plans included in the source code except for the keyslot cache data structure.

@ptaoussanis I've written a sketch of my planned design of said keyslot cache data structure in cluster.clj, could you take a look and see if it's okay? Thanks!

ptaoussanis commented 2 years ago

Closing to focus on https://github.com/ptaoussanis/carmine/issues/274