search

mediocregopher / radix.v2

Redis client for Go
http://godoc.org/github.com/mediocregopher/radix.v2
MIT License
433 stars 92 forks source link

Redis cluster and EVAL/EVALSHA #63

Closed kixelated closed 7 years ago

kixelated commented 7 years ago

Hey, we're using radix at Twitch and recently switched to redis cluster. It seemed like things were working normally but we noticed one of the redis hosts was using more network traffic but performing the same number of operations. I monitored the host and it was receiving more EVALSHA commands than the others.

The issue is that radix always selects the first argument as the key. In the case of EVALSHA, the key is actually the 3rd* argument. Radix was using the SHA of the script instead, which caused all commands to be sent to the same host. Redis cluster handled the situation by returning a redirect to the correct node so everything appeared to work anyway.

I wasn't using util.LuaEval and thought it should fix the issue. However, I took a look at the code and it looks like a typo causes the wrong behavior anyway. The code is using c instead of cc which causes the same broken hashing logic.

I've made my own version of LuaEval to work around the issue and we're seeing perfectly balanced traffic now. It's worth fixing the util and maybe adding some documentation to Cmd. It's also something to keep in mind while you're working on the API for v3.

mediocregopher commented 7 years ago

When I first read this I had a mini-heart attack cause I thought maybe LuaEval on cluster has been putting data on the wrong nodes since 2015! But cluster seems to handle the redirect like it handled it when you were doing the EVALSHA directly, so we're good there. Just pushed up that fix.

Incidentally, since you mentioned it, the v3 branch is being actively worked on (like, I pushed to it this morning) and is like 90% "done", most of what's left are some TODOs and lotsa docs. I'm really pushing myself to get that out the door next weekend, I'll make a new repo and link to it from this one's README. But in the meantime if you have any feedback you want to give on v2 or what's done of v3 I'd love to hear it.