Open jwoertink opened 2 years ago
I have never tried this shard with Redis in cluster mode, and I don't know if anybody has done so, or what it would take to get it to work.
I briefly looked at how the Ruby client does it. I think what that was doing was run the command, if it fails with this "MOVED" command, then you rescue and grab the host that contains the data and run the command against that host.
So in a pseudo-code way:
cluster = Redis.new(cluster: ["redis://1.2.3.4:6379", "redis://4.5.6.7:6379"])
cluster.lrange("thing", 0, -1)
# somewhere deep inside
private def make_call
run
rescue e : RedisFailed
if e.message =~ /MOVED/
host = get_host_from_error(e)
Redis.current_host = host
make_call
end
end
I'm sure it's a lot more complicated than that, but that was the general idea I saw the ruby client doing.
I don't think it was too bad getting it to recreate locally using https://github.com/bitnami/bitnami-docker-redis-cluster. Boot up these cluster redis in Docker locally, and then you can just run LRANGE thing 0 -1
right from the redis-cli and see the same error.
@russ might be able to provide a bit more insight and a docker-compose that worked...
Here is the docker-compose.yml file I used to boot a cluster.
version: '2'
services:
redis-node-0:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-0:/bitnami/redis/data
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-1:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-1:/bitnami/redis/data
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-2:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-2:/bitnami/redis/data
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-3:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-3:/bitnami/redis/data
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-4:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-4:/bitnami/redis/data
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-5:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-5:/bitnami/redis/data
ports:
- "6379:6379"
depends_on:
- redis-node-0
- redis-node-1
- redis-node-2
- redis-node-3
- redis-node-4
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_CLUSTER_REPLICAS=1'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
- 'REDIS_CLUSTER_CREATOR=yes'
volumes:
redis-cluster_data-0:
driver: local
redis-cluster_data-1:
driver: local
redis-cluster_data-2:
driver: local
redis-cluster_data-3:
driver: local
redis-cluster_data-4:
driver: local
redis-cluster_data-5:
driver: local
Then just a simple request for data.
❯ redis-cli -h 172.27.0.1
172.27.0.1:6379> lrange thing 0 -1
(error) MOVED 14607 172.27.0.2:6379
My understanding is that the cluster responds with what node the data is actually on. So if I connect to the node given back, then I can fetch the data.
❯ redis-cli -h 172.27.0.2
172.27.0.2:6379> lrange thing 0 -1
1) "foobar"
Hat tip to @jwoertink for pointing me to this issue.
The way I implemented this in my own Redis shard was by creating an abstraction for clusters that automatically routes commands to the right servers based on the key name (inferred from the second element in the command array) and whether it's a read or write command. This is cleaner and more performant (and, in the case of Redis.current = ...
, more concurrency-safe) than rescuing exceptions and reconnecting to the new server, since that will almost certainly happen a lot.
The server that holds a given key can be derived with server_for_keyslot(crc16(key) % 16_384)
:
crc16(key)
is the CRC16 checksum algorithm
{post}:1234
, you only hash post
. The server will enforce this — check CLUSTER KEYSLOT
docs for more info.RPOPLPUSH
.server_for_keyslot(keyslot)
is implemented as:
CLUSTER NODES
The above docker-compose
config is great, but if you're not developing inside that same Docker network, you'll likely run into issues. The CLUSTER NODES
command gives you IP addresses that the Redis server can connect to, but if those are behind a NAT layer (such as a Docker network) that you won't be running your code from, you may not be able to reach them. I couldn't get it to work that way so I wrote a Ruby script to spin up a cluster to help me in developing my own shard — it spins up the number of Redis servers of each type given at the top of the script and runs them on consecutive ports starting at 6379. As written, this script spins up 3 masters and 2 replicas each, for a 9-node Redis cluster.
We have Redis on AWS in cluster mode, and it's been fine while using Cable-cr. However, we just tried to use this shard directly to store some data, and ran in to some issues:
This ends up throwing an exception. Is there a way to handle this already? Or something that will have to be added in to support?