Closed corymurphy closed 4 months ago
I think it might be related to codec/redis.rs. Should INFO
be added to this list?
Hi and thanks for the issue report!
I've reproduced the issue with the INFO command and im working on a fix.
I couldnt reproduce the issue with SADD and LPUSH, but maybe the INFO bug left shotover in a bad state such that all other commands were broken? I'll double check those commands once ive solved the INFO issue.
Thank you for the quick response! I'd be happy to assist however I can.
I was wondering if it might be related to some command arg size limit, since the LPUSH command contains a fairly large argument in the form of JSON. This is all Sidekiq behavior, so I don't have much control over how it interacts with redis. I will grab a full sanitized output and update this issue shortly.
Thanks again!
I can confirm your theory that some commands are leaving the proxy in a state where the results are returned in the subsequent command. It seems to happen particularly with more of the complex commands where lists are returned. This is how I tested.
root@cam-debug-f7d78597-7sr2c:/usr/src/app# redis-cli -u redis://redis-shotover.ns-cam-debug.svc.cluster.local:6379/0 mget cluster1 cluster12
1) "cluster1"
2) "cluster12"
root@cam-debug-f7d78597-7sr2c:/usr/src/app# redis-cli -u redis://redis-shotover.ns-cam-debug.svc.cluster.local:6379/0 info
cluster1
cluster12
0root@cam-debug-f7d78597-7sr2c:/usr/src/app# redis-cli -u redis://redis-shotover.ns-cam-debug.svc.cluster.local:6379/0 info
# Server
redis_version:5.0.3
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:standalone
os:Amazon ElastiCache
Closing this as we have since removed the TuneableConsistencyScatter transform from shotover.
Is your feature request related to a problem? Please describe.
I'm not sure if this should be a feature request or bug, since this might not be the intended use case for shotover, so please let me know if I'm not using it as intended. This project looks really promising and I like many of the features!
I'm attempting to implement a zero-downtime migration for redis, following this blog post. The concept is that you use a proxy to mirror all writes, but only read from a single instance until the two instances are at a consistent state, allowing you to transparently to the new instance without downtime.
I tried to use envoy proxy, but they don't support the MULTI redis command used for transactions. We're using redis primarily for Sidekiq, which uses this command, making it difficult to replace those commands like the blog recommends.
I've configured my Sidekiq instance to use shotover as a proxy for 2 AWS Elasticache Redis instances, which is working great for simple commands such as
GET
andSET
.I'ved tested this on
0.1.1
using the included dockerfile running on kubernetes.Describe the solution you'd like
My hope is to support commands that Sidekiq uses without timing out.
INFO
command just times out. I enabled theDebugPrinter
, which shows that the command is getting a response, but it never returns to the client. Looking at the shotover logs just hangs after printing out the results from the INFO command.Sidekiq also uses the
SADD
andLPUSH
commands on startup, both of which are timing out the same way asINFO
.Describe alternatives you've considered
I've attempted to adjust the QueryTypeFilter and the read_consistency, but i'm still getting a timeout.
Additional context