search

twitter / twemproxy

A fast, light-weight proxy for memcached and redis
Apache License 2.0
12.13k stars 2.06k forks source link

Twemproxy " Connection refused " in logs on failure of one redis instance #530

Closed mishtika closed 3 years ago

mishtika commented 7 years ago

I have configured twem proxy with 2 redis servers. When one of these redis server fails the twem proxy gives a error saying " Connection refused"

My twem conf beta: listen: 127.0.0.1:22122 hash: fnv1a_64 hash_tag: "{}" distribution: ketama auto_eject_hosts: true timeout: 400 redis: true servers:

i am trying to get a key from redis. If i kill one instance of redis and then try to run the get command then it gives the following error: [2017-07-26 16:25:24.548] nc_core.c:237 close s 15 '127.0.0.1:6381' on event FFFFFF eof 0 done 0 rb 0 sb 0: Connection refused

therealbill commented 7 years ago

Twemproxy is a key distributing proxy. When you have two servers half the keys (roughly) go to one, the other to the other. When one of those is down you don't have access to those keys. If you are looking for master/slave availability you'll need to use Redis' native replication to handle that rather than Twemproxy.

tuananh commented 7 years ago

@therealbill shouldn't i get a cache miss for that case instead of connection refused?

jslusher commented 7 years ago

I think my issue is related to this. I'm proxying memcached, but I have the same problem where if one of the servers goes down the proxy doesn't use the second server: https://github.com/twitter/twemproxy/issues/534

niuchenglei commented 6 years ago

i think my issue is that, for example: we got key a,b,c,d and a,b on redis server No.1, c,b on redis server No.2, and we request key a,c with twemproxy, it will request nothing when No.1 has down.

chrizkiran commented 6 years ago

Even am facing the same issue. I get the below error when one of the Redis instance is down.

[2018-06-09 20:02:17.672] nc_request.c:96 req 91 done on c 14 req_time 0.562 msec type REQ_REDIS_HMGET narg 3 req_len 53 rsp_len 34 key0 'mqtt_user:client1' peer '127.0.0.1:33311' done 1 error 0 [2018-06-09 20:02:17.681] nc_request.c:96 req 93 done on c 13 req_time 0.163 msec type REQ_REDIS_HMGET narg 3 req_len 53 rsp_len 34 key0 'mqtt_user:client1' peer '127.0.0.1:47821' done 1 error 0 [2018-06-09 20:02:20.660] nc_response.c:118 s 15 active 0 is done [2018-06-09 20:02:20.660] nc_core.c:237 close s 15 '127.0.0.1:6379' on event 00FF eof 1 done 1 rb 68 sb 106

I have configured Redis Sentinel . Am I misunderstanding anything with respect to twemproxy??

Any how after some frequent connect requests, I get the response back.

TysonAndre commented 3 years ago

If i kill one instance of redis and then try to run the get command then it gives the following error: [2017-07-26 16:25:24.548] nc_core.c:237 close s 15 '127.0.0.1:6381' on event FFFFFF eof 0 done 0 rb 0 sb 0: Connection refused

That syslog is hard to read but expected - the connection to the server closed, and twemproxy attempted to reconnect to that server(s) when you sent another command and got a Connection refused error because nothing listened on that port

Aside: https://github.com/twitter/twemproxy/issues/608 will significantly refactor the way reconnection is implemented in a future 0.6.0 release (heartbeat, failover).

I have configured Redis Sentinel . Am I misunderstanding anything with respect to twemproxy??

Redis sentinel support hasn't been merged in from a fork yet, but is planned as part of https://github.com/twitter/twemproxy/issues/608 for 0.6.0 - even when the sentinel client implementation is merged in you'd have to configure the list of sentinels.

Closing this because the original question was answered