search

carlos8f / haredis

High-availability redis in Node.js.
https://npmjs.org/package/haredis
154 stars 21 forks source link

haredis on EC2 #18

Open sdarwin opened 11 years ago

sdarwin commented 11 years ago

Hi,

We have an haredis cluster.

I shut down one server to resize it. It comes back up with a different IP address. I fix DNS.

the haredis client shows this error:

2013-08-01 07:20:39.120 - ^[[31merror^[[39m: server.js:120:Redis Error: Error: Redis connection to 10.145.157.144:6380 failed - connect ETIMEDOUT

However, that's the wrong IP. that's the obsolete IP. even after some time has passed.

there are two questions:

  1. what should I do now, today? it's a production system.
  2. is this a bug? should haredis keep track of DNS name instead of IP?

thanks.

carlos8f commented 11 years ago

haredis does a dns lookup to translate hostnames to ips, mainly so if you specify a hostname in one conf and an IP in another, the client doesn't see them as two separate servers. To avoid the lookup, you could use all IPs in your confs. I also could make an option to turn off the dns resolution if that helps your case.

Sorry for the delay, hope you were able to sort it out.

sdarwin commented 11 years ago

It does seem like there should be an option to use and store dns names, and not resolve them to IP's.

However, when implementing it, please consider the circumstances where the change is not seen by all clients instantaneously. Client A has temporarily lost connectivity to Server X due to the IP change, while Client B has done a DNS lookup and resolved the IP for Server X and is able to connect to the server. So there is this interim situation with some clients connecting and some clients not yet connecting. I don't know how haredis deals with that. If it can be handled gracefully, then using DNS names rather than IP's allows better for adjustments to the cluster.

SamBrabble commented 11 years ago

just got hit by this again. we re-IPed some servers. the haredis clients crapped out. would like to use DNS names instead.

sheldonh commented 10 years ago

And support for SRV records would be awesome. :-)