search

leeoo / spymemcached

Automatically exported from code.google.com/p/spymemcached
0 stars 0 forks source link

DNS refresh in MemcachedConnection.attemptReconnects() #245

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What version of the product are you using? On what operating system?
2.7.3, OS X & RedHad

Tell me more...

This is a pain in the butt to test, as it depends on Amazon topology. :-(

Let me tell you a story.

To start with, I have two memcached nodes on two different hosts with the 
following IPs:
mc01 -> 99.88.77.1
mc02 -> 99.88.77.2

I start up my application pointing to these two hosts, configured as 
"mc01:11211 mc02:11211".

(I believe at this point the two InetSocketAddresses are created, resolving to 
99.88.77.1:11211 and 99.88.77.2:11211 respectively.)

Things are going along smoothly for a while (days, maybe) when all of a sudden 
the crappy box at Amazon fails. Great! The spymemcached client recognizes that 
no answer is coming back from 99.88.77.2, and sends all the memcached traffic 
to 99.88.77.2. 

Our Ops team is notified that box shit the bed, so they bring up a new box (at 
the IP provided by Amazon) at 99.88.77.42, and entering the DNS entry that maps 
mc02 to it.

So, now our setup looks like this:
mc01 -> 99.88.77.1
mc02 -> 99.88.77.42

But, (I think) the spymemcached client still thinks the setup looks like this 
(because the InetSocketAddress holds a local caching of the DNS resolution?):
mc01 -> 99.88.77.1
mc02 -> 99.88.77.2

Therefore, the client never fully recovers.

************

Suggested resolution:
In MemcachedConnection.attemptReconnects(), before you actually try to connect 
via the InetSocketAddress do one of two things:
1 - Give InetSocketAddress whatever stimulus it needs to re-resolve the DNS 
name. (Not sure this is capable in the API?)
2 - Re-create the InetSocketAddress for the MemcachedNode. (Again, I'm not sure 
what housekeeping would be necessary if this happens.)

Original issue reported on code.google.com by nelz9999@gmail.com on 18 Apr 2012 at 6:33

GoogleCodeExporter commented 9 years ago 
D'oh! Typo, should read:
"... and sends all the memcached traffic to 99.88.77.1."

Original comment by nelz9999@gmail.com on 18 Apr 2012 at 6:35

GoogleCodeExporter commented 9 years ago 
Which version of the JDK is in use?

The 'net tells me that this is a function of having a security manager defined 
(a cool thing in Java, that's not often understood), and that it probably makes 
sense to redefine it on the platform.  See:
http://docs.oracle.com/javase/6/docs/technotes/guides/net/properties.html
http://java-monitor.com/forum/showthread.php?t=181
http://javaeesupportpatterns.blogspot.com/2011/03/java-dns-cache-reference-guide
.html

p.s.: I've run into this before, but never really bothered to read up on it.

Original comment by ingen...@gmail.com on 19 Apr 2012 at 1:16

GoogleCodeExporter commented 9 years ago 
This is definitely at the outside of my Java networking experience, so please 
pardon me if I'm making incorrect assumptions.

From my reading of the docs on InetSocketAddress, doesn't the DNS resolution 
happen at *instantiation* time of the object, which is then kept immutable over 
the life of that InetSocketAddress instance.

I believe that this means no DNS change (even if it did happen in the 
underlying system's security management) can get picked up by SpyMemecached 
unless you try to instantiate the InetSocketAddress again.

Original comment by nelz9999@gmail.com on 19 Apr 2012 at 5:39

GoogleCodeExporter commented 9 years ago 
From the source code I could find, it seems that my assertion is bourne out: 
the resolution is attempted at instantiation time of the InetSocketAddress.

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/jav
a/net/InetSocketAddress.java#InetSocketAddress.%3Cinit%3E%28java.lang.String%2Ci
nt%29

Original comment by nelz9999@gmail.com on 23 Apr 2012 at 5:37

GoogleCodeExporter commented 9 years ago 
Ah, okay.  I hadn't considered that part of it.  We'll have to just look at 
recreating that InetSocketAddress then.

Original comment by ingen...@gmail.com on 23 Apr 2012 at 5:42

GoogleCodeExporter commented 9 years ago 
I took a look at possibly fixing this myself today. It doesn't look easy. :-( 

Basically, the MemcachedNode would need to hold an un-resolved version of the 
hostname, to be resolved at connection time. Either that or a new MemcachedNode 
would need to be instantiated at connection time, and I have no idea what havoc 
that might wreak throughout the rest of the library.

Original comment by nelz9999@gmail.com on 24 Dec 2012 at 10:59

GoogleCodeExporter commented 9 years ago 
I ran into this issue this week and spent sometime experimenting.

I agree with nelz9999@gmail.com's observation that InetSocketAddress is 
immutable even if the JVM is correctly resolving DNS changes.

Original comment by t...@eelpieconsulting.co.uk on 13 Oct 2014 at 1:30