Voldemort Client Uses Stale Connections After Server Restart

[GoogleCodeExporter]

After a restart of the Voldemort server what I observe is that the client can 
have multiple stale connections to the server.  Each request to the server 
fails until each of these connections fails once and is removed from the pool.  
This can lead to multiple failures (Exceptions) after a server restart.

To reproduce this I test as followd:

1) Client and server in service.
2) Run traffic and monitor Voldemort logs to see how many connections are setup.
3) Restart the server.
4) Make Voldemort requests one at a time and monitor the number of failures.

What I see is that if I had 6 connections established before the server restart 
then the first 6 requests after the restart will fail.  The 7th request will be 
OK because the pool is now empty and a new connection is established.

What I would expect is that the client would handle this issue on the first 
request.  Specifically that it would loop through the connections one at a time 
and if they all fail allocate a new one like it does when the pool is empty.

I am using 90.1.

Perhaps there is a connection setting that will address this issue.  I also 
realize that if I have a 3 server configuration and only require 1 read or 
write then the failure is masked, but, if I restart two or three of the servers 
I am in a similar situation.

Original issue reported on code.google.com by jwl...@gmail.com on 1 Feb 2013 at 2:03

[GoogleCodeExporter]

I started an email thread on this issue.  It includes some additional 
information:

https://groups.google.com/forum/?fromgroups=#!topic/project-voldemort/lZM3DgWuvJ
Q

Original comment by jwl...@gmail.com on 5 Feb 2013 at 9:19