Put and Get are not consistent

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Deploy to 2 remote memcache servers.
1. Write a program to put objects (preferably in ten thousands atleast)in 
Memcached servers by different keys.
2. Write another program to fetch the same set of keys from the Memcached 
servers. If the key is not found put in another object for the same key.
3. All the gets are not fetching the object. Instead I am seeing duplicate key 
in both servers.

What is the expected output? What do you see instead?
The get should be from the same server where put is done. Instead it is trying 
to get the object from the other server and thus duplicate key is generated.

What version of the product are you using? On what operating system?
xmemcached-1.3.2 to xmemcached-1.3.5. Memcached servers running on Red Hat 
Linux systems. And the programs to put and fetch again on separate Red Hat 
Linux Servers.

Please provide any additional information below.

System setup:
============================================================
2 Core 6 GB Red Hat linux machine (Memcached server 1)
2 Core 6 GB Red Hat linux machine (Memcached server 2)
4 Core 6 GB Red Hat linux machine (JVM for putting objects)
4 Core 6 GB Red Hat linux machine (JVM for fetching objects & putting the 
objects by the same key if unable to fetch)

Systems connected by 100Mbps Ethernet

Original issue reported on code.google.com by dhruv....@gmail.com on 12 Nov 2011 at 6:26

[GoogleCodeExporter]

Are u sure the memcached server list is in the same order for two programs?And 
used the same session locator?

Original comment by killme2...@gmail.com on 12 Nov 2011 at 6:39

[GoogleCodeExporter]

Futher feedback?

Original comment by killme2...@gmail.com on 12 Nov 2011 at 8:39

[GoogleCodeExporter]

Sorry for late reply. Yes, the server list is in the same order for both the 
programs. And both are using the same session locator.

The code base for putting and fetching is the same. I had used the 
addServer(java.lang.String hostList) method with the address and port numbers 
for 2 memcached servers.

Original comment by dhruv....@gmail.com on 12 Nov 2011 at 7:21

[GoogleCodeExporter]

May be you can attach a test code to reproduce this problem?
I have tried as you said above to reproduce this probem,but it can not occured 
at all.

Original comment by killme2...@gmail.com on 13 Nov 2011 at 12:55

[GoogleCodeExporter]

And please check the log just like:
 Add a session: 127.0.0.1:12000
 Add a session: 127.0.0.1:12001

And see if the "add session" log in two machines have the same order?

Original comment by killme2...@gmail.com on 13 Nov 2011 at 1:11

[GoogleCodeExporter]

Sorry I will not be posible for me to upload the code now. I will do it as soon 
as possible.

Additional details that may help:
===================================
1. The program to fill data on memcached servers is a multithreaded one. I have 
used a connection pool size of 50 for xmemcached. Each of these threads does a 
setWithNoReply(...)
2. The logs for the memcache filling programs show:

14 Nov 2011 13:33:42,255 WARN main net.rubyeye.xmemcached.XMemcachedClient - 
XMemcachedClient use Binary protocol
14 Nov 2011 13:33:42,295 INFO main 
com.google.code.yanf4j.nio.impl.SelectorManager - Creating 8 reactors...
14 Nov 2011 13:33:42,317 WARN main 
com.google.code.yanf4j.core.impl.AbstractController - The Controller started at 
localhost/127.0.0.1:0 ...
14 Nov 2011 13:33:42,337 WARN Xmemcached-Reactor-0 
com.google.code.yanf4j.core.impl.AbstractController - Add a session: 
172.17.222.216:11211
14 Nov 2011 13:33:42,340 WARN Xmemcached-Reactor-0 
com.google.code.yanf4j.core.impl.AbstractController - Add a session: 
172.17.222.217:11211
14 Nov 2011 13:34:02,077 WARN pool-2-thread-6-entityLevelThread 
net.rubyeye.xmemcached.XMemcachedClient - XMemcachedClient use Binary protocol
14 Nov 2011 13:34:02,078 INFO pool-2-thread-6-entityLevelThread 
com.google.code.yanf4j.nio.impl.SelectorManager - Creating 8 reactors...
14 Nov 2011 13:34:02,098 WARN pool-2-thread-6-entityLevelThread 
com.google.code.yanf4j.core.impl.AbstractController - The Controller started at 
localhost/127.0.0.1:0 ...
14 Nov 2011 13:34:02,238 WARN Xmemcached-Reactor-0 
com.google.code.yanf4j.core.impl.AbstractController - Add a session: 
172.17.222.216:11211    (Total 50 times)
.
.
.
14 Nov 2011 13:34:28,457 WARN Xmemcached-Reactor-0 
com.google.code.yanf4j.core.impl.AbstractController - Add a session: 
172.17.222.217:11211    (Total 50 times)
.
.
.

3. The logs for the program that fetches data from memcached servers:
2011-11-14 13:50:13,980 WARN  [net.rubyeye.xmemcached.XMemcachedClient] 
XMemcachedClient use Binary protocol
2011-11-14 13:50:14,034 INFO  [com.google.code.yanf4j.nio.impl.SelectorManager] 
Creating 4 reactors...
2011-11-14 13:50:14,051 WARN  
[com.google.code.yanf4j.core.impl.AbstractController] The Controller started at 
localhost/127.0.0.1:0 ...
2011-11-14 13:50:34,088 WARN  
[com.google.code.yanf4j.core.impl.AbstractController] Add a session: 
172.17.222.216:11211    (Total 50 times)
.
.
.
2011-11-14 13:50:34,490 WARN  
[com.google.code.yanf4j.core.impl.AbstractController] Add a session: 
172.17.222.217:11211    (Total 50 times)
.
.
.

Original comment by dhruv....@gmail.com on 14 Nov 2011 at 8:10

[GoogleCodeExporter]

Will it be possible to have a DEBUG level log or perhaps a TRACE level log to 
know which key has been put on which server. Sorry if it already exists. I have 
not gone through the source code.

Original comment by dhruv....@gmail.com on 14 Nov 2011 at 8:14

[GoogleCodeExporter]

Are u sure that setting data was began after the client built successfully?

What's you configuration for xmemcached client?

Original comment by killme2...@gmail.com on 14 Nov 2011 at 8:29

[GoogleCodeExporter]

And you can upload a demo code to github or googlecode,thanks a lot.

Original comment by killme2...@gmail.com on 14 Nov 2011 at 8:32

[GoogleCodeExporter]

hi,can you upload the demo code now? It's an important issue,i tried to 
reproduce it on my machine,but i can't.

So Plase  attach the test code ,thanks a lot.

Original comment by killme2...@gmail.com on 16 Nov 2011 at 2:03

[GoogleCodeExporter]

Hi, I have uploaded some sample code in the following path:
http://code.google.com/p/framework-cache/downloads/list

There you can download the following files:
1) TestXmemcached.java - This is the base class where all the configuration is 
set and the client is built.
2) MyMemcachedStateListener.java - This is an implementation of State Listener.
3) CacheInitialization.java and CacheThreadingHelper.java - These are classes 
that spawn multiple threads to fill up the memcached server with data from DB 
using the TestXmemcached class.

The code is not very well documented and structured. Please let me know if you 
need any explanation. Thanks a lot.

Original comment by dhruv....@gmail.com on 19 Nov 2011 at 4:29

[GoogleCodeExporter]

Please look the initialize() method in TestXmemcached.java class where all 
configurations are set. Please let me know if there are any issues.

The command I used to run memcached server is:
memcached -p 11211 -m 2500m -c 1024 -d -v

Original comment by dhruv....@gmail.com on 19 Nov 2011 at 4:33

[GoogleCodeExporter]

Also please let me know if I am correct in my following assumptions:
1) Any key will always land up in the same server if both of the servers are 
live.
2) The determination of where an object will land up is based on the key solely 
and has nothing to do with the value.
2) The algorithm used in the above process is (hashFunction(KEY)%noOfServers)

Original comment by dhruv....@gmail.com on 19 Nov 2011 at 4:40

[GoogleCodeExporter]

1. Write a program to put objects (preferably in ten thousands atleast)in 
Memcached servers by different keys.
2. Write another program to fetch the same set of keys from the Memcached 
servers. If the key is not found put in another object for the same key.

Is the second program ran after the first one? or these programs ran 
concurrently?And you must sure you don't flush memcached before ran the second 
program.

I read your code,but your code has many your application classes i don't 
know.And i write a simple program for this issue,and ran it twice to check it 
the key is duplicate,first time set printIfKeyNotExists=false ,and second time 
set printIfKeyNotExists=true to see if there are some duplicate keys.could you 
check my code is the same logic as your program?
With this program,i can not reproduce this problem.The program is attached.

Original comment by killme2...@gmail.com on 20 Nov 2011 at 1:45

Attachments:

• SimpleExample.java

[GoogleCodeExporter]

1) Any key will always land up in the same server if both of the servers are 
live.
Yes.
2) The determination of where an object will land up is based on the key solely 
and has nothing to do with the value.
Yes.
2) The algorithm used in the above process is (hashFunction(KEY)%noOfServers)
No.you use ketama consist hash locator instead of ArrayMemcachedSessionLocator 
by default.

Original comment by killme2...@gmail.com on 20 Nov 2011 at 1:53

[GoogleCodeExporter]

Yes your program's logic seems to be similar.... But the problem is something 
very specific....

A little update on what I did today... If you find any clues to the problem:
The setup (Hardware):
=======================================
A) 172.17.222.216:11211 Memcached server 1 (HOST name: SA01)
B) 172.17.222.217:11211 Memcached server 2 (HOST name: SA02)
C) 172.17.218.192
=======================================
1) I ran a simple utility program from machine A (i.e. 172.17.222.216) that 
puts in a set of specific keys that I had found problematic to memcached 
servers on both machine A and B. They are: (The text in bracket indicates where 
the key was actually put)
claid-cladcnid-39-30-1300061-1 (Put on 216)
claid-cladcnid-39-30-1300062-1 (Put on 217)
claid-cladcnid-39-30-1300063-1 (Put on 217)
claid-cladcnid-39-30-1300065-1 (Put on 217)
claid-cladcnid-39-30-1300713-1 (Put on 216)
claid-cladcnid-39-30-1300760-1 (Put on 216)
claid-cladcnid-39-30-1300762-1 (Put on 216)
claid-cladcnid-39-30-1300810-1 (Put on 217)

Now if I run another program to fetch the same set of keys from machine A, it 
is able to fetch all the sets of keys successfully.

If I run the program to fetch keys from machine B it is not able to find any of 
the keys at all.

Again, If I run the same program to fetch the keys from machine C the behaviour 
is again different.

MY INFERENCE: NO MATTER WHERE I RUN PROGRAMS, IF THE PROGRAM TO PUT AND THE 
PROGRAM TO FETCH ARE RUN ON THE SAME SERVERS, IT ALWAYS IS ABLE TO FIND THE 
CORRECT KEYS.

ONLY VISUAL DIFFERENCE NOTICED BY ME: When I run the programs from machine A. 
The logs that appear on the built of client due to client listener 
implementation (MyMemcachedStateListener.java) are:
Connected to SA01.11211 (50 times)
Connected to 172.17.222.217.11211 (50 times)

If I run from machine B:
Connected to 172.17.222.216.11211 (50 times)
Connected to SA02.11211 (50 times)

If I run from machine C:
Connected to 172.17.222.216.11211 (50 times)
Connected to 172.17.222.217.11211 (50 times)

This led me another question. Is there any relation as to how the hostnames are 
resolved in the individual machines and the algorithm by which xmemcache 
calculates which server to put data on?

Original comment by dhruv....@gmail.com on 20 Nov 2011 at 5:26

[GoogleCodeExporter]

I got you.Yes ,this is the problem. You machines resolve hostnames differently 
in each machine,it changed the consist hash session locator's finding map which 
used to find session by the item key.Because KetamaMemcachedSessionLocator 
build the session map for finding by the sesson's remote address(just use 
toString()) and weight.So in the different machines,the map is different,and 
the result session for the key is different.

Original comment by killme2...@gmail.com on 20 Nov 2011 at 11:02

[GoogleCodeExporter]

To solve this problem,you can try to configure your machines resolving 
hostnames with the same result in each machine.
Or you can try new KetamaMemcachedSessionLocator with a boolean parameter to 
use the consist hash compatible with nginx-upstream-consistent:

    new KetamaMemcachedSessionLocator(true);

And when nginx-upstream-consistent is set to true,then building the finding map 
will use session's raw ip and port insteadof toString().

Original comment by killme2...@gmail.com on 20 Nov 2011 at 11:06

[GoogleCodeExporter]

Thanks a lot for your help. The problem is resolved.

Anyways is there any advantage of building session map by finding the sessions 
remote address rather than sessions's raw ip and port? I think implementing 
this by default would be more helpful as raw IP ports will remain same from 
every machine but the hostnames might differ if not configured correctly.

Original comment by dhruv....@gmail.com on 21 Nov 2011 at 4:40

[GoogleCodeExporter]

--Anyways is there any advantage of building session map by finding the 
sessions remote address rather than sessions's raw ip and port? I
The answer is no.But xmemcached's consist hash implementation is the same with 
spymemcached,it just wanted to be compatible with spy.And now there are many 
projects using xmemcached,i can't changed the behavior at all.As you see,the 
Compatibility is the most important thing for a client library.

Original comment by killme2...@gmail.com on 22 Nov 2011 at 12:16

[GoogleCodeExporter]

I closed this issue,and added a FAQ section to wiki for this problem.
Thanks a lot.

Original comment by killme2...@gmail.com on 22 Nov 2011 at 5:39

• Changed state: WontFix