slimina / xmemcached

Automatically exported from code.google.com/p/xmemcached
Apache License 2.0
0 stars 0 forks source link

StackOverflowError caused by sendIncrOrDecrCommand infinitely retrying a key that exists but doesn't contain a number #277

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
What steps will reproduce the problem?
1. At random times I've been finding this exception in my logs

Caused by: java.lang.StackOverflowError
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544)
        at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240)
        at java.lang.StringCoding.encode(StringCoding.java:272)
        at java.lang.String.getBytes(String.java:946)
        at net.rubyeye.xmemcached.utils.ByteUtils.getBytes(ByteUtils.java:53)
        at net.rubyeye.xmemcached.HashAlgorithm.computeMd5(HashAlgorithm.java:214)
        at net.rubyeye.xmemcached.HashAlgorithm.hash(HashAlgorithm.java:138)
        at net.rubyeye.xmemcached.impl.KetamaMemcachedSessionLocator.getSessionByKey(KetamaMemcachedSessionLocator.java:151)
        at net.rubyeye.xmemcached.impl.MemcachedConnector.findSessionByKey(MemcachedConnector.java:557)
        at net.rubyeye.xmemcached.impl.MemcachedConnector.send(MemcachedConnector.java:477)
        at net.rubyeye.xmemcached.XMemcachedClient.sendCommand(XMemcachedClient.java:265)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2466)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2482)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2482)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2482)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2482)
        at net.rubyeye.xmemcached.XMemcachedClient.sendIncrOrDecrCommand(XMemcachedClient.java:2482)
... [21 pages of this same sendIncrOrDecrCommand(XMemcachedClient.java:2482)  ]

It does not seem normal that sendIncrOrDecrCommand(XMemcachedClient.java:2482) 
gets called 1000's of times until the stack overflows, does it? 

2. This issue happens because at some point my code is requesting to incr/decr 
a key that contains a string rather than a number counter, here is the source 
code of class XMemcachedClient:

(in reality this starts in line 2458)
1   private long sendIncrOrDecrCommand(final String key, final long delta,
2           long initValue, final CommandType cmdType, boolean noreply,
3           long operationTimeout, int exp) throws InterruptedException,
4           TimeoutException, MemcachedException {
5       final byte[] keyBytes = ByteUtils.getBytes(key);
6       ByteUtils.checkKey(keyBytes);
7       final Command command = this.commandFactory.createIncrDecrCommand(key,
8               keyBytes, delta, initValue, exp, cmdType, noreply);
9       this.sendCommand(command);
10      if (!command.isNoreply()) {
11          this.latchWait(command, operationTimeout);
12          command.getIoBuffer().free();
13          this.checkException(command);
14          if (command.getResult() == null) {
15              throw new MemcachedException(
16                      "Operation fail,may be caused by networking or timeout");
17          }
18          final Object result = command.getResult();
19          if (result instanceof String) {
20              if (((String) result).equals("NOT_FOUND")) {
21                  if (this.add(key, exp, String.valueOf(initValue),
22                          this.opTimeout)) {
23                      return initValue;
24                  } else {
25                      return this.sendIncrOrDecrCommand(key, delta,
26                              initValue, cmdType, noreply, operationTimeout,
27                              exp);
28                  }
29              } else {
30                  throw new MemcachedException(
31                          "Unknown result type for incr/decr:"
32                                  + result.getClass() + ",result=" + result);
33              }
34          } else {
35              return (Long) command.getResult();
36          }
37      } else {
38          return -1;
39      }
40  }

3.  notice that in line 25 after the command returned successfully something 
that is a string and is not "NOT_FOUND" it retries the same operation again. 
Why try the same operation that just returned the wrong value?

What is the expected output? What do you see instead?
This operation should fail throwing an exception or return null rather than 
re-trying. This implementation doesn't try to "fix" the value in Memcached and 
there's no assurance anybody else will which eventually causes the app to run 
into the stack overflow issues I'm experiencing.

What version of the product are you using? On what operating system?
Xmemcached client V 1.3.8 
Ubuntu 12.04.2 LTS

Please provide any additional information below.

Original issue reported on code.google.com by claudio....@gmail.com on 17 Dec 2013 at 9:42

GoogleCodeExporter commented 9 years ago
The incr or decr commands suppose the item's value to be a number,so if it find 
the item is not exists, it will try to add it with initial value, if it add 
fail,it will try to incr/decr it again.

So it seems that in your system, the xmemcached try to add the item failed,then 
try to incr/decr the exists value,but the value is not a number(who added it?).

Original comment by xzhu...@avos.com on 18 Dec 2013 at 1:00

GoogleCodeExporter commented 9 years ago
I guess I´ll have to change my own implementation to log the key giving me a 
hard time because when this incr/decr operation is retried and eventually 
fails, stack trace doesn´t show the line in my code where this code is being 
called.

Original comment by claudio....@gmail.com on 20 Dec 2013 at 3:33