search

bigdata4u / spymemcached

Automatically exported from code.google.com/p/spymemcached
0 stars 0 forks source link

ClassCastExceptions using binary protocol #96

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What version of the product are you using? On what operating system?

2.4rc1 on linux 2.6.17

Note: we're working on upgrading to 2.4.1, but are having some issues with our 
memcached-related unit tests.

Tell me more...

Under load testing, we've been seeing large numbers of the exceptions below.

INFO  net.spy.memcached.MemcachedConnection - Reconnecting due to exception on 
{QA sa=xxx/10.1.24.39:9101, #Rops=2, 
#Wops=1, #iq=0, 
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@3df9807
d, topWop=net.spy.memcached.protocol.binary.GetOperationImpl@4762ae54, 
toWrite=0, interested=4}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be ca
st to net.spy.memcached.ops.GetOperation$Callback\n at 
net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)\n  
     at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.ad
dOperation(OptimizedGetImpl.java:28)\n  at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)\n        at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl
.optimizeGets(BinaryMemcachedNodeImpl.java:44)\n    at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)\n        at 
net.spy.memcached.protocol.TC
PMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:140)\n     at 
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:334)
\n   at 
net.spy.memcached.MemcachedConnec
tion.handleIO(MemcachedConnection.java:307)\n       at 
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:191)\n  
     at 
net.spy.memcached.MemcachedClient.run(MemcachedClient
.java:1453)"

INFO  net.spy.memcached.MemcachedConnection - Reconnecting due to exception on 
{QA sa=xxx/10.1.24.39:9101, #Rops=2, 
#Wops=2, #iq=0, 
topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@5e8fd
f44, topWop=net.spy.memcached.protocol.binary.GetOperationImpl@378b19dc, 
toWrite=0, interested=4}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be
cast to net.spy.memcached.ops.GetOperation$Callback\n       at 
net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)\n  
     at 
net.spy.memcached.protocol.binary.OptimizedGe
tImpl.addOperation(OptimizedGetImpl.java:28)\n  at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)\n        at 
net.spy.memcached.protocol.binary.BinaryMemcached
NodeImpl.optimizeGets(BinaryMemcachedNodeImpl.java:44)\n    at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)\n        at 
net.spy.memcached.pro
tocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:140)\n     
at 
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:338)
\n   at net.spy.memcached.Memcach
edConnection.handleIO(MemcachedConnection.java:307)\n       at 
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:191)\n  
     at 
net.spy.memcached.MemcachedClient.run(Memcach
edClient.java:1453)"

INFO  net.spy.memcached.MemcachedConnection - Reconnecting due to exception on 
{QA sa=xxx/10.1.24.39:9101, #Rops=2, 
#Wops=1, #iq=0, 
topRop=net.spy.memcached.protocol.binary.OptimizedSetImpl@7e9ab36
7, topWop=net.spy.memcached.protocol.binary.GetOperationImpl@5e787020, 
toWrite=0, interested=4}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be ca
st to net.spy.memcached.ops.GetOperation$Callback\n at 
net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)\n  
     at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.ad
dOperation(OptimizedGetImpl.java:28)\n  at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)\n        at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl
.optimizeGets(BinaryMemcachedNodeImpl.java:44)\n    at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)\n        at 
net.spy.memcached.protocol.TC
PMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:140)\n     at 
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:338)
\n   at 
net.spy.memcached.MemcachedConnec
tion.handleIO(MemcachedConnection.java:307)\n       at 
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:191)\n  
     at 
net.spy.memcached.MemcachedClient.run(MemcachedClient
.java:1453)"

Original issue reported on code.google.com by x%xtophe...@gtempaccount.com on 21 Oct 2009 at 4:06

GoogleCodeExporter commented 8 years ago 
We both upgraded to 2.4.2 and disabled optimization in the connection factory, 
and haven't seen this problem 
again.  We'll try reenabling optimization w/ 2.4.2 when we get a chance.

Original comment by x%xtophe...@gtempaccount.com on 30 Oct 2009 at 2:47

GoogleCodeExporter commented 8 years ago 
I don't easily see how it would happen and you say you're not having it.

If you can reproduce it in a test for me, that'd be great.

Original comment by dsalli...@gmail.com on 11 Nov 2009 at 2:11

GoogleCodeExporter commented 8 years ago 
I am having the same problem.
If you are still investigating the issue i'll be glad to reproduce the 
exception case.

Original comment by ilkinulas on 19 Nov 2009 at 12:54

GoogleCodeExporter commented 8 years ago 
Thanks, please do.  I haven't had a chance to try re-enabling optimization yet.

Original comment by x%xtophe...@gtempaccount.com on 19 Nov 2009 at 1:05

GoogleCodeExporter commented 8 years ago 
I've upgraded to version 2.4.2 still having classCastException but this time
exception is caught while casting MultiGetsOperationCallback to 
GetOperation$Callback

version 2.3.1
==============================
ava.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be cast 
to
net.spy.memcached.ops.GetOperation$Callback
        at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl
.java:28)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)
        at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:31)
        at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNode
Impl.java:141)
        at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:335)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:308)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:192)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1444)

version 2.4.2
==============================
java.lang.ClassCastException: net.spy.memcached.ops.MultiGetsOperationCallback 
cannot
be cast to net.spy.memcached.ops.GetOperation$Callback
    at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
    at
net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl
.java:28)
    at net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:21)
    at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMem
cachedNodeImpl.java:44)
    at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)
    at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNode
Impl.java:157)
    at net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:336)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:299)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:193)
    at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1458)

Original comment by ilkinulas on 19 Nov 2009 at 2:18

GoogleCodeExporter commented 8 years ago 
I'd love a test case if you guys can provide one.

Original comment by dsalli...@gmail.com on 19 Nov 2009 at 11:11

GoogleCodeExporter commented 8 years ago 
java version "1.6.0_14"
OS : Ubuntu-9.04
memcached server 1.4.0
I've attached a test. I got ClassCastException when i run the test. 
using : 
*  BinaryConnectionFactory
*  gets, cas and add operations
can you please check.

Original comment by ilkinulas on 20 Nov 2009 at 7:29

Attachments:

GoogleCodeExporter commented 8 years ago 
Thanks for the test.  I'll see if Eric can make sense of it.

Original comment by dsalli...@gmail.com on 20 Nov 2009 at 11:53

GoogleCodeExporter commented 8 years ago 
Any progress in this issue?

Original comment by ilkinulas on 24 Nov 2009 at 11:54

GoogleCodeExporter commented 8 years ago 
I've asked Eric to look at it, but it's kind of a holiday week so I don't 
expect it
to be fixed this week.  It does appear that there's enough information captured 
for
us to do something useful with it, though.

Original comment by dsalli...@gmail.com on 24 Nov 2009 at 5:06

GoogleCodeExporter commented 8 years ago 
Have also run into this issue.

Running on RHEL 4 (Update 8), WebSphere 6.1 SR9 x86_64 Java 1.5 
pxa64devifx-20071025
(SR6b) with your client version 2.5RC1.

Have switched to DefaultConnectionFactory until resolved.

This issue is rather strange.  It didn't happen in our dev or test 
environments; it
only happened on our load testing environment and before we started the load 
testing.

It seems very likely to be a class loading issue but I can't see why.  We do run
multiple instances of our application in the same JVM, but we have parent_last 
set so
there should be no issues from that.  Our dev, test and load test environments 
are
all configured the same.  A real mystery then why only one environment is 
affected.

2010-01-21 16:05:49,889 ERROR [McConnectionObserver] ConnectionLost to
aklia782/10.16.160.6:11213
2010-01-21 16:05:51,890 INFO  [MemcachedConnection] Reconnecting {QA
sa=aklia782/10.16.160.6:11213, #Rops=1, #Wops=30843, #iq=0, topRop=null,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@29912991, toWrite=0,
interested=0}
2010-01-21 16:05:51,891 INFO  [MemcachedConnection] Connection state changed for
sun.nio.ch.SelectionKeyImpl@64946494
2010-01-21 16:05:52,893 INFO  [McConnectionObserver] ConnectionEstablished to
aklia782/10.16.160.6:11213, retries 1, version is null
2010-01-21 16:05:52,894 INFO  [MemcachedConnection] Reconnecting due to 
exception on
{QA sa=aklia782/10.16.160.6:11213, #Rops=2, #Wops=30843, #iq=0,
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@29912991,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@659d659d, toWrite=0,
interested=8}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 incompatible 
with
net.spy.memcached.ops.GetOperation$Callback
        at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl
.java:28)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)
        at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMem
cachedNodeImpl.java:44)
        at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)
        at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNode
Impl.java:157)
        at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:337)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:300)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:194)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1504)
2010-01-21 16:05:52,894 WARN  [MemcachedConnection] Closing, and reopening {QA
sa=aklia782/10.16.160.6:11213, #Rops=2, #Wops=30843, #iq=0,
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@29912991,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@659d659d, toWrite=0,
interested=8}, attempt 0.
2010-01-21 16:05:52,894 ERROR [MemcachedConnection] Closing, and reopening {QA
sa=aklia782/10.16.160.6:11213, #Rops=2, #Wops=30843, #iq=0,
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@29912991,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@659d659d, toWrite=0,
interested=8}, attempt 0.
2010-01-21 16:05:52,894 WARN  [BinaryMemcachedNodeImpl] Discarding partially
completed op: net.spy.memcached.protocol.binary.GetOperationImpl@29912991
2010-01-21 16:05:52,894 ERROR [BinaryMemcachedNodeImpl] Discarding partially
completed op: net.spy.memcached.protocol.binary.GetOperationImpl@29912991
2010-01-21 16:05:52,894 ERROR [McConnectionObserver] ConnectionLost to
aklia782/10.16.160.6:11213
2010-01-21 16:05:56,081 INFO  [MemcachedConnection] Reconnecting {QA
sa=aklia782/10.16.160.6:11213, #Rops=1, #Wops=30843, #iq=0, topRop=null,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@659d659d, toWrite=0,
interested=0}
2010-01-21 16:05:56,082 INFO  [MemcachedConnection] Connection state changed for
sun.nio.ch.SelectionKeyImpl@279f279f
2010-01-21 16:05:56,433 WARN  [CaptchaFacade] No X-Forwarded-For found in 
request !
2010-01-21 16:05:56,810 WARN  [CaptchaFacade] No X-Forwarded-For found in 
request !
2010-01-21 16:05:57,084 INFO  [McConnectionObserver] ConnectionEstablished to
aklia782/10.16.160.6:11213, retries 1, version is null
2010-01-21 16:05:57,085 INFO  [MemcachedConnection] Reconnecting due to 
exception on
{QA sa=aklia782/10.16.160.6:11213, #Rops=2, #Wops=30843, #iq=0,
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@659d659d,
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@7860786, toWrite=0,
interested=8}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 incompatible 
with
net.spy.memcached.ops.GetOperation$Callback
        at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl
.java:28)
        at
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)
        at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMem
cachedNodeImpl.java:44)
        at
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:33)
        at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNode
Impl.java:157)
        at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:337)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:300)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:194)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1504)

Original comment by lord.bud...@gmail.com on 22 Jan 2010 at 6:52

GoogleCodeExporter commented 8 years ago 
Following upgrade to 2.4.2 I'm encountering this problem trying to use the 
binary
protocol. We have seen this error logged on Windows, Solaris, and Mac OS.

In a test program, running v2.4.2 on a Windows client & Windows server (STAT 
version
1.4.4-53-g0b7694c), I encounter the exception using the attached test as soon 
as I
dial threads to 5 (1 or 2 seems OK) and set optimize to true. Set optimize to 
false
and it works fine.

Eg this will log the ClassCastException:

        final int num_thread = 5;
        final boolean optimize = true;

This will not:

        final int num_thread = 5;
        final boolean optimize = false;

Sample output and quick & dirty test program (you have to modify code to toggle
threads & optimization) attached.

Hope that helps.

Original comment by rod.shee...@gmail.com on 21 Apr 2010 at 1:30

Attachments:

GoogleCodeExporter commented 8 years ago 
We experienced this problem as well (stack trace below).  As far as we know 
nothing was done differently than normal.  So far we could not reproduce.

2010-09-30 11:15:16.377 INFO net.spy.memcached.MemcachedConnection:  
Reconnecting due to exception on {QA sa=/127.0.0.1:11211, #Rops=2, #Wops=1, 
#iq=0, topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@60e1b2f4, 
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@1abe2dd, toWrite=0, 
interested=4}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be 
cast to net.spy.memcached.ops.GetOperation$Callback
     at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
     at net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl.java:28)
     at net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:21)
     at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMemcachedNodeImpl.java:46)
     at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcachedNodeImpl.java:35)
     at net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:177)
     at net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:374)
     at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:336)
     at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:219)
     at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1591)
2010-09-30 11:15:16.378 WARN net.spy.memcached.MemcachedConnection:  Closing, 
and reopening {QA sa=/127.0.0.1:11211, #Rops=2, #Wops=1, #iq=0, 
topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@60e1b2f4, 
topWop=net.spy.memcached.protocol.binary.GetOperationImpl@1abe2dd, toWrite=0, 
interested=4}, attempt 0.
2010-09-30 11:15:16.378 WARN 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl:  Discarding 
partially completed op: 
net.spy.memcached.protocol.binary.StoreOperationImpl@60e1b2f4

Original comment by lonbin...@gmail.com on 30 Sep 2010 at 8:05

GoogleCodeExporter commented 8 years ago 
I meet the same problem, either. Looks like it is not easy to reproduce.

stack trace as below:

java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be 
cast to net.spy.memcached.ops.GetOperation$Callback
    at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
    at net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl.java:28)
    at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMemcachedNodeImpl.java:53)
    at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcachedNodeImpl.java:35)
    at net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:177)
    at net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:374)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:336)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:219)
    at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1591)

Original comment by schumi....@gmail.com on 19 Oct 2010 at 8:09

GoogleCodeExporter commented 8 years ago 
This is a popular issue and Eric hasn't been able to put time in on it; taking 
it over (unless this wakes Eric up).

Original comment by ingen...@gmail.com on 3 Jan 2011 at 1:26

GoogleCodeExporter commented 8 years ago 
This occurred on our production system this morning.  We are running 
spymemcached 2.6rc1, so it's still a valid issue.  All of our classes and the 
spymemcached JAR should be loaded via the same class loader, so a class loader 
issue seems surprisingly... but what else could explain it?

14 Jun 2011 14:02:12,996 ERROR [STDERR] Exception in thread "Memcached IO over 
{MemcachedConnection to <IP's removed>}"
14 Jun 2011 14:02:12,997 ERROR [STDERR] java.lang.ClassCastException: 
net.spy.memcached.MemcachedClient$5 cannot be cast to 
net.spy.memcached.ops.GetOperation$Callback
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl
.java:28)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:
21)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMem
cachedNodeImpl.java:46)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcach
edNodeImpl.java:35)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNode
Impl.java:194)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:365)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.MemcachedConnection.handleInputQueue(MemcachedConnection.java:
254)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:162)
14 Jun 2011 14:02:12,997 ERROR [STDERR]         at 
net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1622)

Original comment by brandon....@gmail.com on 14 Jun 2011 at 9:29

GoogleCodeExporter commented 8 years ago 
facing the same problem:
OS: Linux version 2.6.26-2-amd64 (Debian 2.6.26-25lenny1)
Server: Membase version 1.7.0
Client: spymemcached-2.7

INFO: Reconnecting due to exception on {QA sa=mb1/172.30.0.42:11210, #Rops=0, 
#Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=4}
java.lang.ClassCastException: net.spy.memcached.MemcachedClient$6 cannot be 
cast to net.spy.memcached.ops.GetOperation$Callback
    at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
    at net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl.java:28)
    at net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:21)
    at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMemcachedNodeImpl.java:46)
    at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcachedNodeImpl.java:35)
    at net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:196)
    at net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:468)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:430)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:280)
    at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2063)

the errors accour unpredictable
this problem was first reported almost 2 years ago...

Original comment by andi...@gmail.com on 8 Jul 2011 at 12:44

GoogleCodeExporter commented 8 years ago 
Looking at this issue it seems like there are two issues here. One gives this:

net.spy.memcached.MemcachedClient$5 cannot be cast to 
net.spy.memcached.ops.GetOperation$Callback

and the other gives this:

net.spy.memcached.ops.MultiGetsOperationCallback cannot
be cast to net.spy.memcached.ops.GetOperation$Callback

I can't really tell what the issue is with the first error message, but the 
second one happens because right now all binary get operations (get, gets, 
getl, and gat) are processed by the same object, GetOperationImpl. If you look 
at the constructors the main difference is that you pass in a callback class 
corresponding to the operation you are doing. Gets for example uses the 
GetsOperation.Callback class. The issue comes into play because all of these 
get operations look the same to the BinaryMemcachedNodeImpl. When we optimize 
we just check to see if an operation is a get (so in this case a get, getl, 
gets, or gat) and then put it into a multi-get. The ClassCastException happens 
because all of these operations look like get operations, but have different 
callbacks. So when a 'gets' gets thrown into the multi-get the class cast 
exception is inevitable since it doesn't have the "right" callback that the 
multiget is looking for. That sums up the reason one of the error messages is 
seen and could possibly be the reason for the other. I'll try to get this fixed 
for the next release of Spymemcached. I think the proper behavior here is to 
only have get operations being put into the multi-get since I don't think it 
makes sense to have any of the others in there. I assume also that putting them 
in there causes incorrect behavior.

Original comment by mikewie...@gmail.com on 11 Jul 2011 at 3:51

GoogleCodeExporter commented 8 years ago 
I disagree with that last bit of the assessment -- I think *everything* should 
be done the same way the multi-get and multi-set are done in binary.  
Effectively, we should reduce every command to a quiet command and interleave 
them freely, occasionally putting a cap on them to keep stuff from getting 
pushed out too far.

Since we can infer everything and we've seen great performance increases for 
both multiget and multiset, having it all be 
multi{get,set,delete,incr,decr,etc...} would be pretty awesome.

Original comment by dsalli...@gmail.com on 11 Jul 2011 at 4:58

GoogleCodeExporter commented 8 years ago 
This is hitting our production environment as well during heavy load times.  I 
didn't see it at first because I have the log level for 
net.spy.memcached.MemcachedConnection set to WARN because it is very verbose at 
INFO.  Unfortunately, the above error is also logged at INFO :)

I changed our log level and verified this during a time when we got excessive 
operation time outs.  I have seen multiple bug reports related to excessive 
time outs and they keep getting fixed, but our time outs persist.  I wonder if 
this is one more cause...

I can also confirm the above explanation regarding multi-get escalations.  If I 
disable multi-get escalations, our time outs instantly go away.

Original comment by brandon....@gmail.com on 11 Jul 2011 at 4:25

GoogleCodeExporter commented 8 years ago 
This is a valid bug.  If I look at the fifth anonymous inner class in 
MemcachedClient.java, it is an inline interface instantiation of 
GetsOperation.Callback.  Notice the s.  Gets not get.  So a GET operation and 
GETS (used for CAS) operation are not compatible polymorphically.

Yet, in ProxyCallback.addCallbacks, o.getCallback() is blindly cast to a 
GetOperation.Callback.  Thus, the following ClassCastException occurs:

java.lang.ClassCastException: net.spy.memcached.MemcachedClient$5 cannot be 
cast to net.spy.memcached.ops.GetOperation$Callback
        at net.spy.memcached.protocol.ProxyCallback.addCallbacks(ProxyCallback.java:25)
        at net.spy.memcached.protocol.binary.OptimizedGetImpl.addOperation(OptimizedGetImpl.java:28)
        at net.spy.memcached.protocol.binary.OptimizedGetImpl.<init>(OptimizedGetImpl.java:21)
        at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimizeGets(BinaryMemcachedNodeImpl.java:46)
        at net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl.optimize(BinaryMemcachedNodeImpl.java:35)
        at net.spy.memcached.protocol.TCPMemcachedNodeImpl.fillWriteBuffer(TCPMemcachedNodeImpl.java:194)
        at net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:365)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:327)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:199)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1622)

We are running version 2.6rc1 if that matters at all.

Original comment by brandon....@gmail.com on 11 Jul 2011 at 5:42

GoogleCodeExporter commented 8 years ago 
It's definitely a valid bug and will be fixed shortly.

Original comment by dsalli...@gmail.com on 11 Jul 2011 at 5:50

GoogleCodeExporter commented 8 years ago 
Let me know if I can help in any way.  We are definitely interested in testing 
as soon as it's fixed.  I'll see if I can create a reproducible test case.

Original comment by brandon....@gmail.com on 11 Jul 2011 at 6:04

GoogleCodeExporter commented 8 years ago 
Test case attached.  You can compile and run the Java class directly or if you 
have ant installed, just run "ant".  The default target will compile and run 
the test case.

Original comment by brandon....@gmail.com on 11 Jul 2011 at 8:17

Attachments:

GoogleCodeExporter commented 8 years ago

Original comment by ingen...@gmail.com on 11 Jul 2011 at 9:13

GoogleCodeExporter commented 8 years ago 
Ok, I have just checked in code to spymemcached with the fix. I have tested the 
code with all of the tests that have been posted in this forum. Thanks to 
everyone that helped us located the issue. Attached is a spymemcached jar that 
contains the fix. This code will be officially released in the near future. 
Please post any feedback you have.

Original comment by mikewie...@gmail.com on 11 Jul 2011 at 10:06

Attachments:

GoogleCodeExporter commented 8 years ago

Original comment by ingen...@gmail.com on 13 Jul 2011 at 8:07

GoogleCodeExporter commented 8 years ago

Original comment by mikewie...@gmail.com on 13 Jul 2011 at 8:08