search

roc230 / spymemcached

Automatically exported from code.google.com/p/spymemcached
0 stars 0 forks source link

BaseSerializingTranscoder gives EOFException in deserialize on readObject under high load #273

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What version of the product are you using? On what operating system?
2.6
CentOS 5

Tell me more...
We have a java socket server (SmartFox) utilizing SpyMemcached to access a 
memcache store on the same physical server machine. We are using Terracotta DSO 
to cluster the socket server, and the JVM is custom (Azul Zing). Loads get very 
high when this happens. 

We get an exception and then from that point we start losing performance until 
eventually a couple hours later, the socket server is unable to serve any 
clients at all. 

The server works fine for 24 hours putting our custom POJO in, but then it hits 
this issue and we have to restart it. 

There seems to be no way to catch this and report more information about it 
(because of the async setup), and we're not sure what could be causing this. 

INFO   | jvm 1    | 2013/03/29 07:20:37 | 2013-03-29 07:20:37.811 WARN 
net.spy.memcached.transcoders.SerializingTranscoder:  Caught IOException 
decoding 14181 bytes of data
INFO   | jvm 1    | 2013/03/29 07:20:37 | java.io.EOFException
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.DataInputStream.readInt(DataInputStream.java:375)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:27
75)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readInt(ObjectInputStream.java:949)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.HashSet.readObject(HashSet.java:280)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.lang.reflect.Method.invoke(Method.java:597)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.concurrent.ConcurrentHashMap.readObject(Unknown Source)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
sun.reflect.GeneratedMethodAccessor739.invoke(Unknown Source)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.lang.reflect.Method.invoke(Method.java:597)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
net.spy.memcached.transcoders.BaseSerializingTranscoder.deserialize(BaseSerializ
ingTranscoder.java:100)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
net.spy.memcached.transcoders.SerializingTranscoder.decode(SerializingTranscoder
.java:66)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
net.spy.memcached.transcoders.TranscodeService$1.call(TranscodeService.java:42)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:270)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.concurrent.FutureTask.run(FutureTask.java:55)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
net.spy.memcached.transcoders.TranscodeService$Task.run(TranscodeService.java:89
)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:8
86)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
INFO   | jvm 1    | 2013/03/29 07:20:37 |       at 
java.lang.Thread.run(Thread.java:662)

Original issue reported on code.google.com by vazor...@gmail.com on 2 Apr 2013 at 10:23