Out of Memory due to Serialization

[GoogleCodeExporter]

Version: 2.3.1
OS: Linux

I am currently using spymemcached 2.3.1 for the performance test of 
memcahced. In my jython script, I used memcached a set method of 
spymemcached.

While I was conducting the performance test with grinder executing the 
jython script. I encountered the following error. For your information, 
each grinder agent was given 1 GB. This happened even with 1 GB memory 
although it happened later than the former one. 

I suspect the the serialization code in 
net.spy.memcached.transcoders.BaseSerializingTranscoder.serialize eats up 
lots of memory.

I did the same test with Whalin but it did not happen.

Java exception calling TestRunner
        File "/home1/grinder/./Set_Test-file-
store/current/set_oneSize_io.py", line 49, in grinder_set
        File "/home1/grinder/./Set_Test-file-
store/current/set_oneSize_io.py", line 45, in __call__
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:2786)
        at java.io.ByteArrayOutputStream.toByteArray
(ByteArrayOutputStream.java:133)
        at 
net.spy.memcached.transcoders.BaseSerializingTranscoder.serialize
(BaseSerializingTranscoder.java:79)
        at net.spy.memcached.transcoders.SerializingTranscoder.encode
(SerializingTranscoder.java:126)
        at net.spy.memcached.MemcachedClient.asyncStore
(MemcachedClient.java:262)
        at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:619)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyMethod.__call__(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyMethod.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at 
net.grinder.engine.process.jython.InstrumentedPyJavaInstanceForPyMethods.ac
cess$401(InstrumentedPyJavaInstanceForPyMe
thods.java:38)
        at 
net.grinder.engine.process.jython.InstrumentedPyJavaInstanceForPyMethods$5.
call(InstrumentedPyJavaInstanceForPyMethod
s.java:131)
        at net.grinder.engine.process.TestData$Dispatcher.dispatch
(TestData.java:229)
        at net.grinder.engine.process.TestData$DispatcherHolder.dispatch
(TestData.java:166)
        at net.grinder.engine.process.TestData.dispatch(TestData.java:102)
        at 
net.grinder.engine.process.jython.JythonScriptEngine$PyDispatcher.dispatch
(JythonScriptEngine.java:474)
        at net.grinder.engine.process.jython.InstrumentationHelper.dispatch
(InstrumentationHelper.java:104)
        at 
net.grinder.engine.process.jython.InstrumentedPyJavaInstanceForPyMethods.in
voke(InstrumentedPyJavaInstanceForPyMethod
s.java:128)
        at org.python.core.PyInstance.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
        at org.python.pycode._pyx0.grinder_set$5(/home1/grinder/./Set_Test-
file-store/current/set_oneSize_io.py:49)
        at org.python.pycode._pyx0.call_function(/home1/grinder/./Set_Test-
file-store/current/set_oneSize_io.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
7/14/09 2:44:30 PM (thread 1 run 86 test 1): Aborted run due to Java 
exception calling TestRunner
Java exception calling TestRunner
        File "/home1/grinder/./Set_Test-file-
store/current/set_oneSize_io.py", line 49, in grinder_set
        File "/home1/grinder/./Set_Test-file-
store/current/set_oneSize_io.py", line 45, in __ca

Original issue reported on code.google.com by hankm2...@gmail.com on 20 Jul 2009 at 9:31

[GoogleCodeExporter]

It's very likely that you just ran the thing out of memory enqueueing large 
amounts
of data asynchronously without ever slowing down queue insertion rates.

I can't see what your test actually did here, but based on the filename, it 
sounds
like it might be an invalid queue overflow flood (i.e. the kind of thing I 
wouldn't
expect to see in an application, and if your application did need something 
similar,
it should be using CacheLoader).

Original comment by dsalli...@gmail.com on 11 Nov 2009 at 8:28

• Changed state: Invalid