Error zipping/unzipping serialized java object

[GoogleCodeExporter]

Hi,

I'm using jmemcached 0.7as the server, spymemcached as the client, all in
Windows XP.

I have a large Java object. Once serialized is about 2.2 megabytes. When I
send it to the server, it gets compressed because it is bigger than the
threshold. However, when I try to retrieve it from the server, the
unzipping fails with an exception.
To fix this issue I'm Base64-encoding the compressed stream, then Base64
decoding it before unzipping. It works.

I have written a new class "Base64ConnectionFactory extends
DefaultConnectionFactory". The only thing I'm doing is overriding
getDefaultTranscoder so it will return a new Base64Transcoder, which
inherits from SerializingTranscoder.
In Base64Transcoder, I override compress and decompress to add the Base64
encoding and decoding. 
Then, I create the client with "new MemcachedClient(new
Base64ConnectionFactory(), listConnections)".

I'm using com.sun.mail.util.BASE64DecoderStream.java and
com.sun.mail.util.BASE64EncoderStream.java to do the job. I don't know if
they are compatible with your license. (see
http://www.java2s.com/Open-Source/Java-Document/EJB-Server-GlassFish/mail/com.su
n.mail.util.htm)

I'm new to memcached. I believe the root cause of this issue is that the
data is being sent and retrieved using a "text" stream instead of a
"binary" stream. I didn't debug your code very deeply, I just had a hunch
when I saw the zip exception and the Base64 encoding/decoding worked.

I cannot send you my code now; I wrote it in another computer with no
internet connection. I'll send the code to you in the next few days if you
want to.

As far as I understand memcached protocol, using a "text" stream could be
unavoidable. If that is the case, you might consider adding the Base64
encoding/decoding step in your client, maybe as an optional step.

Thanks.

Original issue reported on code.google.com by McP...@gmail.com on 15 Jun 2009 at 8:48

[GoogleCodeExporter]

I don't think this is the issue you're seeing.  There's no issue with saving 
binary
data through the text protocol (similarly to the way you can send and receive
arbitrary data through HTTP, which is also a text protocol).  In fact, I store
compressed values, serialized java objects and raw images in memcached with no 
problem.

I don't know exactly what your problem is, but you definitely don't need to 
ascii
encode stuff for that.

Original comment by dsalli...@gmail.com on 15 Jun 2009 at 9:03

• Changed state: NeedInfo

[GoogleCodeExporter]

Hi again,

I have written a small program to help you replicate the issue. Please, follow 
the
steps and let me know if you can replicate it.

0. Unzip the attached file.
1. Run the server: java -jar ./3rdparty/jmemcached-cli-0.7-main.jar -l
<host> -p <port> -v
2. Run the first test: java -Xms100m -Xmx1000m -classpath
"./bin;./3rdparty/memcached-2.3.1.jar" org.mcpolu.issue73.test1 <host>
<port>

I get the following trace:

2009-06-16 12:20:08.710 INFO net.spy.memcached.MemcachedConnection:
Added {QA sa=HOST/xxx.xxx.xxx.xxx:yyyyy, #Rops=0, #Wop

s=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2009-06-16 12:20:08.710 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@64f6

cd
2009-06-16 12:20:09.726 INFO
net.spy.memcached.transcoders.SerializingTranscoder:  Compressed
java.util.HashMap from 2879038 to 73

9793
2009-06-16 12:20:09.851 WARN
net.spy.memcached.transcoders.SerializingTranscoder:  Failed to
decompress data
java.util.zip.ZipException: too many length or distance symbols
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147)
        at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
        at java.io.FilterInputStream.read(FilterInputStream.java:90)
        at
net.spy.memcached.transcoders.BaseSerializingTranscoder.decompress(BaseSerializi
ngTranscoder.java:148)
        at
net.spy.memcached.transcoders.SerializingTranscoder.decode(SerializingTranscoder
.java:53)
        at net.spy.memcached.MemcachedClient$GetFuture.decode(MemcachedClient.java:1712)
        at net.spy.memcached.MemcachedClient$GetFuture.get(MemcachedClient.java:1708)
        at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:833)
        at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:855)
        at org.mcpolu.issue73.test1.main(test1.java:22)
2009-06-16 12:20:09.851 INFO net.spy.memcached.MemcachedClient:  Shut
down memcached client

3. Run the second test: java -Xms100m -Xmx1000m -classpath
"./bin;./3rdparty/memcached-2.3.1.jar;./3rdparty/mail.jar"
org.mcpolu.issue73.test2 <host> <port>

I get the following trace:

2009-06-16 12:24:13.260 INFO net.spy.memcached.MemcachedConnection:
Added {QA sa=HOST/xxx.xxx.xxx.xxx:yyyyy, #Rops=0, #Wop

s=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2009-06-16 12:24:13.260 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@16fa

474
2009-06-16 12:24:14.323 INFO org.mcpolu.issue73.Base64Transcoder:
Compressed java.util.HashMap from 2879038 to 986392
2009-06-16 12:24:14.885 INFO net.spy.memcached.MemcachedClient:  Shut
down memcached client

Note that the server is jmemcached, not memcached. Have you used your client 
with
this particular server? It might be a server issue. I'm reporting this bug to 
you
because the compressing / uncompressing happens in the client side, but I don't 
know
enough about the protocol to know which side (client or server) is responsible. 
May
be this is a not specified case?

Please, let me know if you are able to replicate the bug. May be you are rigth 
and
the Base64 encoding is not needed but it is causing some collateral effect 
(different
data size or something like that) that make it work.

Thank you for your time.

Original comment by McP...@gmail.com on 16 Jun 2009 at 7:44

Attachments:

• Issue73.zip

[GoogleCodeExporter]

I have also seen this problem with jmemcached, I believe it is a jmemcached 
issue
because the same failing tests run ok against memcached. I am going to request 
a 0.8
jmemcached release with support for the binary protocol.

Original comment by fullerga...@gmail.com on 18 Jun 2009 at 2:41

[GoogleCodeExporter]

I don't think binary protocol will help you.  I looked through the jmemcached 
source
a bit (not very deeply) and didn't see the specific types of failures I might 
expect.
 The one thing I did see was that there weren't tests sending in arbitrary bits and
getting them back in tact.

I'm going to declare this not an issue with spymemcached as my apps wouldn't 
work if
this were true.  :)

Original comment by dsalli...@gmail.com on 18 Jun 2009 at 11:23

• Changed state: Invalid

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Thanks!

So it is definitely a jmemcached issue. Anyone knows how to contact jmemcached
developers? I have been looking for an email or a forum or something and I 
cannot
find it. They are using Trac but apparently you need a user/password to open 
new tickets.

Regards.

Original comment by McP...@gmail.com on 19 Jun 2009 at 9:32

[GoogleCodeExporter]

I'm replying to myself just for the sake of completeness.

I was able to contact jmemcached developers and they agreed it was a bug on the
server side. It has already been fixed in jmemcached v0.8. Jmemcached has been 
moved
to http://code.google.com/p/jmemcache-daemon/ so now it is very easy to report 
bugs,
discuss issues, etc.

Original comment by McP...@gmail.com on 23 Jun 2009 at 7:22