Deserialize failed when reading objects from ObjectInput/DataInput

[GoogleCodeExporter]

version: 1.0.4 and 1.0.5-SNAPSHOT

I serialize a java bean to byte stream then deserialize it. Got error 
"com.dyuproject.protostuff.ProtobufException: While parsing a protocol message, 
the input ended unexpectedly in the middle of a field.  This could mean either 
than the input has been truncated or that an embedded message misreported its 
own length." and "java.lang.NegativeArraySizeException".

I attached the test project. Please run SerializeTest.

I did some debug on this issue. I think the issue was caused by below code in 
GraphIOUtil.java

    public static <T> void mergeDelimitedFrom(DataInput in, T message, Schema<T> schema) 
    throws IOException
    {
        final byte size = in.readByte();
        if(size == -1)
            throw ProtobufException.truncatedMessage();

        final int len = size < 0x80 ? size : CodedInput.readRawVarint32(in, size);

the size could be -1.  The len code should be :
        final int len = (size & 0x80) == 0 ? size : CodedInput.readRawVarint32(in, size);

Other *IOUtil.hava have same problem.

Original issue reported on code.google.com by spitfi...@gmail.com on 9 Mar 2012 at 7:07

Attachments:

• protostuff.zip

[GoogleCodeExporter]

More information

OS: Win7 32bit & Mac 64 bit
JDK: 1.6_20

Original comment by spitfi...@gmail.com on 9 Mar 2012 at 7:09

[GoogleCodeExporter]

Your test code should not directly serialize HotelAvailRS if you want to use 
the message's Externalizable (this was designed for situations where the 
message is wrapped by a regular pojo)

E.g
public static final class Wrapper implements Serializable
{
    HotelAvailRS target;
    public Wrapper() {}
    public Wrapper(HotelAvailRS target) { this.target = target; }
}

// then in your test case:
os.writeObject(new Wrapper(createHotelAvailRS()));

If you want to directly serialize HotelAvailRS (a message with a schema), why 
go through the externalizable route when you can use ProtostuffIOUtil.write* 
methods?

Original comment by david.yu...@gmail.com on 9 Mar 2012 at 8:39

[GoogleCodeExporter]

In my project, we have a normal java wrapper class named "Message". The Class 
"Message" have a "payload" field. We put HotelAvailRQ and HotelAvailRS into it. 
We need to serialize and deserialize the Message object when we call RPC. We 
use protostuff for payload class because protostuff is very fast.

I changed my test code and add a Wrapper class as you said. I get a new error. 
How can I fix it?

Here is the code.

    static public class Wrapper implements Serializable {
        private HotelAvailRS payload;

        public Wrapper() {
        }

        public Wrapper(HotelAvailRS payload) {
            this.payload = payload;
        }
    }

    @Test
    public void test() throws IOException, ClassNotFoundException {
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        ObjectOutputStream os = new ObjectOutputStream(byteArrayOutputStream);
        os.writeObject(new Wrapper(createHotelAvailRS()));
        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
        ObjectInputStream is = new ObjectInputStream(byteArrayInputStream);
        Object o = is.readObject();
        System.out.println(o.getClass().toString());
    }

Here is the exception information.

java.lang.IllegalStateException: unread block data
    at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2377)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1361)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at SerializeTest.test(SerializeTest.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:71)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Original comment by spitfi...@gmail.com on 9 Mar 2012 at 8:58

[GoogleCodeExporter]

I got that also.  Been debugging for over an hour.
It is a jvm bug in which if you comment out some fields (no particular 
pattern), the error goes away.  I'm on ubuntu x64 with java 1.6u26 btw.
Try searching for "unread block data" and you'll have an idea.

Original comment by david.yu...@gmail.com on 9 Mar 2012 at 9:17

[GoogleCodeExporter]

Hi David,
  I did some search on google but not get valuable results on this issue. 

  I try to fix it. After I changed the code to 
final int len = (size & 0x80) == 0 ? size : CodedInput.readRawVarint32(in, 
size);

and remove 

 if(size == -1)
            throw ProtobufException.truncatedMessage();

All is OK. Does this fix have any issues?

Original comment by spitfi...@gmail.com on 9 Mar 2012 at 10:44

[GoogleCodeExporter]

You are quite right.  If the "if(size == -1)" check is not removed, it fails if 
the length is exactly 255.

Your solution works but we'll give up checking truncated messages.

The other solution is to simply change:
final byte size = in.readByte();

to:
final int size = 0xFF & in.readByte();

Not entirely sure if this is desirable.

I'm fine with your approach too.

Original comment by david.yu...@gmail.com on 9 Mar 2012 at 1:21

[GoogleCodeExporter]

Note that this affects only when DataInput/ObjectInput is used.
I assumed that InputStream.read() == DataInput.readByte() ... which was wrong.

I've attached the patch.
Will make a release soonish.

Original comment by david.yu...@gmail.com on 9 Mar 2012 at 4:57

Attachments:

• issue-103.patch

[GoogleCodeExporter]

I've implemented it like you suggested.  E.g removed -1 check and use 0 == 
(size & 0x80).  Its better this way as the DataInput/ObjectInput will contain 
the logic for EoF from readByte().

@rev 1451

Original comment by david.yu...@gmail.com on 25 Mar 2012 at 12:51

• Changed title: Deserialize failed when reading objects from ObjectInput/DataInput
• Changed state: Fixed

[GoogleCodeExporter]

Many thanks. How long will release 1.0.5?

Original comment by spitfi...@gmail.com on 25 Mar 2012 at 2:04

[GoogleCodeExporter]

Targetting by the end of this week.

Original comment by david.yu...@gmail.com on 25 Mar 2012 at 2:27