search

plutoo / protobuf-csharp-port

Automatically exported from code.google.com/p/protobuf-csharp-port
Other
0 stars 0 forks source link

Unknown fields in LITE framework cause Invalid Protocol Buffer Exception when spanning across input buffer #54

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Write several length-prefixed messages using the LITE runtime
2. Ensure that an unkonwn field of a message spans across the buffer size(4096)
3. Follow this with another valid length-prefixed message
4. Attempt to read

What is the expected output? What do you see instead?
Should successfully call SkipBytes across the buffer boundry.  SkipBytes has a 
bug that is miscalculating the number of used bytes (totalBytesRetired).

see CodedInputStream.cs, line 1784
  // Skipping more bytes than are in the buffer.  First skip what we have.
  int pos = bufferSize - bufferPos;
  totalBytesRetired += pos;

this should 'retire' the entire size of the buffer:
  totalBytesRetired += bufferSize;

Original issue reported on code.google.com by Grig...@gmail.com on 7 May 2013 at 7:43

GoogleCodeExporter commented 9 years ago 
Fixed, see changeset 518:86ffec3ac633

Original comment by Grig...@gmail.com on 7 May 2013 at 8:42

GoogleCodeExporter commented 9 years ago 
Hi there,
Is this issue fixed in the latest version? cause I kept got 
InvalidProtocolBufferException on calling ParseFrom(xxx)

Original comment by aple....@gmail.com on 19 Jul 2013 at 3:53

GoogleCodeExporter commented 9 years ago 
attach is my demo protofile, I got InvalidProtocolBufferException everytime on 
call

GetFeedResult.ParseFrom(xx); xx is a bytestring serialized by protocol c++.

Original comment by aple....@gmail.com on 19 Jul 2013 at 3:58

GoogleCodeExporter commented 9 years ago 
attach is my demo protofile, I got InvalidProtocolBufferException everytime on 
call

GetFeedResult.ParseFrom(xx); xx is a bytestring serialized by protocol c++.

Original comment by aple....@gmail.com on 19 Jul 2013 at 3:59

Attachments:

GoogleCodeExporter commented 9 years ago 
@aple.smx: Which version are you using?

Original comment by jonathan.skeet on 19 Jul 2013 at 4:05

GoogleCodeExporter commented 9 years ago 
@aple.smx, can you also attach a binary file containing the invalid message 
bytes?

Original comment by Grig...@gmail.com on 19 Jul 2013 at 4:06

GoogleCodeExporter commented 9 years ago 
@jonathan.skeet， version is 2.4.1.521, the latest one.

Original comment by aple....@gmail.com on 20 Jul 2013 at 6:51

GoogleCodeExporter commented 9 years ago 
error message is While parsing a protocol message, the input ended unexpectedly 
in the middle of a field.  This could mean either than the input has been 
truncated or that an embedded message misreported its own length.

the exception stack as below:

   at Google.ProtocolBuffers.CodedInputStream.ReadMessage(IBuilderLite builder, ExtensionRegistry extensionRegistry)
   at Coop.Web.ProtoModels.CRM.DemoFeed.Builder.MergeFrom(ICodedInputStream input, ExtensionRegistry extensionRegistry) in c:\Project\FS.Service\Codes\Services\Coop.Web2\ProtoModels\FSCRMBuz.cs:line 1430
   at Google.ProtocolBuffers.AbstractBuilderLite`2.Google.ProtocolBuffers.IBuilderLite.WeakMergeFrom(ICodedInputStream input, ExtensionRegistry registry)
   at Google.ProtocolBuffers.CodedInputStream.ReadMessage(IBuilderLite builder, ExtensionRegistry extensionRegistry)
   at Coop.Web.ProtoModels.CRM.GetFeedResult.Builder.MergeFrom(ICodedInputStream input, ExtensionRegistry extensionRegistry) in c:\Project\FS.Service\Codes\Services\Coop.Web2\ProtoModels\FSCRMBuz.cs:line 2184
   at Google.ProtocolBuffers.AbstractBuilderLite`2.MergeFrom(ByteString data, ExtensionRegistry extensionRegistry)
   at Coop.Web.ProtoModels.CRM.GetFeedResult.ParseFrom(ByteString data) in c:\Project\FS.Service\Codes\Services\Coop.Web2\ProtoModels\FSCRMBuz.cs:line 2004
   at Coop.Web.Controllers.HomeController.Test() in c:\Project\FS.Service\Codes\Services\Coop.Web2\Controllers\HomeController.cs:line 63

Original comment by aple....@gmail.com on 20 Jul 2013 at 7:06

Attachments:

GoogleCodeExporter commented 9 years ago 
the raw file.

Original comment by aple....@gmail.com on 20 Jul 2013 at 7:07

Attachments:

GoogleCodeExporter commented 9 years ago 
@aple.smx

This issue is not the one you are experiencing as this only happens with very 
long messages.  Based on the sample data you sent me I would guess that your 
proto definition does not match the definition used to generate the message.  
Specifically if you look at the message contents you see this raw data:

*Format: (depth for nesting) FieldType(#id)@start-end   = value

LengthDelimited(1)@1-4   =
        Varint(1)@1-2    = 10
LengthDelimited(2)@5-48  =
        Varint(1)@1-2    = 100
        Varint(2)@3-4    = 1
        Varint(3)@5-6    = 0
        Varint(4)@7-8    = 1
        Varint(5)@9-10   = 14
        Varint(6)@11-13  = 2012
        Varint(7)@14-15  = 0
        LengthDelimited(8)@16-38         = "\345\210\206\344\272\253\345\233\276\347\211\207\345\222\214\351\231\204\344\273\266"
        LengthDelimited(9)@39-42         =
                Varint(1)@1-2    = 1

this results in two top-level fields (1, 2) which are both messages themselves. 
 The proto message you are trying to extract is as follows:

message GetFeedResult
{
  optional bool success = 1;
  optional string message = 2;
  optional DemoFeed feed = 3;
}

Field 1 is certainly not a boolean field as it would not be length delimited.  
Field 2 is extremely unlikely to be anything but a message due to the number of 
fields and length delimiter match.  I don't see a message in your proto that 
looks like it matches your data.  My suspicion is that it may be something like 
:

message TestResult
{
    optional GetFeed feed_id = 1;
    optional DemoFeed feed_data = 2;
}

which then reads as the following:

{
    "feed_id": {
        "feed_id": 10
    },
    "feed_data": {
        "feed_id": 100,
        "type": "kShare",
        "is_public": false,
        "source": "kWeb",
        "reply_count": 14,
        "create_time": 2012,
        "update_time": 0,
        "content": "\u5206\u4eab\u56fe\u7247\u548c\u9644\u4ef6",
        "creator": {
            "id": 1
        }
    }
}

I'm not real sure that the first message is of type GetFeed, but there is no 
way to determine from the data you gave me.  Anyway, it looks like your proto 
is incorrect?  Hope the above helps.

Original comment by Grig...@gmail.com on 20 Jul 2013 at 4:34

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Sorry for giving an old proto file.
But good for me is that I got some idea from Grig..'s debug, and the issue was 
solved. It's my rpc fault on processing received data from byte[] to string.

thank you for ur help.

Original comment by aple....@gmail.com on 21 Jul 2013 at 8:12

GoogleCodeExporter commented 9 years ago 
BTW, it's a great project.

Original comment by aple....@gmail.com on 21 Jul 2013 at 8:13