A vector<int> instance, serialised in C++, is not deserialised correctly in C#

[sebjf]

I have encountered an issue passing a set of integers between C++ and C#.

To recreate, create two projects, one C++ and one C#, which transfer a set of integers (vector to list). Example code is below.

When running this, the List() in C# will contain the wrong values (mine were [10, 13, 10, 13, ... ]); if any other members are present MessagePack will throw an exception as it fails to deserialise them. This, along with seeing the 'wrong' data in the value fields (for other members) when looking at objects unpacked manually on the managed side with Unpacker.ReadItem() suggests that MessagePack loses its place.

This only occurs with vectors of integers, not vectors of floats or other arbitrary/custom data types, which are serialised/deserialised correctly.

I do not know enough about the code to say what is causing the issue, but two places I thought were possibilities stepping through with the debugger, is the serialisation stage, since it appeared when the array was being updated each integer was being passed with a length of 1. It may also be in the deserialiser as when looking at the object using Unpacker manually, each entry in the created array was of type Byte, not Int.

//C++
class myClass
{
public:
    vector<int> myInts;
    MSGPACK_DEFINE(myInts);//,r,r2,cls,myString);
};

int _tmain(int argc, _TCHAR* argv[])
{
    myClass c;

    for(int i = 0; i < 10; i++)
    {
        c.myInts.push_back(10);
    }

    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, c);
    ofstream myFile;
    myFile.open("E:\\test.test");
    myFile.write(sbuf.data(), sbuf.size());
    myFile.close();

    return 0;   
}

//C#
namespace MessagePackTest_Managed
{
    public class myClass
    {
        [MessagePackMember(0)]
        public List<float> myInts;
    }

    class Program
    {
        static void Main(string[] args)
        {
            FileStream fs = new FileStream("E:\\test.test", FileMode.Open, FileAccess.Read);
            MessagePackSerializer<myClass> c = MessagePackSerializer.Create<myClass>();
            myClass v = c.Unpack(fs);
        }
    }
}

[yfakariya]

Thank you for your feedback.

I don't know C++ implementation very well, and there are some possibilities to fail deserialization. Would you upload serialized binary ("test.test") file? Or, would you tell me the result of following?

FileStream fs = new FileStream("E:\\test.test", FileMode.Open, FileAccess.Read);
byte[] buf = new byte[fs.Length];
fs.Read(buf, 0, buf.Length);
Console.WriteLine(BitConverter.ToString(buf));

[sebjf]

Hello yfakariya,

Here is the content of test.test extracted exactly as you specified above:

91-9A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A-0D-0A

(Windows confirms the file is 22 bytes in length)

[yfakariya]

Thank you sebjf.

I wonder why C++ serializer puts 0x0D in the stream. It looks like just tiny integer value 13, not any header value. I think the serialized value should be: 91-9A-0A-0A-0A-0A-0A-0A-0A-0A-0A-0A (12bytes)

MessagePack CLI's unpacker interprets the C++ generated binary as following:

1 element array
  10 elements array, its values are 13, 10...
Extra 10 bytes(13,10,...)

I believe 0x0D after 0x9A represents integer value 13, so I will ask C++ implementers to why msgpack-c outs such values. Would you tell me what version msgpack-C do you use?

[sebjf]

I use 0.5.4, the same one as available here: http://sourceforge.net/projects/msgpack/files/msgpack/cpp/

Thank you for MessagePack - I was very impressed when looking through the CLI source investigating this bug, this project is quite a feat!

[iwadon]

@sebjf

   myFile.open("E:\\test.test");

This opens a file with text mode. Then 0x0a (LF) is converted to 0x0d 0x0a (CR LF).

You can correct your code:

   myFile.open("E:\\test.test", std::ios_base::binary);

[sebjf]

Ah. Yes. Thank you iwadon! And sorry about the erronous issue.

[yfakariya]

@iwadon Thank you for your great post!