Unpacking a collection which should be ignored returns an incorrect result

[james-ecargo]

Recently we felt the need to remove an unused field from a POCO that we have been serializing and deserializing with MsgPack.

Based on the documentation in the wiki about IDs we assumed that fields with missing IDs would be ignored during deserialization.

Missing IDs are considered that those members are omitted in the runtime (CTS type) representation. That is, the nil will be emittes in serialization, the values at those IDs will be skipped in deserialization.

This would give us backwards compatibility with existing serialized instances of this POCO which is highly desirable.

This appears to be true for simple fields like string, bool and int but collections appear to break this.

Below is an example of what's happening. In the test the new version of the POCO removes the array field SecondField. When the serializer unpacks the object it incorrectly returns the first value from the array instead of skipping it and returning the value of the last field.

I've also added an example of removing a string field which does work.

using System.IO;
using MsgPack.Serialization;
using NUnit.Framework;

namespace MsgPack
{
    public class MsgPackTests
    {
        static byte[] Serialize<T>(T thisObj)
        {
            var serializer = MessagePackSerializer.Get<T>();

            using (var byteStream = new MemoryStream())
            {
                serializer.Pack(byteStream, thisObj);
                return byteStream.ToArray();
            }
        }

        static T Deserialize<T>(byte[] bytes)
        {
            var serializer = MessagePackSerializer.Get<T>();

            using (var byteStream = new MemoryStream(bytes))
            {
                return serializer.Unpack(byteStream);
            }
        }

        public class A1
        {
            [MessagePackMember(0)]
            public int FirstField { get; set; }
            [MessagePackMember(1)]
            public string[] SecondField { get; set; }
            [MessagePackMember(2)]
            public string ThirdField { get; set; }
        }

        public class A2
        {
            [MessagePackMember(0)]
            public int FirstField { get; set; }
            [MessagePackMember(2)]
            public string ThirdField { get; set; }
        }

        [Test]
        public void Given_a_new_version_without_collection_When_unpacking_Then_the_collection_is_ignored()
        {
            // Arrange
            var originalVersion = new A1 { FirstField = 1, SecondField = new[] { "2", "3", "4" }, ThirdField = "5" };
            var packedObject = Serialize(originalVersion);

            // Act
            var newVersion = Deserialize<A2>(packedObject);
            Assert.AreEqual(newVersion.FirstField, originalVersion.FirstField);

            // NOTE: This assertion fails because the unpacked version is "2".
            Assert.AreEqual(newVersion.ThirdField, originalVersion.ThirdField);
        }

        public class B1
        {
            [MessagePackMember(0)]
            public int FirstField { get; set; }
            [MessagePackMember(1)]
            public string SecondField { get; set; }
            [MessagePackMember(2)]
            public string ThirdField { get; set; }
        }

        public class B2
        {
            [MessagePackMember(0)]
            public int FirstField { get; set; }
            [MessagePackMember(2)]
            public string ThirdField { get; set; }
        }

        [Test]
        public void Given_a_new_version_without_field_When_unpacking_Then_the_field_is_ignored()
        {
            // Arrange
            var originalVersion = new B1 { FirstField = 1, SecondField = "2", ThirdField = "3" };
            var packedObject = Serialize(originalVersion);

            // Act
            var newVersion = Deserialize<B2>(packedObject);
            Assert.AreEqual(newVersion.FirstField, originalVersion.FirstField);

            // NOTE: This assertion passes.
            Assert.AreEqual(newVersion.ThirdField, originalVersion.ThirdField);
        }
    }
}

So a couple of questions:

• Is this behaviour actually supposed to be supported in MsgPack?
• If not is there an idiomatic way to remove a collection field while maintaining backward compatibility?