Cannot serialize class with a generics in NativeAOT

[InstinctTheDevil]

Hello, I'm currently trying to add MemoryPack to my project, but I've encountered a roadblock

I cannot serialize a type if it has a generic, even if the generic type is also memorypackable.

here's an example:

[MemoryPackable]
public partial class TestData(string one, string two)
{
    public string One { get; set; } = one;
    public string Two { get; set; } = two;
}

[MemoryPack.MemoryPackable]
[Serializable]
public partial class Test<T>(bool testBool,  T data)  where T : IMemoryPackable<T>
{
    public T Data { get; set; } = data;
    public bool TestBool { get; set; } = testBool;
}

and I want to serialize it like this:

TestData testData = new(one: "instinct",two: "1");
var apiResponse = new Test<TestData>(testBool: true, data: testData);
var serialized = MemoryPackSerializer.Serialize(apiResponse);

but I get the following error:

Process terminated. Failed to create generic virtual method implementation

Declaring type: MemoryPack.Formatters.MemoryPackableFormatter`1<TestProject.Test.ApiResponse`1<TestProject.Test.UserData>>
Method name: Serialize
Instantiation:
  Argument 00000000: MemoryPack.Internal.ReusableLinkedArrayBufferWriter

I'm using the latest version of MemoryPack and .NET 8. Is there anything I can do against this?

[meryuhi]

Hi, I think there seems to be a bit of a compatibility issue with reference types (class or record) and static virtual members in interfaces (which MemoryPack source generation part rely on) in NativeAOT. In some cases, serializing non-generic class still get this error.

One easy workaround is to use value types (struct or record struct).

[Verfin]

One easy workaround is to use value types (struct or record struct).

Doesn't seem to work, at least in the current version available in nuget. This code immediately crashes with the above exception when AOT compiled, but works fine when AOT is turned off

[MemoryPackable]
public partial struct TestRecordv1
{
}

[MemoryPackable]
public partial class Someclass<T>
{
    public T testRecord;
}

class Program
{
    static void Main(string[] args)
    {
        var val = new Someclass<TestRecordv1>();
        ArrayBufferWriter<byte> writer = new ArrayBufferWriter<byte>();
        MemoryPackSerializer.Serialize(writer, val);
    }
}

[meryuhi]

Try using struct for Someclass<T> as well.

[Verfin]

Try using struct for Someclass as well.

Problem with that is, is that you can't use the DeserializeAsync(Type, Stream) or the Deserialize(Type, ReadOnlySpan) methods as they crash even in non-AOT if the root object is a struct. Deserialize/Async do work correctly if you use the generic versions though.. but that's not good enough for my use case unfortunately.

        var val = new SomeStruct<TestRecordv1>(new TestRecordv1());
        ArrayBufferWriter<byte> writer = new ArrayBufferWriter<byte>();
        MemoryPackSerializer.SerializeBuffer(writer, val);
        MemoryStream ms = new MemoryStream(writer.WrittenMemory.ToArray());
        var result = MemoryPackSerializer.Deserialize<SomeStruct<TestRecordv1>>(writer.WrittenMemory.Span);

[Verfin]

If I use the [MemoryPackable(VersionTolerant)] attribute instead of the default one, the Typed version runs, but results in zeroed data..

Seems like the root object must be a class for the library to work properly

[meryuhi]

Well, I don't seem to find SerializeBuffer method in MemoryPackSerializer. And I tested with similar code and it works. Is there something different about your use case?

using MemoryPack;
using System.Diagnostics;
using TestData = TestContainer<TestItem>;

var data = new TestData(new TestItem("a", "b"));
var writer = new System.Buffers.ArrayBufferWriter<byte>();
MemoryPackSerializer.Serialize(writer, data);
Debug.Assert(data == MemoryPackSerializer.Deserialize<TestData>(writer.WrittenMemory.Span));
using var ms = new MemoryStream(writer.WrittenMemory.ToArray());
Debug.Assert(data == await MemoryPackSerializer.DeserializeAsync<TestData>(ms));

[MemoryPackable]
public partial record struct TestContainer<T>(T Data) {}
[MemoryPackable]
public partial record struct TestItem(string One, string Two) {}

[Verfin]

I don't seem to find SerializeBuffer

Apologies, that "Buffer" seems to have gotten pasted in somehow..

I think I figured out the actual problem... a nested generic struct with the inner struct containing only 1 integer causes a crash... If you try this code, you can see that the TestRecordv1 with an integer + extra stuff works fine, but the TestRecordv2 crashes

[MemoryPackable] public partial record struct TestRecordv1(int i, string str1, string str2);
[MemoryPackable] public partial record struct TestRecordv2(int i);
[MemoryPackable] public partial record struct TestContainer<T>(T testRecord);

class Program
{
    static void Main(string[] args)
    {
        {
            var val1 = new TestContainer<TestRecordv1>(new TestRecordv1(42, "one", "two"));
            ArrayBufferWriter<byte> writer = new ArrayBufferWriter<byte>();
            MemoryPackSerializer.Serialize(writer, val1);
            var obj = MemoryPackSerializer.Deserialize(typeof(TestContainer<TestRecordv1>), writer.WrittenMemory.Span);
            if (obj is TestContainer<TestRecordv1> td)
                Console.WriteLine(td.testRecord.i);
        }

        {
            var val2 = new TestContainer<TestRecordv2>(new TestRecordv2(123));
            ArrayBufferWriter<byte> writer2 = new ArrayBufferWriter<byte>();
            MemoryPackSerializer.Serialize(writer2, val2);
            var obj = MemoryPackSerializer.Deserialize(typeof(TestContainer<TestRecordv2>), writer2.WrittenMemory.Span);
            if (obj is TestContainer<TestRecordv2> td)
                Console.WriteLine(td.testRecord.i);
        }
    }
}

[Verfin]

Actually, having more integers doesn't fix the issue either.. Adding a string field does. Very peculiar..

[MemoryPackable] public partial record struct TestRecordv2(int i, int i2, int i3); ^ still crashes

[meryuhi]

Oh I see, the root object must be a reference type if use Deserialize(Type, ReadOnlySpan), MemoryPack won't check that if the type is value type or something special. It doesn't matter what the combination of fields.

This brings us back to the original problem, the core issue is around MemoryPackableFormatter, maybe it could be fixed in .net 9.0. dotnet/runtime#104913