Encounter InvalidProgramException

[jckling]

I'm testing NetJSON in Unity with this simple code:

    struct SimpleObjectStruct
    {
        public int ID;
        public string Name;
        public string Value;
    }

    public void TestSimpleStruct()
    {
        var settings = new NetJSON.NetJSONSettings();
        var data = new SimpleObjectStruct() { ID = 10, Name = "Test", Value = "Tester" };
        var json = NetJSON.NetJSON.Serialize(data, settings);
        UnityEngine.Debug.Log(json);

        var data2 = NetJSON.NetJSON.Deserialize<SimpleObjectStruct>(json, settings);
        UnityEngine.Debug.Log(data2);
    }

It throws error on var json = NetJSON.NetJSON.Serialize(data, settings);

InvalidProgramException: Invalid IL code in (wrapper dynamic-method) object:DeserializeValueTextReaderSettings (System.IO.TextReader,NetJSON.NetJSONSettings): IL_0006: ldarg.2   

System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) (at <801a4b5b9f964ad7bbc7574456774626>:0)
System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method) (at <801a4b5b9f964ad7bbc7574456774626>:0)
System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType) (at <801a4b5b9f964ad7bbc7574456774626>:0)
NetJSON.NetJSON+DynamicNetJSONSerializer`1[T].CreateDeserializerWithTextReaderSettings () (at <b1d0275b3ef44169a919f0dca690658f>:0)
NetJSON.NetJSON+DynamicNetJSONSerializer`1[T]..ctor () (at <b1d0275b3ef44169a919f0dca690658f>:0)
NetJSON.NetJSON+NetJSONCachedSerializer`1[T].GetSerializer () (at <b1d0275b3ef44169a919f0dca690658f>:0)
NetJSON.NetJSON+NetJSONCachedSerializer`1[T]..cctor () (at <b1d0275b3ef44169a919f0dca690658f>:0)
Rethrow as TypeInitializationException: The type initializer for 'NetJSONCachedSerializer`1' threw an exception.
NetJSON.NetJSON.Serialize[T] (T value) (at <b1d0275b3ef44169a919f0dca690658f>:0)
moveDetection.TestSimpleStruct () (at Assets/moveDetection.cs:97)
moveDetection.Start () (at Assets/moveDetection.cs:18)

I've also written a class for testing, same error on Serialize function.

    public class SimpleObject
    {
        public int ID { get; set; }
        public string Name { get; set; }
        public string Value { get; set; }
    }

    // NetJSON
    void TestNetJSON()
    {
        var o = new SimpleObject() { ID = 100, Name = "Test", Value = "Value" };

        var output = NetJSON.NetJSON.Serialize(o);
        UnityEngine.Debug.Log(output);

        var newObject = NetJSON.NetJSON.Deserialize<SimpleObject>(output);
        UnityEngine.Debug.Log(newObject);
    }

I'm using Unity2021.3.18f1 with .NET Standard 2.1

[jckling]

I found errors when deserializing Bool array and Char array...

using System;
using Bogus;
using ProtoBuf;

namespace DotnetSerializationBenchmark.Models
{
    [ProtoContract]
    [Serializable]
    public class BuiltInClass
    {
        [ProtoMember(1)]
        public bool Bool { get; set; }
        [ProtoMember(2)]
        public byte Byte { get; set; }
        [ProtoMember(3)]
        public sbyte Sbyte { get; set; }
        [ProtoMember(4)]
        public char Char { get; set; }
        [ProtoMember(5)]
        public decimal Decimal { get; set; }
        [ProtoMember(6)]
        public double Double { get; set; }
        [ProtoMember(7)]
        public float Float { get; set; }
        [ProtoMember(8)]
        public int Int { get; set; }
        [ProtoMember(9)]
        public uint Uint { get; set; }
        [ProtoMember(10)]
        public long Long { get; set; }
        [ProtoMember(11)]
        public ulong Ulong { get; set; }
        [ProtoMember(12)]
        public short Short { get; set; }
        [ProtoMember(13)]
        public ushort Ushort { get; set; }

        //[ProtoMember(14)]
        //public bool[]? BoolArray { get; set; }
        [ProtoMember(15)]
        public byte[]? ByteArray { get; set; }
        [ProtoMember(16)]
        public sbyte[]? SbyteArray { get; set; }
        //[ProtoMember(17)]
        //public char[]? CharArray { get; set; }
        [ProtoMember(18)]
        public decimal[]? DecimalArray { get; set; }
        [ProtoMember(19)]
        public double[]? DoubleArray { get; set; }
        [ProtoMember(20)]
        public float[]? FloatArray { get; set; }
        [ProtoMember(21)]
        public int[]? IntArray { get; set; }
        [ProtoMember(22)]
        public uint[]? UintArray { get; set; }
        [ProtoMember(23)]
        public long[]? LongArray { get; set; }
        [ProtoMember(24)]
        public ulong[]? UlongArray { get; set; }
        [ProtoMember(25)]
        public short[]? ShortArray { get; set; }
        [ProtoMember(26)]
        public ushort[]? UshortArray { get; set; }

        [ProtoMember(27)]
        public string? String { get; set; }
        [ProtoMember(28)]
        public string[]? StringArray { get; set; }
    }

    // https://stackoverflow.com/questions/71786891/create-a-list-of-numbers-in-bogus
    class BuiltInClassFaker : Faker<BuiltInClass>
    {
        public BuiltInClassFaker(int count)
        {
            RuleFor(o => o.Bool, f => f.Random.Bool());
            RuleFor(o => o.Byte, f => f.Random.Byte());
            RuleFor(o => o.Char, f => f.Random.Char());
            RuleFor(o => o.Decimal, f => f.Random.Decimal());
            RuleFor(o => o.Double, f => f.Random.Double());
            RuleFor(o => o.Float, f => f.Random.Float());
            RuleFor(o => o.Int, f => f.Random.Int());
            RuleFor(o => o.Long, f => f.Random.Long());
            RuleFor(o => o.Sbyte, f => f.Random.SByte());
            RuleFor(o => o.Short, f => f.Random.Short());
            RuleFor(o => o.Uint, f => f.Random.UInt());
            RuleFor(o => o.Ulong, f => f.Random.ULong());
            RuleFor(o => o.Ushort, f => f.Random.UShort());

            //RuleFor(o => o.BoolArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Bool()).ToArray());    // NetJSON 反序列化时索引越界
            RuleFor(o => o.ByteArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Byte()).ToArray());
            RuleFor(o => o.SbyteArray, f => Enumerable.Range(1, count).Select(_ => f.Random.SByte()).ToArray());
            //RuleFor(o => o.CharArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Char()).ToArray());    // NetJSON 反序列化时卡住
            RuleFor(o => o.DecimalArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Decimal()).ToArray());
            RuleFor(o => o.DoubleArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Double()).ToArray());
            RuleFor(o => o.FloatArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Float()).ToArray());
            RuleFor(o => o.IntArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Int()).ToArray());
            RuleFor(o => o.UintArray, f => Enumerable.Range(1, count).Select(_ => f.Random.UInt()).ToArray());
            RuleFor(o => o.LongArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Long()).ToArray());
            RuleFor(o => o.UlongArray, f => Enumerable.Range(1, count).Select(_ => f.Random.ULong()).ToArray());
            RuleFor(o => o.ShortArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Short()).ToArray());
            RuleFor(o => o.UshortArray, f => Enumerable.Range(1, count).Select(_ => f.Random.UShort()).ToArray());

            RuleFor(o => o.String, f => f.Lorem.Word());
            RuleFor(o => o.StringArray, f => f.Lorem.Words());
        }
    }
}

[rpgmaker]

Thanks for the details. Is net standard the only available version for unity? Do you have a .net 5 or 6 option available?

Thanks

[jckling]

Unity supports two .NET profiles: .NET Standard and .NET Framework. Each profile provides a different set of APIs so that C# code can interact with .NET class libraries. The Api Compatibility Level property has two settings:

.NET Standard: .NET Standard 2.1, as published by the .NET Foundation. .NET Framework 4.8, as published by Microsoft, plus additional APIs in .NET Standard 2.1.

Errors found in bool and char array are using .Net 7.0

[rpgmaker]

I will look into the char array and boolean issue. Thanks for clarification.

[rpgmaker]

@jckling , can you try to update to latest nuget package in a test project and see if using the 7.0 support works differently or same error? Want to see if it is related to code or versioning here.

[jckling]

Using Rider with .Net 7.0

test code

using System;
using System.Linq;
using Bogus;

namespace ConsoleApp1
{
    [Serializable]
    public class BuiltInClass
    {
        public bool Bool { get; set; }
        public byte Byte { get; set; }
        public sbyte Sbyte { get; set; }
        public char Char { get; set; }
        public decimal Decimal { get; set; }
        public double Double { get; set; }
        public float Float { get; set; }
        public int Int { get; set; }
        public uint Uint { get; set; }
        public long Long { get; set; }
        public ulong Ulong { get; set; }
        public short Short { get; set; }
        public ushort Ushort { get; set; }

        public bool[]? BoolArray { get; set; }
        public byte[]? ByteArray { get; set; }

        public sbyte[]? SbyteArray { get; set; }

        public char[]? CharArray { get; set; }
        public decimal[]? DecimalArray { get; set; }
        public double[]? DoubleArray { get; set; }
        public float[]? FloatArray { get; set; }
        public int[]? IntArray { get; set; }
        public uint[]? UintArray { get; set; }
        public long[]? LongArray { get; set; }
        public ulong[]? UlongArray { get; set; }
        public short[]? ShortArray { get; set; }
        public ushort[]? UshortArray { get; set; }

        public string? String { get; set; }
        public string[]? StringArray { get; set; }
    }

// https://stackoverflow.com/questions/71786891/create-a-list-of-numbers-in-bogus
    class BuiltInClassFaker : Faker<BuiltInClass>
    {
        public BuiltInClassFaker(int count)
        {
            RuleFor(o => o.Bool, f => f.Random.Bool());
            RuleFor(o => o.Byte, f => f.Random.Byte());
            RuleFor(o => o.Char, f => f.Random.Char());
            RuleFor(o => o.Decimal, f => f.Random.Decimal());
            RuleFor(o => o.Double, f => f.Random.Double());
            RuleFor(o => o.Float, f => f.Random.Float());
            RuleFor(o => o.Int, f => f.Random.Int());
            RuleFor(o => o.Long, f => f.Random.Long());
            RuleFor(o => o.Sbyte, f => f.Random.SByte());
            RuleFor(o => o.Short, f => f.Random.Short());
            RuleFor(o => o.Uint, f => f.Random.UInt());
            RuleFor(o => o.Ulong, f => f.Random.ULong());
            RuleFor(o => o.Ushort, f => f.Random.UShort());

            // RuleFor(o => o.BoolArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Bool()).ToArray());
            RuleFor(o => o.ByteArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Byte()).ToArray());
            RuleFor(o => o.SbyteArray, f => Enumerable.Range(1, count).Select(_ => f.Random.SByte()).ToArray());
            // RuleFor(o => o.CharArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Char()).ToArray());
            RuleFor(o => o.DecimalArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Decimal()).ToArray());
            RuleFor(o => o.DoubleArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Double()).ToArray());
            RuleFor(o => o.FloatArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Float()).ToArray());
            RuleFor(o => o.IntArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Int()).ToArray());
            RuleFor(o => o.UintArray, f => Enumerable.Range(1, count).Select(_ => f.Random.UInt()).ToArray());
            RuleFor(o => o.LongArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Long()).ToArray());
            RuleFor(o => o.UlongArray, f => Enumerable.Range(1, count).Select(_ => f.Random.ULong()).ToArray());
            RuleFor(o => o.ShortArray, f => Enumerable.Range(1, count).Select(_ => f.Random.Short()).ToArray());
            RuleFor(o => o.UshortArray, f => Enumerable.Range(1, count).Select(_ => f.Random.UShort()).ToArray());

            RuleFor(o => o.String, f => f.Lorem.Word());
            RuleFor(o => o.StringArray, f => f.Lorem.Words());
        }
    }

    class HelloWorld
    {
        static void Main()
        {
            var obj = new BuiltInClassFaker(5).Generate();
            Console.WriteLine(obj);
            var json = NetJSON.NetJSON.Serialize(obj);
            Console.WriteLine(json);
            var o = NetJSON.NetJSON.Deserialize<BuiltInClass>(json);
            Console.WriteLine(o);
        }
    }
}

bool array

ConsoleApp1.BuiltInClass
{"Bool":true,"Byte":28,"Sbyte":30,"Char":"ꍍ","Decimal":0.2982627934573370,"Double":0.9877039326613706,"Float":0.7254105,"Int":-1695955748,"Uint":3355254484,"Long":5847815957531525239,"Ulong":1816404614352237680,"Short":-8981,"Ushort":46954,"BoolArray":[true,true,true,true,false],"ByteArray":"Ymlh+k4=","SbyteArray":[-2,65,-106,-4,-117],"DecimalArray":[0.286156107560330,0.3863094135344980,0.05676680149352710,0.4238397186056560,0.9989409527449180],"DoubleArray":[0.5531192049917278,0.7292470320898853,0.3620456871266525,0.8930039869735799,0.5145119278584449],"FloatArray":[0.12715112,0.6715425,0.65436506,0.60259855,0.1611852],"IntArray":[-1611031272,1219647907,780716125,311672435,568657198],"UintArray":[3265113181,1171962044,997951470,2667764103,641310524],"LongArray":[-4329690052162482001,-3861709721239004129,-2086817730421647541,7627716376531507149,-3586591424862312451],"UlongArray":[18241688206578323456,8699176316600569856,13366425025817864192,7458019796484663296,17576731301674213376],"ShortArray":[-1072,-14454,-16031,7676,-29506],"UshortArray":[17175,973,813,28557,30430],"String":"nihil","StringArray":["qui","omnis","est"]}
Unhandled exception. System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at CreateListBooleanArray(Char*, Int32&, NetJSONSettings)
   at SetBuiltInClass(Char*, Int32&, BuiltInClass, String, NetJSONSettings)
   at CreateClassOrDictBuiltInClass(Char*, Int32&, NetJSONSettings)
   at ReadBuiltInClass(String, NetJSONSettings)
   at NetJSON.NetJSON.DynamicNetJSONSerializer`1.Deserialize(String value)
   at NetJSON.NetJSON.Deserialize[T](String json)
   at ConsoleApp1.HelloWorld.Main() in /Users/jck/RiderProjects/Solution1/ConsoleApp1/Program.cs:line 91

char array

ConsoleApp1.BuiltInClass
{"Byte":184,"Sbyte":-116,"Char":"�","Decimal":0.2037234687738470,"Double":0.6293818952604664,"Float":0.89884037,"Int":721772907,"Uint":2187413009,"Long":4357301589966590435,"Ulong":13818459595554727936,"Short":-19432,"Ushort":29177,"ByteArray":"5cyH1Ko=","SbyteArray":[-26,22,72,122,-102],"CharArray":["","梹","ꬨ","鳧","藪"],"DecimalArray":[0.587729138549780,0.7618116938549940,0.7021300654370190,0.49510673727977206474304494073650],"DoubleArray":[0.3081686452470578,0.023764916079639486,0.6753168145761712,0.8709820603312806,0.5675704183043394],"FloatArray":[0.5810342,0.7355462,0.36814243,0.7681243,0.8726531],"IntArray":[218087669,-1956842152,-1039566053,896801899,-750606515],"UintArray":[288714541,1988881873,2562028902,1558260100,1993126913],"LongArray":[82727026753674186,8477946062618346788,976624612840680068,-7771316551285596720,-3227631419420778134],"UlongArray":[8401532367503679488,13851819932396554240,16020947268532314112,7150320351557955584,3002079856888903680],"ShortArray":[24416,-9802,-9906,-30641,-28928],"UshortArray":[18943,51547,58816,13469,64339],"String":"maxime","StringArray":["quibusdam","atque","quam"]}
Out of memory.

Process finished with exit code 134.

[jckling]

Would it be some error with random generated data?

[rpgmaker]

It might be how I am doing allocation that might be causing it to do out of memory or the list creation is buggy therefore it is creating an object that is too large causing OOM.

I will need to debug it locally to figure out what is going on.

[wmjordan]

OOM might usually be caused by attempting to allocate an array with a negative size number.

[rpgmaker]

Is it possible that the int is not initialized and get passed to the method to construct a list object?

[jckling]

Is it possible that the int is not initialized and get passed to the method to construct a list object?

The json data seems right, and errors occur while deserializing

[rpgmaker]

I meant there is a bug in the code and issue is not the data. 👍

[wmjordan]

Is it possible that the int is not initialized and get passed to the method to construct a list object?

Typically no in the .NET world, unless:

1. The int is a shared variable (static field, or reused variable, etc) and the value remains previously assigned.
2. The InitLocals property of a MethodBuilder instance is specified to be false, then the local variable may not be zero at runtime.
3. The int comes from memory allocated or manipulated by unmanaged code.

I guess we can add a simple check into the CreateListBooleanArray method and throw an exception with the variable values which might cause trouble, and let @jck help run that version to gather the stack trace.

[rpgmaker]

Cool. I will try it out this weekend and see what works. 👍

[rpgmaker]

@jckling @wmjordan . I am still struggling to understand what I am missing for testing the issue. I can't reproduce it

Running the BuiltInClassFaker Test

Using .NET 7 to run all the test

Verified the configuration is using net7

I am going to push the test that I added for you review.

[rpgmaker]

Here is the link to the code changes that I added - https://github.com/rpgmaker/NetJSON/blob/master/NetJSON.Net7_0.Tests/NetJSONTest.cs#L43

https://github.com/rpgmaker/NetJSON/blob/master/NetJSON.Net7_0.Tests/TestClasses.cs#L57

[jckling]

@rpgmaker Char array and bool array are empty, you should uncomment either of these two lines.

https://github.com/rpgmaker/NetJSON/blob/d24e3e213b5e2f40c036aa3b132c77acbf7169e1/NetJSON.Net7_0.Tests/TestClasses.cs#L75

https://github.com/rpgmaker/NetJSON/blob/d24e3e213b5e2f40c036aa3b132c77acbf7169e1/NetJSON.Net7_0.Tests/TestClasses.cs#L78

[rpgmaker]

Thanks, will check it out.

[wmjordan]

It might be how I am doing allocation that might be causing it to do out of memory or the list creation is buggy therefore it is creating an object that is too large causing OOM.

I read the post before the above statement and saw that the exception was not OutOfMemoryException but IndexOutOfRangeException only. Things are quite different.

If it is IndexOutOfRangeException, it is usually caused by wrong pointer position calculation. I met with that kind of exception very frequently while I was dealing with some binary byte array deserialization with my own library. I usually catch the exception thrown from the dynamic deserializer and attach the pointer position and the array length information into the exception. So I can verify which byte is causing trouble and determine whether the deserializer should throw the frame away or bubble up the exception.

EDIT: array length above should be changed to array segment.

[rpgmaker]

@wmjordan @jckling . I fixed both issues. I will check in soon once i rerun all the other tests.

First issue i was missing "GenerateUpdateCurrent" and this special scenario never got handle since most folks stored complex object in array

The second issue was missing character type in the list of string type, so it did not know how to handle and terminate the reading of the string

[rpgmaker]

@wmjordan @jckling . I have pushed the fix and included more test from the .net 4 into .net 7 that should valid 131 scenarios.

Could you please test it again and let me know. So I can patch the current version.

[jckling]

It's fixed, both bool array and char arry can be deserialized correctly.

[rpgmaker]

Awesome. I just published the fix in 1.4.3 - > https://www.nuget.org/packages/NetJSON