JamesNK / Newtonsoft.Json

Json.NET is a popular high-performance JSON framework for .NET
https://www.newtonsoft.com/json
MIT License
10.76k stars 3.25k forks source link

Cannot Deserialize byte[], which has been Serialized by NewtonJson #1932

Open mhaghighat20 opened 5 years ago

mhaghighat20 commented 5 years ago

Source/destination types

var data = new List<byte[]>();
            data.Add(new byte[300 * 1024 * 1024]);  //300 MB
            data.Add(new byte[300 * 1024 * 1024]);  //300 MB

Expected behavior

Deserialize correctly

Actual behavior

System.OutOfMemoryException: 'Array dimensions exceeded supported range.' at Newtonsoft.Json.Utilities.BufferUtils.RentBuffer(IArrayPool`1 bufferPool, Int32 minSize) in \ConsoleApp30\Newtonsoft.Json\Utilities\JavaScriptUtils.cs:line 48

Steps to reproduce

class Program
    {
        static void Main(string[] args)
        {
            var test = new Test();

            var serializedBinary = test.SerializedLargeData();
            var deserialiedObject = test.DeserializeLargeData(serializedBinary);

            Console.WriteLine("Finish");
            Console.ReadLine();
        }
    }

    public class Test
    {
        public byte[] SerializedLargeData()
        {
            var data = new List<byte[]>();
            data.Add(new byte[300 * 1024 * 1024]);  //300 MB
            data.Add(new byte[300 * 1024 * 1024]);  //300 MB

            var serializer = new JsonSerializer();
            using (var stream = new MemoryStream())
            {
                using (var writer = new StreamWriter(stream))
                {
                    serializer.Serialize(writer, data);
                }
                stream.Flush();
                return stream.ToArray();
            }
        }

        public List<byte[]> DeserializeLargeData(byte[] serializedBinary)
        {
            var serializer = new JsonSerializer();
            using (var stream = new MemoryStream(serializedBinary))
            {
                using (var reader = new StreamReader(stream))
                {
                    return (List<byte[]>)serializer.Deserialize(reader, typeof(List<byte[]>));
                }
            }
        }
    }
albracko commented 3 years ago

I have a similar issue...it seems that JSON serializers generally have issues with big JSON objects or fields with lots of data (Microsoft JSON serializer has same issue). I have a JSON with a field that contains about 600MB of HEX string, and encounter same issue as reported here. Also memory usage spikes to 3.3GB for such JSON, so almost 6x more than data itself, which would suggest there is a lot of data copying included.