OutOfMemory Exception thrown during Serialization

[XeClutch]

Please forgive the masking of the true names of my fields & data types. This is a piece of closed-source proprietary corporate software and I can't risk potentially exposing any of our trade secrets to our aggressive competition. The application where this code lives is our "secret sauce".

For some context, this normally works but since implementing multi-threading, we're seeing this occur after 3 hours and 10 minutes of continuous runtime.

NOTE: In the Diagnostic Tools window on the right-hand side of the screen the runtime is incorrect. The actual runtime is 3h 11m 15s. NOTE: In the "Process Memory" section of the Diagnostic Tools on the right-hand side, this has been moved over to the left and isn't showing the memory utilization at the time of the crash. See below for that screenshot.

During normal operation, the application does not use more than ~300MB of memory. Once we've been running for over 3 hours, this spikes to >3GB when calling JsonConvert.SerializeObject.

IMPORTANT!: My primary development machine is Windows-based, with 16GB of physical memory. Our server where this software is typically deployed is a Linux machine with 8GB of physical memory. The issue persists on both.

Source/destination types

public class TestClass1 {
    public string ID; // 40 character string

    public string str1; // typically no more than 1-3 words
    public string str2; // does not exceed 20 characters
    public TestClass2[] tst1; // average .Length = ~30, highest .Length observed = ~80 (can be higher)
    public string str3; // no real limit on the length of this string, often gets up to 1,000+ characters
    public string[] str4; // (collection of URL's) average .Length = 4, highest .Length observed = 12
    public string str5; // typically no more than 10 words
    public string str6; // does not exceed 20 characters

    public void Clean() {
        str1 = str1.Decode();
        str3 = str3.Decode();
        str5 = str5.Decode(); // Decode() is an in-house string extension to remove HTML formatting from a string
    }

    public bool IsValid() {
        return !string.IsNullOrWhiteSpace(str2) &&
                str2 != "N/A" &&
                tst1 != null &&
                tst1.Length > 0 &&
                !string.IsNullOrWhiteSpace(str6) &&
                str6 != "N/A";
    }
}

public class TestClass2 {
    public int i1 = 0;
    public string str1 = "";
    public string str2 = "";

    public string str3;
    public int i2;
    public int i3;
    public float f1;
    public string str4;
    public float f2;
    public int i4;
    public float f3;

    public bool b1 = false;

    // Methods
    public bool Equals(ProductCompatibility compare) {
        if (compare == null) {
            return false;
        }

        if (i1 == compare.i1 &&
                str1 == compare.str1 &&
                str2 == compare.str2 &&
                str3 == compare.str3 &&
                i2 == compare.i2 &&
                i3 == compare.i3 &&
                f1 == compare.f1 &&
                str4 == compare.str4 &&
                f2 == compare.f2 &&
                i4 == compare.i4 &&
                f3 == compare.f3 &&
                b1 == compare.b1) {
            return true;
        } else {
            return false;
        }
    }
    public override bool Equals(object obj) {
        if (ReferenceEquals(this, obj)) {
            return true;
        }

        if (ReferenceEquals(obj, null)) {
            return false;
        }

        return false;
    }
    public override int GetHashCode() {
        return (
            i1, str1, str2, str3,
            i2, i3, f1, str4,
            f2, i4, f3
        ).GetHashCode();
    }

    // Operators
    public static bool operator ==(ProductCompatibility left, ProductCompatibility right) {
        return left.Equals(right);
    }
    public static bool operator !=(ProductCompatibility left, ProductCompatibility right) {
        return !left.Equals(right);
    }
}

Source/destination JSON

[
    {
        "ID": "40 character string XXXXXXXXXXXXXXXXXXXX",
        "str1": "1-3 full words",
        "str2": "sub-20 character string",
        "tst1": [
          {
            "i1": 0,
            "str1": "1-2 words",
            "str2": "1-2 words",
            "str3": null,
            "i2": 0,
            "i3": 0,
            "f1": 0.0,
            "str4": null,
            "f2": 0.0,
            "i4": 0,
            "f3": 0.0,
            "b1": false
          },
        ],
        "str3": "Long string, not uncommon to get past 1,000 characters",
        "str4": [
          "https://github.com/JamesNK/Newtonsoft.Json",
          "https://google.com/"
        ],
        "str5": "1-10 full words",
        "str6": "sub-20 character string"
    },
]

Expected behavior

JsonConvert.SerializeObject does not spike memory usage 10x.

Actual behavior

When calling JsonConvert.SerializeObject, memory utilization spikes from ~250MB up to 3GB+, creating an OutOfMemory exception. On Linux, the process is killed and the system error log reports that the application was attempting to use over over 250GB of memory.

Steps to reproduce

List<TestClass1> sample = new List<TestClass1>();

//...

lock (lockObject) {
    if (sample.Count >= 1000) { // tried this in increments of 1000, 2500, 5000 and 10000
        if (Export(sample.ToArray())) {
            sample = new List<TestClass1>();
        }
    }
}

private bool Export(object[] obj) {
    string json = Serialize(obj);

    // write "json" to disk
}
private string Serialize(object obj) {
    return JsonConvert.SerializeObject(obj, Formatting.Indented, new JsonSerializerSettings {
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
    });

    // I have also tried without specifying any JsonSerializerSettings
}

Regarding the comment made about increments of 1000, 2000, etc - This issue still occurs after roughly 3hrs regardless of the increment.

[XeClutch]

Forgot to mention in the thread that I'm currently using v13.0.3 via NuGet.

[dbc2]

I notice that you are serializing your object array to a string, then writing that string to disk:

private bool Export(object[] obj) {
  string json = Serialize(obj);

  // write "json" to disk
}

Newtonsoft's documentation page Performance Tips: Optimize Memory Usage recommends that you should not do that and suggests serializing directly to a Stream instead:

To keep an application consistently fast, it is important to minimize the amount of time the .NET framework spends performing garbage collection. Allocating too many objects or allocating very large objects can slow down or even halt an application while garbage collection is in progress.

To minimize memory usage and the number of objects allocated, Json.NET supports serializing and deserializing directly to a stream. Reading or writing JSON a piece at a time, instead of having the entire JSON string loaded into memory, is especially important when working with JSON documents greater than 85kb in size to avoid the JSON string ending up in the large object heap.

For examples of serializing to directly to files, see the documentation pages Serialize JSON to a file and Serializing and Deserializing JSON: JsonSerializer.

If you make the change as suggested by Newtonsoft, does that resolve your problem?