Switch from Newtonsoft.Json to System.Text.Json

[Morilli]

This switches the entire codebase from using Newtonsoft.Json to System.Text.Json. This is probably more future-proof, supported by .NET directly and uhh see #2261 I guess. It would be nice to be able to load older configs with less ugly exception boxes that delete your entire config if you don't read it carefully enough, and currently all 2.9.1 configs and .tasprojs will fail to load due to the EmuHawk->BizHawk.Client.EmuHawk assembly name change. Polymorphism and deserializing based on a given $type in the json is not really supported in System.Text.Json, but this is by design and for security reasons. I've changed the code to work without this parameter and because this $type caused the assembly name mismatches, this should also fix all config errors.

There's a couple caveats and potential failure cases here:

• Newtonsoft.Json and System.Text.Json are naturally not 100% interchangable; a couple important differences are listed here: https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/migrate-from-newtonsoft.
• System.Text.Json does not include non-public fields and properties of objects in serialization, and that behavior can only be changed by applying the [JsonInclude] attribute to each non-public field or attribute, there is no serializer option for it.
• As in Newtonsoft.Json, System.Text.json serializes byte arrays as base64-encoded strings, which means we still need to code our own converter to get them serialized as number arrays.
• There are hardcoded conventions for how certain interface types are handled, for example a class implementing IEnumerable<T> will be serialized as IEnumerable<T>, ignoring all class field and properties entirely. There is no straightforward way to change this, the suggested workaround is to implement a full JsonConverter for each affected class and decorate the class with it. This is... unfortunate to say the least, see https://github.com/dotnet/runtime/issues/63791 and related issues for discussion around it. Given the issues age, this is most likely not going to be fixed soon.
• Multi-dimensional arrays seem to not be serializable by default in System.Text.Json, so I had to write a JsonConverter for them. Additional headache when you have a multi-dimensional byte array: You cannot decorate the field with both the multidimensional array converter and the byte array converter, so either the multidimensional array converter needs to hardcode its serialization behavior and choose the number array or base64 string representation, or the serializer for the parent class needs to be passed the byte array converter if desired (this is currently the case).
• Float serialization is a bit garbage on .net framework; passing certain float values can result in too much precision in the string representation. Apparently .net framework's default float/double ToString behavior is inaccurate and does not guarantee round-trip safety, so to counter that System.Text.Json uses a method that guarantees enough precision, but often overshoots. I've implemented a simple FloatConverter to handle this case nicely, but doubles may have the same issue.
• Serialization order of elements could be different, like with private fields manually [JsonInclude]d.

Now, having considered and handled most of the above points, some of those issues are hard to spot and/or invisible until an existing type is serialized and exhibits unexpected behavior or serialization failure.

TODO:

• create more test cases to test the added converters, like multi-dimensional arrays, byte array serialization, round-trip serialization for floats / doubles done
• check all existing serialized classes to ensure they are all serialized the same as before and there are no mismatches in representation, missing fields, extra fields, wrong field names, etc.

Check if completed:

• [x] I have run any relevant test suites
• [x] I, the committer, have read the licensing terms for contributors (last updated 2024-03-20) and am compliant

[kalimag]

Tool settings can't be deserialized yet

System.ArgumentException
  HResult=0x80070057
  Message=Object of type 'System.Text.Json.JsonElement' cannot be converted to type 'BizHawk.Client.EmuHawk.LuaConsole+LuaConsoleSettings'.
  Source=mscorlib
  StackTrace:
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index) in f:\dd\ndp\clr\src\BCL\system\reflection\propertyinfo.cs:line 643
   at BizHawk.Client.EmuHawk.ToolManager.InstallCustomConfig(IToolForm tool, Dictionary`2 data) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 422
   at BizHawk.Client.EmuHawk.ToolManager.Load[T](Boolean focus, String toolPath) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 145
   at BizHawk.Client.EmuHawk.MainForm.OpenLuaConsole() in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.cs:line 1543
   at BizHawk.Client.EmuHawk.MainForm.LuaConsoleMenuItem_Click(Object sender, EventArgs e) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.Events.cs:line 1192

[Morilli]

Tool settings can't be deserialized yet

System.ArgumentException
  HResult=0x80070057
  Message=Object of type 'System.Text.Json.JsonElement' cannot be converted to type 'BizHawk.Client.EmuHawk.LuaConsole+LuaConsoleSettings'.
  Source=mscorlib
  StackTrace:
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index) in f:\dd\ndp\clr\src\BCL\system\reflection\propertyinfo.cs:line 643
   at BizHawk.Client.EmuHawk.ToolManager.InstallCustomConfig(IToolForm tool, Dictionary`2 data) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 422
   at BizHawk.Client.EmuHawk.ToolManager.Load[T](Boolean focus, String toolPath) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 145
   at BizHawk.Client.EmuHawk.MainForm.OpenLuaConsole() in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.cs:line 1543
   at BizHawk.Client.EmuHawk.MainForm.LuaConsoleMenuItem_Click(Object sender, EventArgs e) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.Events.cs:line 1192

should be fixed with the latest commit

[kalimag]

should be fixed with the latest commit

Seems to work. Some tools now fail with different errors:

System.NotSupportedException
  Message=Deserialization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is not supported. Type 'BizHawk.Client.EmuHawk.RollColumn'. Path: $.Columns[0] | LineNumber: 2 | BytePositionInLine: 11.
  Source=System.Text.Json
  StackTrace:
   at System.Text.Json.JsonSerializer.Deserialize(JsonElement element, Type returnType, JsonSerializerOptions options) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Element.cs:line 63
   at BizHawk.Client.EmuHawk.ToolManager.InstallCustomConfig(IToolForm tool, Dictionary`2 data) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 407
   at BizHawk.Client.EmuHawk.ToolManager.Load[T](Boolean focus, String toolPath) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 145
   at BizHawk.Client.EmuHawk.MainForm.OpenLuaConsole() in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.cs:line 1543
   [...]

System.Text.Json.JsonException
  Message=The JSON value could not be converted to System.Drawing.Color. Path: $.Background | LineNumber: 1 | BytePositionInLine: 23.
  Source=System.Text.Json
  StackTrace:
   at System.Text.Json.JsonSerializer.Deserialize(JsonElement element, Type returnType, JsonSerializerOptions options) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Element.cs:line 63
   at BizHawk.Client.EmuHawk.ToolManager.InstallCustomConfig(IToolForm tool, Dictionary`2 data) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 407
   at BizHawk.Client.EmuHawk.ToolManager.Load[T](Boolean focus, String toolPath) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\tools\ToolManager.cs:line 145
   at BizHawk.Client.EmuHawk.MainForm.HexEditorMenuItem_Click(Object sender, EventArgs e) in H:\Projekte\Visual Studio\BizHawk-upstream\src\BizHawk.Client.EmuHawk\MainForm.Events.cs:line 1213
   [...]

  This exception was originally thrown at this call stack:
    System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ExpectedString(System.Text.Json.JsonTokenType) in ThrowHelper.cs
    System.Text.Json.Utf8JsonReader.GetString() in Utf8JsonReader.TryGet.cs
    BizHawk.Emulation.Common.Json.TypeConverterJsonAdapter<T>.Read(ref System.Text.Json.Utf8JsonReader, System.Type, System.Text.Json.JsonSerializerOptions) in TypeConverterJsonAdapter.cs
    [...]

Inner Exception 1:
InvalidOperationException: Cannot get the value of a token type 'StartObject' as a string.

[Morilli]

^ That first error should be fixed with 16a8e923d4834de396df2a899ac8dfec3e0da3d1, and the second is due to an incorrect config generated from a commit without the TypeConverter handling, so you'll need to clear that part of the config and let it regenerate.

[kalimag]

^ That first error should be fixed with 16a8e92

The exception is gone, but the column display is broken because of the properties with private setters. This seems to work though:

[JsonConstructor]
private RollColumn(string name, string text, ColumnType type)
{
    Name = name;
    Text = text;
    Type = type;
}

and the second is due to an incorrect config generated from a commit without the TypeConverter handling, so you'll need to clear that part of the config and let it regenerate.

Right, my bad.

[Morilli]

You'll also need to clear the RollColumn config to let it regenerate, that did work fine when I tested it.

[kalimag]

Doesn't work for me

1. Delete config
2. Start BizHawk
3. Open and close RAM Search
4. Restart BizHawk
5. Open RAM Search again
6. RAM Search looks like this

7. Close RAM Search and save config/close BizHawk
8. Config section looks like this:

  "CustomToolSettings": {
    "BizHawk.Client.EmuHawk.RamSearch": {
      "Settings": {
        "Columns": [
          {
            "Width": 60,
            "Left": 0,
            "Right": 60,
            "Name": null,
            "Text": null,
            "Type": 0,
            "Visible": true,
            "Emphasis": false,
            "Rotatable": false
          }
        ],