search

SmartlyDressedGames / Unturned-3.x-Community

Community portion of the Unturned-3.x repo. If you have access to the source code you can find it here:
https://github.com/SmartlyDressedGames/Unturned-3.x/
88 stars 18 forks source link

Optimisation for EconInfo.json storage #4213

Closed archie426 closed 3 weeks ago

archie426 commented 11 months ago

While on my previous performance testing adventure, I found another area that could be considerably optimised. EconInfo.json is 4.9mb and takes 8seconds (1.5% of time to load and join a server) to be deserialised. Whilst JSON is good for storing persistent data, I think this is too much for its purpose. Instead, using Avro with the Chr.Avro library would be far faster as it takes up to 12x less time. It is not human readable once formatted, but I don't believe EconInfo is something that needs changing in at least legitimate circumstances.

I don't know whether there are other files that this could be applied to, but this is the worst for performance for all non-asset I/O operations. From the performance statistics that I linked above, Newtonsoft is the slowest library, so if you were to stick to JSON it would be preferable to use System.Text.Json as that appears to give a slight performance boost.

DanielWillett commented 11 months ago

You may could also just look into using manual serialization for that file, may help some. https://www.newtonsoft.com/json/help/html/Performance.htm There are some other tips here about improving performance too but at the bottom it shows an example of manual serialization. More info here: https://www.newtonsoft.com/json/help/html/ReadingWritingJSON.htm

If you do switch to System.Text.Json, keep in mind it can be frustrating to get working with Unity now that .NET standard 2.1 is supported (since many plugins are also switching to .NET Standard). Versions 4.7.0 and 4.7.1 (netstandard2.0) seem to be the only versions that are compatible with the version of System.Threading.Tasks.Extensions and System.Memory that unity's facades map to.

Only slight issue for you i think is that 4.7.X doesn't support fields, only properties, so any public fields would have to be given a property.

Best way I've found to get it working (add the following libraries):

While yes you can use the .NET framework (net461) versions to avoid lowering the version so much, any plugins using .NET Standard 2.1 to compile (quite a few now) won't be able to interact with it at all beacuse the later versions of System.Text.Json pull from newer versions of System.Memory and System.Threading.Tasks.Extensions than available in Unity's mappings.

SDGNelson commented 11 months ago

There isn't currently any reason it needs to be json. Originally, it was json because the Steam Inventory item definition files were json, and the game used it as a fallback for reading properties from Steam's item data. Especially after switching over to GUIDs I have wanted to convert it to a binary format or perhaps just use Steam's API again now that it's more reliable. Reading a binary file would be significantly faster.

archie426 commented 11 months ago

Considering what you said about wanting to make files binary, would it perhaps be faster to do the same for bundle files? Currently loading them takes up a considerable amount of time, especially the deserialisation part. It would be perhaps possible to have a sort of optional compilation model:

  1. Files are developed as .dat by modders
  2. These files may then be compiled into a binary format
  3. People may decompile, edit, copy, recompile etc these files as .dat files
  4. Unturned loads either the .dat file or the binary file on start up, depending which one is in the asset folder

This could also give way to GUI binary editors, which would make developing these files even more user friendly than they were originally. I would happily make this myself, certainly not volunteering anyone to do it.

Obviously the .dat format is not the same as JSON, so I don't know how much performance boost one would get, but I feel as if it would be ultimately better. Currently in total, TryGetValue in the DatParser takes 3.6seconds, asset population as a whole takes around 13seconds on my machine. Currently this works on a basis of checking if a key is there, and then populating if it is. Well, if it was stored in binary the initial check need not be there, rather just be a single byte of 0. Signalling with binary is in general far faster than checking for keys, separating lines and finding whitespaces/indentations etc

SDGNelson commented 1 month ago

As part of adding some additional info to this file I took the opportunity to change it to binary serialization for the next update. 🙂