New persistence strategy for better cross-version compatibility

[Elesbaan70]

Note - This issue originally called for the use of Json, which was determined be unfeasible due to the lack of a good full-featured Json serializer that is compatible with Unity.

Describe your idea

In TMPE's current persistence strategy, the introduction of a new type into the persistent data will break backward compatibility. Although old versions are not supported, players reverting from alpha to stable should not lose savegame data needlessly.

The same issue also affects forward compatibility in ways that restrict future development choices. Ideally, the save data would not be bound to specific types. A move away from strongly typed serialization would give everyone more flexibility moving forward.

Requirements

1. This is an opt-in change. All existing serialization will remain untouched until there is a reason to do it the new way, as determined feature by feature.
2. A manager-based persistence mechanism in which SerializableDataExtension enumerates a collection of managers and calls LoadData() and SaveData() methods basically like the ones we have today.
3. SerializableDataExtension will save the data from each manager separately, to isolate them and minimize the risks associated with changes to persistent data.
4. A manager will have a means of identifying other managers upon which its data depends. This will determine the enumeration order during load and save.
5. Linq to XML will be used for persistence.

TrafficLightSimulationManager will be the first converted to the new persistence strategy, since displaced lane support was the breaking change that prompted this issue. Its conversion to the new persistence strategy will be delivered separately from and prior to displaced lane support.

Tasks

• [x] Persistence framework
• [x] Add a DOM to the serialized data and make a mechanism for opt-in transition away from the old serialization strategy (SerializableDataExtension)
• [x] Update timed traffic lights to be saved into the DOM (half-baked persistable model)
• [x] Update junction restrictions to be saved into the DOM (fully persistable model)
• [x] Compress XML data, with fallback to uncompressed if it fails (especially on some Linux distros)
• [x] Test versioning in a relevant feature branch such as https://github.com/Elesbaan70/TMPE/tree/lane-grouping
• [x] Final code cleanup
• [x] Wiki articles for future development
• [x] Mark #1559 ready for review

[kianzarrin]

we can ask in the harmony discord about this. they probably know something.

[Elesbaan70]

Since we have limited size to be stored, @Elesbaan70 have you measure worst case scenario, how much data we really need to store - assuming we use container per manager?

Two-step light on a 4-way node without complex lane groupings: 2309 bytes, or about 7,000 total stoplights of this configuration.

The more I think about this, the more I realize it's all or nothing. Otherwise we end up with savegames that some people can't read.

Here's a possible ugly way of addressing the issue:

1. Immediately before doing a compression operation, write something into the global options that says we're doing compression stuff.
2. Clear it in a finally block.
3. Check this flag at startup, and if it is set, explain to the user what happened and provide a link to the installer they need.

[Elesbaan70]

This code is basically ready to go. But the Linux compression issue needs to be worked out.

[Elesbaan70]

I am reluctant to rely on an external library for a core feature. Also, a third-party compression library will be managed code, and so it will be slower, and compression is the kind of thing where that matters.

So, what I'm looking at for the compression issue is:

1. Run some tests to verify zlib's compatibility with .net's built-in compression.
2. Add an option that let's you pick native (.net) or managed (zlib) compression, with managed being the default on Linux, and native the default on all other platforms.
3. Test that both work and are fully compatible with each other.
4. Have someone verify that zilb works on Linux.

[kianzarrin]

its not like we have a better option.

[Elesbaan70]

@krzychu124 finally got around to mentioning that there's a compression library distributed with CS. Wish I'd known that earlier! LOL

[Elesbaan70]

Here is the wiki page. I'll improve this based on feedback.

Loading and Saving Data

[krzychu124]

@krzychu124 finally got around to mentioning that there's a compression library distributed with CS. Wish I'd known that earlier! LOL

I thought you've tried "obvious" solution and it hadn't worked😂 I haven't tried using it for other things than collecting files from disk to create a zip file.