Change the Saved Game Format

[BenjamenMeyer]

Presently the game uses a CSV format, and the former maintainers tried to also split the CSV data into multiple CSV files which we've found to be problematic (see https://github.com/vegastrike/Assets-Production/pull/7 for details).

Let's move to a different save game format.

Currently JSON, TOML, and YML are all currently excepted formats. However, only JSON has native support in Python (even Py3) without requiring additional libraries so if we need to open it from within the Python code, then we should limit it to the JSON format. (NOTE: JSON and TOML are both related to YML; I don't know why Python doesn't have a built-in YML parser.)

Tasks:

• [ ] Decide on the format to use
• [ ] Make the saved game file functionality a specific function of the game engine that is used by the assets so the assets don't have to be concerned about the file format, but can instead just do a load/save with the game engine.
• [ ] Make any conversion tools required to upgrade old file formats to the new file format.
• [ ] Provide a saved game editor tool that works with both the old and new formats, perhaps even providing a delta between the two so folks can ensure that the conversion worked properly.

[pyramid3d]

My recommendation:

Serialization Formats

There are many formats available for serialization, the most commonly used being CSV, XML, JSON, YAML.

Our requirement for data serialization is that it would be:

• Editable with a text editor
• Human readable
• Machine readable
• Compact even for large data sets
• Libraries available in popular programming languages

Following the analysis and recommendations in this article, we equally recommend JSON as the select serialization data format for fulfilling all of the above criteria better than other formats.

[1] CSV vs XML vs JSON – Which is the Best Response Data Format?. Digital Hospital. 2016. Retrieved on 2020-02-23.

Source: https://github.com/LibreGamingManifest/libre-gaming-manifest/blob/master/libre-gaming-manifest.md#serialization-formats

[stephengtuggy]

Cool. Sounds like JSON might be the way to go then.

[stephengtuggy]

Question: Are we making the correct distinction between the CSV format of the assets (which don't change) vs. the saved game format?

[nabaco]

I think @BenjamenMeyer meant the CSV assets table, since the savefile doesn't seem to be a CSV file at all. In that regard, I would think about YAML, since it's more human friendly, especially for editing. And it is a superset of JSON, so moving from one to another should be easy.

I would also add to this discussion the game's config file.

Some more info on that: https://www.json2yaml.com/yaml-vs-json http://stackoverflow.com/questions/1726802/ddg#1729545

[BenjamenMeyer]

So this particular issue is the Save Game format itself; however, we might want to expand it towards what all the file formats should be. There is also some benefit to using one format versus a dozen formats as it can simplify any tool chains.

@nabaco my only issue with YAML is that it requires an additional library for Python to load it natively, and for how we use Python it'd be best to keep to just what is built-in if we can. JSON fulfills that requirement. Otherwise I wouldn't have any objection to YAML.

[BenjamenMeyer]

I was editing a saved game today and discovered that we have a couple places where saved game data gets saved - not just the ~/.vegastrike/save/<game> file but also some data in ~/.vegastrike/serialized_xml/<saved game>/ for additional things, like ships.

We should bring this down to one location - one file. Perhaps a good way to do it would be to adopt a zip file like format similar to the ODT specification so that all saved game data is stored on disk within a single file, but we actually have multiple files within it.

At the very least, we should moved everything under ~/.vegastrike/saved/ and turn the file into a directory so everything is stored together.

Either way, I propose we do:

\saved <- maps to the current saved game file
\ships\<ships> <- the ship data from the current serialized_xml directory

This at least brings things into one location.

We probably will want to do this for for 1.0 even if we don't otherwise change the data formats of the "internal" files.

[BenjamenMeyer]

we should probably do anything of that nature as a dedicated library; perhaps have some interfaces that could be adhered to for serializing/deserializing data from it so we can move the core of the file data holding out, but still let the internal format be controlled by the existing code.