Saving and loading games

[byorgey]

It should be possible to save the current state of your game, and then later to load from a saved game. There is nothing theoretically difficult about this, but it will take a good deal of engineering. Off the top of my head, a save would need to include.

• A version number, so that we can do something reasonable with saves created by a previous version of the game.
• The current state of the world:
  • The seed used to generate the world
  • Which tiles have been loaded
  • The contents of those tiles (either the complete contents, or perhaps just a catalog of the cells whose entity contents have changed from their initial value)
• All (active and inactive) robots, including their CEK machine state.
• Note, when saving entities (either in the world map, or in an inventory, or a robot entity), if the hash matches the hash of a standard entity, we only have to save the hash. Otherwise we must serialize the entire entity.
  • Note that for this to work well on old save files, we need to keep old/outdated entities around (perhaps in a separate old_entities.yaml file) so that we can load them.
  • Alternatively (this might be simpler), every save file could just contain a complete description of every entity which needs to be referenced?
• Probably various other fields of the GameState and/or UIState.

For loading entities and recipes, we use .yaml files, so that they can be easily human-editable. However, for saving the entire game state we definitely don't want to do that. Instead we should use some kind of binary serialization framework so save files will be as compact as possible. Since we can already load scenarios from .yaml files, it makes sense at this point to just double down on that and make save files use the same format.

Once we enable this feature, we'll have to be very careful about versioning things such as entities and recipes, world generation, etc., and keep around old versions of things (as much as is reasonable) so that we can correctly load save files produced by older versions of the game.

Note, a related issue is #7 .

[xsebek]

Thanks for making this issue! I think a more directly related issue is Generate world by tiles (#29).

I could then save loaded tiles and generate new ones in whatever is the current world generation function. Thus, if I save the game on v1.41.0 and load it on v1.42.0, I could send robots to explore tiles with new features from version 1.42.0.

If the save was done per square, then changing world generation could introduce problems (:mountain:), but with tiles, I do not care what would have been on the unexplored ones in the previous version.

TLDR I would argue for the world state to be the seed and complete tiles, so versioning world generation is not a headache.

[byorgey]

Hmm, thanks, that's definitely worth considering! That would certainly simplify the need to keep track of versions for world generation. On the other hand, it might create weird seams at tile boundaries where e.g. half a lake on a previously loaded tile is suddenly right next to half a mountain on a new tile. But I'm not sure how important that is.

[xsebek]

I wonder if we could dramatically simplify this by implementing #29, #7 and give each entity a unique character and actually try storing the game as YAML.

- robots:
  ...
- seed: 0
- world:
  - tile:
    - x: 0
    - y: 0
    - map: >
      +------------------------+
      |   *       AAA    TTT   |
      | ~~ *    AAAÅÅ     TT   |

Most users likely will not change that many tiles, so the map will not take much space.

I guess with #7, robot definitions should not be an unbound number of lines either, but I do not know how many robots would need to keep a non-default state and how long that would be.

I still see this as viable option with some pros:

• the whole save might be less than a 1000 lines
• human editable
• simplifies #15
• simplifies #112
• easier to implement
• simple to support minor versions
• space conscious users can use text file compression

[byorgey]

I suppose it could be worthwhile to try a text-based solution. At the very least it opens up a fun new avenue for cheating ;-).

As for giving each entity a unique character, I think that would be rather restrictive (for example with #186 we now have multiple kind of mountain/vein entities with the same display, which is on purpose). But I don't think we have to do that: we could store a separate "entity palette" which lists characters and what entity they correspond to in this save file. We can try hard to use the same characters that are used for the actual display, so the saved map will mostly look the same as in the game. But in cases where multiple entities have the same character, we can just arbitrarily generate new unique characters for some of them.

Note that for the game state of robots we have to store their entire CEK machine state, not just their definitions.

[sullyj3]

Maybe this could be an option?

https://hackage.haskell.org/package/store-0.7.12/docs/Data-Store.html

[byorgey]

Ah, yes, that seems like it could be a good option if we go with a binary format (as opposed to the text-based format @xsebek is advocating for above). I didn't know about that package, thanks for the link @sullyj3 !

[byorgey]

As for versioning, https://hackage.haskell.org/package/DataVersion seems like an interesting option. I am not sure if this is too crazy/overkill.

[byorgey]

We seem to have continued moving in the potential direction of using ToJSON/FromJSON instances to save things in .yaml files. This makes sense as we have already invested a bunch in reading scenarios in that format. I think one concrete next step would be to focus on the ability to round-trip values through ToJSON and then back via FromJSON (we should add some tests to this effect). Many instances currently just derive instances via Generic but I suspect that will need to change in some cases.

• For example, Env containues Values which contain VRefs that reference a Store. Perhaps it works fine to just serialize the Store and the Values verbatim. However, we could also use it as an opportunity to do a bit of garbage collection (which we don't currently do at all).

[xsebek]

For a simpler example, the robot JSON instance which we can see using the web interface is too verbose.

The problem is mainly the inventory and installed devices. What I would like is for it to look like the scenario format. Getting a list of entities with unique names separately would IMHO make everything more readable.

---

^{Currently, new entities are not created during the game, but we will always be able to create unique names for them. It is just something we should keep in mind if we wanted to design the JSON format this way instead of inlining the entities. :slightly_smiling_face:}

[xsebek]

As an alternative idea, we could let JSON be fully inlined, but in YAML we would use aliases. Something like:

installed:
  - *entities.copper_wire

I do not think the expanded JSON is worth it, but it would make some JSON queries simpler. :shrug:

[xsebek]

@byorgey shall we update the Issue description and go with YAML? :slightly_smiling_face:

It would be a lot simpler for us. Also, we could use it for scenarios and the web API. That way we can incrementally create the ToJSON(E) instances and use them before having full game saves.

We could make this a Meta Issue and link other Issues that we need to finish first. Honestly this task looks so big that it scares me and I hope that splitting it into smaller tasks will make it less scary. :sweat_smile:

[valyagolev]

given the programming focus of the game, a fun suggestion could be to use Dhall

[sullyj3]

~Is the point of Dhall not to reduce the maintenance burden for manually written configuration? For a save game, the program serializes it, the program deserializes it, and at no point should a human need to edit it. This seems to obviate the need for functions and Dhall, and a simpler solution like yaml makes more sense to me~

Actually this whole objection doesn't apply, I hadn't realised user-editability was a goal.

[xsebek]

@valyagolev @sullyj3 Dhall is certainly an interesting suggestion, but I do not immediately see what it would be good for. :thinking:

I mostly want to use YAML for saves because of the ubiquitous support and compatibility with JSON. By using aeson API, we get JSON and YAML for free, so we can easily output JSON over the web API for tooling.

Secondly, we already use YAML for scenarios and I hope to get some value out of using a similar or even the exact same format. Specifically, I think it would be nice to copy-paste some part of your save file to a scenario - maybe you could use it for a workflow similar to docker commit.

Now if someone wants to use Dhall and pass it to Swarm, then that should be really easy:

swarm --scenario <(dhall-to-yaml-ng my-scenario.dhall)

[valyagolev]

I do have doubts Dhall is the right way here, but it does have a few benefits here.

One of them is much simpler migrations. Basically we can migrate data... in Dhall. We could save data by using functions to create entities, and then we will just have to change those functions if our data format is updated.

If we want for the players to be able to change the save files, Dhall also has greater potential for safe compression. YAML aliases are easy to get wrong, and not even notice until it's too late. Dhall can be much more readable and safe to edit.

I'm also a bit curious about people using advanced Dhall to create e.g. interesting proc-generated scenarios.

So yeah I think it's a fun option and it has benefits.

[xsebek]

If we do not duplicate information in the save file too much, then using aliases will not be necessary. 🤷 If the save file grows linearly with your game, then it's size should be manageable and a compression with zip all you ever need.

I don’t expect the players will change the save file too often, not unless they are hard core cheaters. 😆 But it would be great if the format was readable so it can be used for debugging.

---

The good thing is you can use Dhall to create parametrised scenarios and similar things. You just expand it to YAML for Swarm as I showed before. :wink:

If that turns out to work better then YAML, then we can start by allowing different file formats (.json, .yml.zip, .dhall) and after that we can implement saving in Dhall.

More importantly… 🚲

To be clear what is blocking us here is not being unable to make up our minds about save file format. 🤣 It is that some parts of the game are missing serialisation entirely, namely world (update) function needs to be changed to even be serialisable and to not get exponential explosion the instances have to use outer environment (ToJSONE - think world palette) and finally it needs to be all carefully put together.

I would like to split this into smaller tasks so we can iteratively improve this in our free time. In the end the problem is that someone needs to do it. 😅

---

PS: If anyone has implemented a Dhall serialisation for Swarm, then we would be thrilled to accept it. We do not lock ourselves out of using it with current YAML plan, in fact because they share JSON structure, it would be easy to migrate in future. 🙂

PSS: I hope this clarifies it a bit and does not turn you away @valyagolev and @sullyj3. 😁 TLDR; we are trying to do less work by using YAML atm., and Dhall output looks like more work, but Dhall input works already in a way.

[byorgey]

I think #1138 is a good concrete next step towards this issue, since saving multiple independent map patches will be helpful for creating save files as well. e.g. we could analyze the map, see where it has changed from the default, and use some heuristics to identify rectangular patches with modified cells to save.