Implement Save Files Format

[tilkinsc]

Let's start work on the file format that diablo 1 uses so that we at least have the file head and the immediate character body able to be loaded in. This is more of a self assign thing, but I think this would be a great addition to carry on more of the building up of stats and menus and intro to leveling up.

Possible duplicate of #24 (which is really just a discussion)

So it appears that we should create a 'freeablo proprietary file format'. Which I am a guru of creating such file formats to load beautifully and efficiently (look at libpng api holds lunch). It was also stated that we should have the basic, immediate stats like the dungeon levels, player type, name, stats, and ground items. This should be easy to serialize. I noticed components/serial which is cool and noticed how it plays into FAWord::ActorStats however, this way of loading and saving doesn't support a lot of features, but gives sanity that things aren't mixed up - especially because hardcodedness that we need to circumvent for modders. Perhaps more work on serial will yield good results.

I would highly recommend to use binary format - we don't need to depend on XML or JSON or parse plain text and just use magic bytes for the immediate stats, all strings should come last as they are variable. Or, or, just use a struct with definite sizes (which leads to endianess issues, circumvented easily by forced-endianess exporting) Perhaps a procedural file format is the best for this like BMP has 'chunks'. Ideas?

[mewmew]

Let's start work on the file format that diablo 1 uses so that we at least have the file head and the immediate character body able to be loaded in.

Just to make sure I understand your intention. Do you want to load the original Diablo 1 save file format?

I definitely think that Freeablo should use a standardized format for save/load. And then add support for converting old save files to this format (convert of legacy save files).

The legacy save format is quite simple, once the .sv MPQ has been opened, the game and hero files extracted, and the contents decrypted. The password used to encrypt single player saves is xrgyrkj1 and the password used for multiplayer saves is szqnlsk1 along with the Windows computer name. These are used to create a key for the decryption of save files (ref: 0x4035DB in v1.09). The decryption key is essentially a SHA1 hashsum produced from the password and a randomly generated 136 byte long string. The save files are then decrypted using XOR, and the key is updated based on the decrypted contents of each block in the save file, updating the SHA1 hash, which is the decryption key (ref: 0x4034D9).

Once the game and hero save file has been decrypted, parsing their contents is straight forward. The hero save file is 1288 bytes in length when encrypted. Once decrypted, the first 1266 bytes are copied into a Hero struct, that is later propagated to the Player struct. The Hero struct contains the most essential information about the player.

Further information about items on ground, missiles in the air, monsters on the map, etc is kept in the game file.

The Hero and HeroItem structs are documented in https://github.com/sanctuary/notes/blob/master/structs.h

Note: not all fields are known, so the offsets of known fields are specified and their size in bytes.

Perhaps a procedural file format is the best for this like BMP has 'chunks'. Ideas?

Rather than inventing a new format, lets use something established. How about bson, a binary version of JSON.

[wheybags]

I've already implemented saving, and am actively working on it at the moment. It's implemented as just a simple stream, into which you write variables. It's an interface, and ATM there's a text output implementation for ease of debugging, but a binary format can be easily swapped in later if needed (but tbh, I don't think it will, unless saves start to hit >10mb).

As for loading old saves, sure, but our implementation of stats isn't ready for that yet.

[mewmew]

Wtote a tool that may be used for decoding the Diablo 1 save files. This could be used to create a legacy import feature for Freeablo.

https://github.com/sanctuary/djavul/tree/master/cmd/sv

[tilkinsc]

@wheybags You already write a hard coded binary file which is hard coded. I talked about developing the serialization process. We should actually implement the player stats, etc in lua or python. That way we have a sandbox for modding set up too. We could dynamically handle sending data in bunches using lua or python whereas the hard coded version will get messy.

[tilkinsc]

@mewmew yes I meant standardized format. Other people may want to load in the file by themselves and exit it as a tool or something.

[mewmew]

Now a -json flag has been added to the sv tool, to output the contents of save files in JSON format.

$ sv -json hero

{
    "DAction": -1,
    "Param1": 0,
    "Param2": 0,
    "DLvl": 0,
    "X": 75,
    "Y": 68,
    "TargetX": 75,
    "TargetY": 68,
    "Name": "a",
    "PlayerClass": "Warrior",
    "StrBase": 30,
    "MagBase": 10,
    "DexBase": 20,
    "VitBase": 25,
    "CLvl": 1,
    "Points": 0,
    "Exp": 0,
    "GoldTotal": 100,
    "HPBaseCur": 4480,
    "HPBaseMax": 4480,
    "MPBaseCur": 640,
    "MPBaseMax": 640,
    "SpellLvlFromSpellID": [
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
    ],
    "KnownSpells": [
        0, 0
    ],
    "BodyItems": [
        {
            "Seed": 1217677573,
            "CF": 0,
            "ItemID": "Short Sword",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 20,
            "DurabilityMax": 20,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 0,
            "OnlyUsedByEar": 0
        },
        {
            "Seed": 441146358,
            "CF": 0,
            "ItemID": "Buckler",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 10,
            "DurabilityMax": 10,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 0,
            "OnlyUsedByEar": 0
        }
    ],
    "InvItems": [
        {
            "Seed": 1306853907,
            "CF": 0,
            "ItemID": "Club",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 20,
            "DurabilityMax": 20,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 0,
            "OnlyUsedByEar": 0
        },
        {
            "Seed": 808692058,
            "CF": 0,
            "ItemID": "Gold",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 0,
            "DurabilityMax": 0,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 100,
            "OnlyUsedByEar": 0
        }
    ],
    "InvNumFromInvGrid": [
        -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0
    ],
    "NInvItems": 2,
    "BeltItems": [
        {
            "Seed": 1262929936,
            "CF": 0,
            "ItemID": "Potion of Healing",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 0,
            "DurabilityMax": 0,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 0,
            "OnlyUsedByEar": 0
        },
        {
            "Seed": 1942205617,
            "CF": 0,
            "ItemID": "Potion of Healing",
            "IdentifiedAndItemQuality": 0,
            "DurabilityCur": 0,
            "DurabilityMax": 0,
            "ChargesMin": 0,
            "ChargesMax": 0,
            "GoldPrice": 0,
            "OnlyUsedByEar": 0
        }
    ],
    "OnBattlenet": 0,
    "HasManashild": 0,
    "Difficulty": 0
}

[tilkinsc]

Great job!

[Stromner]

On the off-chance that someone in this years old thread happens to come back, I had a couple of questions about mewmew's comment.

1) I generated a new save file and just went down into the dungeons and up and then saved but the save file is nowhere near big enough to have an address like 0x4035DB or 0x4034D9. My test file only goes up to 0x4DD04

2) How is the key created from the hard-coded password and the SHA1 hashsum?

3) Also I don't understand what's the difference between the described 0x4035DB and 0x4034D9 address

4) "The save files are then decrypted using XOR" XOR with what? The decryption key? In how large blocks?

5) "key is updated based on the decrypted contents of each block in the save file" How exactly?

Either way, thanks for what is already described as this was a hellish rabbit-hole to go down into.