Store hash-version map files for acquiring version number

[heinezen]

Required skills: Python

Difficulty: Easy

Motivation

Age of Empires 2 and other Genie games do not store their version number in a plaintext file or anything else that is easy to read. Therefore, openage uses the hashes of files to determine the version of a game. The idea behind this is that a file from a specific release will always have the same hash value. By calculating the hash for a file and looking up the hash in a dict of hash values and version numbers, we can find the release version the file belongs to.

Task

The dicts of hashes and version numbers should ideally be stored in a config file. TOML can be used as a format since we already use that elsewhere in the converter. There should be one config file for every game edition and game expansion.

Example (using TOML):

# Header
# The version of this config file format
file_version = "1.0"
hash_algo = "SHA3-256"

[empires2dat]
path = "Data/empires2.dat"

    [empires2dat.map]
    "d45b001ee8493c1c61a1" = "1.0"
    "4239b6d45b001ee38128" = "1.2"

[age2_x1]
path = "age2_x1/age2_x1.exe"

    [age2_x1.map]
    "001ee8493c1c61a1d45b" = "1.0"
    "b001ee381284239b6d45" = "1.2"

Every file that we want to hash should get its own table with a subtable for the hash-version map. The recommended hashing algorithms are SHA3-256 or SHA-512.

[drs-11]

Hey I can do this one too! Seems similar to the manifest file one.

[heinezen]

@drs-11 Great :D

In comparison to the manifest file task, the hashes do not need to be generated this time. The file is manually created by the openage team and read in by the converter into GameFileVersion instances.

https://github.com/SFTtech/openage/blob/master/openage/convert/value_object/init/game_file_version.py#L10-L14

I can provide the SHA3-256 hashes for several older releases if you don't own them.

[drs-11]

Yes please, I do not currently own any of the releases right now.

Ok so the file is need to be created and then GameEdition enumeration is needed to be modified to use the map file to enumerate the versions?

[heinezen]

The Conqueros:

Version: 1.0c

d02ef2f4935c5ef568eaa873442b4b2b547067ed8e3572758bbbbee8  age2_x1/age2_x1.exe
4568f37f309c0a89fa9e84a0c0f6c61b0f394c28adc82c0b3f2203b3  age2_x1/age2_x1_nocd.exe
6aab4c7468c3d319ec00f577835dc7799b70ffab1de6bfd25ac227b5  Data/empires2_x1_p1.dat

Rise of Rome

Version: 1.0B

bd55a288136ea5457c61fbc794e3e4a4ac526b1d53809f1b16762119  EMPIRESX.EXE
0376368ce4c78be31596dea7b76f592b2352f6b5c3aca222b7939b4a  data2/empires.dat

Star Wars: Galactic Battlegrounds

Version: GOG

0a5e2b7828b5b0a45cb6dfbf52e3becb2021e97259ba23a5cee94230  Game/Battlegrounds.exe
f1f641f201dac9d7663df4cf1ece7a60a78770a09e70e202399505d9  Game/Data/GENIE.DAT

Clone Campaigns

Version: GOG

fb722624bae8a4626925926d5709e1397766dd3ee2dcd1df7849ad27  Game/battlegrounds_x1.exe
036a16aefd1a66f5bad121786e66b8cc9542cbe88b43c4eb8f73eadb  Game/Data/genie_x1.dat

AOE1DE

Version: Steam

3356c013f2cb732601e1acf286654e4360d7e7304a092bdbeaa63d48  AoEDE_s.exe
4d1f3cb7ce20074bfa23445c62755b95e4ec193b41507d5196921033  Data/empires.dat

AOE2DE

Version: 36906

01ff7cc7819e4f77be059c8efc6c07472dd1a58f429621e937328750  AoE2DE_s.exe
cf13c553c78df192da1af957d1519870db15c9dbca314b245a5abe6f  resources/_common/dat/empires2_x2_p1.dat

HDEDITION

Version: 5.8

73a068f353efd50e058762ca090062600a7ca50e435fe55a6ca87efa  AoK HD.exe
91f7370b75b2fadf238b25f6fdd2f43a88a1f01ea022cd6086e44f43  resources/_common/dat/empires2_x1_p1.dat

AFRI_KING

Version: 5.8

b2a69980a1ef39e68de2be1771392858cb639a9f2797373ab1649acf  resources/_common/dat/empires2_x2_p1.dat

[TheJJ]

Don't forget that pretty much all the user-patched exes have differend hashes depending on the selected userpatch features etc. That said, this version detection approach should only be an addition to the other methods, e.g. file discovery. Maybe there's some build ID in the exe we can extract, too?

[heinezen]

We can store multiple hashes for a filename that all reference the same version. I'm pretty sure there is a build version in the DE2 exe, but I found none in the older games' exes. Maybe it's there but has no descriptive string in front of it.

I don't think we can get away without hashing though because it's the only way I can think of to detect non-standard versions of the game (i.e. modded versions). We cannot detect a modded datfile otherwise.

[drs-11]

@heinezen the hashes you provided are different from the ones in game_version.py even though they are of the same version I think. Why so?

And I'm not really sure what we are trying to do here. Are we using to toml map file to separate the version infos? So that it can be used to build up the GameEdition enum?

[heinezen]

the hashes you provided are different from the ones in game_version.py even though they are of the same version I think. Why so?

The hashes currently in game_version.py are MD5 hashes, not SHA3.

Are we using to toml map file to separate the version infos? So that it can be used to build up the GameEdition enum?

Yes, we want to get them out of the Python code into regular files. That way, we can easily update them without changing code. This is important as otherwise we would have to create a new converter release for every patch of a game (in a worst case scenario). Having the data in an auxiliary file is much more convenient because they can be downloaded and read in without Python code changes.

In the long run, we also want to get rid of the GameEdition enum (see https://github.com/SFTtech/openage/issues/1314) an save all version info in TOML files.

[drs-11]

The hashes currently in game_version.py are MD5 hashes, not SHA3.

Oh okay of course. Silly me.

In the long run, we also want to get rid of the GameEdition enum (see #1314) an save all version info in TOML files.

Hey since the aim is to get rid of GameEdition enum I can take that issue as well(it's idle for a week) and save all info into TOML files. If that's okay.

[heinezen]

I can take that issue as well(it's idle for a week) and save all info into TOML

That would be really awesome :D :rocket: