search

happyleavesaoc / aoc-mgz

Age of Empires II recorded game parsing and summarization in Python 3.
MIT License
197 stars 42 forks source link

DE saved files aren't parsed correctly #13

Closed connorjclark closed 4 years ago

connorjclark commented 4 years ago

I understand DE saved files are kinda unsupported right now, Just wanna offer a corpus of ~250 DE games if that would be useful. Only 10 of my saved files parse without errors. Here is an example of the type of errors I get (each error message is from a different save file):

invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 124
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: expected 16 to 16, found 6
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 126
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 4
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 0
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 1032, found 718
    (parsing) -> scenario -> triggers
invalid mgz file: could not read enough bytes, expected 65535, found 29825
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31170
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 65535, found 29871
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2515
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 29204
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27996
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27167
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2
    (parsing) -> scenario
invalid mgz file: expected 16 to 16, found 15
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 10
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 59392, found 40667
    (parsing) -> scenario -> messages -> victory
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31621
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31020
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31179
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 31623
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 30962
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31599
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27066
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 5855
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 1
    (parsing) -> scenario -> players

If it'd be useful, I could upload all of my saves.

I'll start to study this code myself and see if I can help fix things.

connorjclark commented 4 years ago

Here's what I got so far:

happyleavesaoc commented 4 years ago

Hey @connorjclark, thanks for the note. There can be basically three reasons for failing to parse DE matches:

1) Old rec: Checking out an older commit may be necessary, we aren't very backwards compatible. 2) Singleplayer/AI: Not supported. 3) New rec: We lag a bit behind the current patch for obvious reasons.

Your collection would be quite helpful, ideally with preserved filenames so I know which version a file was from.

connorjclark commented 4 years ago

Files (just the MP ones): https://drive.google.com/file/d/1DHVf-vwprwkWYEzc8jEALy1PzqMquBGJ/view?usp=sharing

20 / 197 parsed without errors

Version 101.101.33164.0 (0 / 10) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.33315.0 (0 / 9) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 48147934 to 48147934, found 1062702     (parsing) -> map_info
         invalid mgz file: expected 48147934 to 48147934, found 1070934     (parsing) -> map_info
         invalid mgz file: could not read enough bytes, expected 22651011072, found 2176587     (parsing) -> initial -> particles
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.34055.0 (0 / 2) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.34223.0 (0 / 29) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 27268     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27260     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 65535, found 43575     (parsing) -> scenario -> messages -> hints
         invalid mgz file: could not read enough bytes, expected 12544, found 2179     (parsing) -> scenario
         invalid mgz file: could not read enough bytes, expected 32768, found 29096     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.34397.0 (0 / 17) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 31227     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 30914     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 28926     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.34699.0 (0 / 8) parsed without errors
         invalid mgz file: could not read enough bytes, expected 12544, found 981     (parsing) -> scenario
         invalid mgz file: expected 64925150 to 64925150, found 1280979     (parsing) -> map_info
         invalid mgz file: could not read enough bytes, expected 32768, found 27321     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 12544, found 11216     (parsing) -> scenario
         invalid mgz file: could not read enough bytes, expected 32768, found 27065     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27065     (parsing) -> scenario -> messages -> instructions
Version 101.101.34793.0 (0 / 41) parsed without errors
         invalid mgz file: could not read enough bytes, expected 65535, found 51775     (parsing) -> scenario -> messages -> hints
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 27367     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 3     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 28898     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 30944     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 29009     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 29183     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 12544, found 10     (parsing) -> scenario
         invalid mgz file: could not read enough bytes, expected 32768, found 27298     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 2     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 65535, found 49385     (parsing) -> scenario -> messages -> hints
         invalid mgz file: could not read enough bytes, expected 32768, found 31002     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 31159     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: no decoding mapping for 5     (parsing) -> de -> de
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 30991     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 65535, found 27086     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27065     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: unspecified builtin map: 0
         invalid mgz file: unspecified builtin map: 0
         invalid mgz file: could not read enough bytes, expected 32768, found 31644     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 31617     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 31634     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 31616     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
Version 101.101.35209.0 (0 / 52) parsed without errors
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 16 to 16, found 4     (parsing) -> scenario -> players
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: unspecified builtin map: 0
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 1032, found 718     (parsing) -> scenario -> triggers
         invalid mgz file: could not read enough bytes, expected 65535, found 29825     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 31170     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 65535, found 29871     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 12544, found 2515     (parsing) -> scenario
         invalid mgz file: could not read enough bytes, expected 32768, found 29204     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27996     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27167     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 12544, found 2     (parsing) -> scenario
         invalid mgz file: expected 16 to 16, found 15     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 10     (parsing) -> scenario -> players
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 59392, found 40667     (parsing) -> scenario -> messages -> victory
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 31621     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 31020     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 27266     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 31179     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 31623     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 30962     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: could not read enough bytes, expected 32768, found 31599     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 32768, found 27066     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: could not read enough bytes, expected 12544, found 5855     (parsing) -> scenario
         invalid mgz file: could not read enough bytes, expected 32768, found 27266     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 32768, found 27065     (parsing) -> scenario -> messages -> instructions
         invalid mgz file: expected 16 to 16, found 1     (parsing) -> scenario -> players
         invalid mgz file: could not read enough bytes, expected 12544, found 12478     (parsing) -> scenario
Version 101.101.35584.0 (20 / 29) parsed without errors
         invalid mgz file: unspecified builtin map: 124
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: expected 16 to 16, found 6     (parsing) -> scenario -> players
         invalid mgz file: unspecified builtin map: 126
         invalid mgz file: unspecified builtin map: 123
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings
         invalid mgz file: unspecified builtin map: 123
         invalid mgz file: expected 23 to 23, found 0     (parsing) -> de -> de -> strings

btw, my python script printed out the above (with local modifications to aoc-mgz to add the enums I mentioned)

Checking out an older commit may be necessary, we aren't very backwards compatible.

My goal is to have a tool that I can use to analyze all the games me and my friends play together. I'll play around with older commits and see if it works better for some set of files. If so, I'll look into sending a patch to fix whatever broke + add some older game files for test cases. Would that be in scope for this aoc-mgz?

happyleavesaoc commented 4 years ago

Thanks for the files! Yes, backwards compatibility + tests would definitely be in-scope. The current tests are pretty bare-bones, feel free to refine.

happyleavesaoc commented 4 years ago

I just pushed https://github.com/happyleavesaoc/aoc-mgz/commit/383daf13f63ce426043b939e2de19c58246ee27a, which provides some better compatibility with older versions. For reference, here's how the build in the filename corresponds to the save_version (which is all we can use):

13.07 = 35584
13.06 = 35209
13.03 = 34793
13.03 = 34699
12.97 = 34397
12.97 = 34223
12.97 = 34055
12.97 = 33315
12.97 = 33164
12.97 = 33059
12.97 = 32911
12.97 = 32244

The problem is that they switched from a 6-byte tile structure to an 8-byte tile structure somewhere in December 2019, but did not increment the save_version. That means we can't detect that in a reasonable way. So, recs before December 2019 should parse fine, and recs from 13.03 onwards should work too.

Finally, I looked at some of your recs and updated the GameType enum as you indicated. I think you're playing a scenario or with AI in some of those recs. They have a string table in the de struct right at the start of the file, that the current parser can't handle. There's also at least one case of something AI-related failing to parse in the scenario struct.

I updated the tests with a few DE versions.

connorjclark commented 4 years ago

My first thought would be to use the date, parsed from the filename, as a signal. I would think this would cater to most use cases, since the date is included within the default name.

Less ideal, perhaps some knowledge on valid values for elevation or terrain_type would be a useful signal for if the 6-byte or 8-byte format should be used. For example, if elevation is never greater than some value, one way of interpreting the tiles would lead to invalid values. I think this would require a preprocess pass - the byte unpacking library being seems too declarative to do such a thing.

So, recs before December 2019 should parse fine, and recs from 13.03 onwards should work too.

Any idea what date range we're talking about here?

I think you're playing a scenario or with AI in some of those recs.

Yeah, probably about half of our games include some AIs. I don't think we play any scenario games.

happyleavesaoc commented 4 years ago

Just checked through my files, looks like 2019-11-16 through 2019-12-16 (I think https://www.ageofempires.com/news/aoe2de-update-34055/ is the culprit). Then, https://www.ageofempires.com/news/aoe2de-update-34699/ on 2020-01-20 increments to 13.03 and we resume support.

I don't want to rely on the filename, since many recs get renamed for my use cases, namely tournaments.

You're right, we might be able to Peek() at the next terrain_type and elevation bytes and decide then whether to read the extra 2 bytes. construct should be able to handle that. I do similar stuff throughout to account for various versions.

happyleavesaoc commented 4 years ago

https://github.com/happyleavesaoc/aoc-mgz/commit/c2dff7dd2ef32cdcbca98876d4d63009d8b6d2e7 fixes the 12.97 tile version.

happyleavesaoc commented 4 years ago

https://github.com/happyleavesaoc/aoc-mgz/commit/7dc619440d9140c001771e2d3aa9574d771b4453 fixes string parsing in the DE-specific header for all your sample recs.

connorjclark commented 4 years ago

Sweet, that's some good progress!

Here's the remaining errors:

invalid mgz file: expected 5 to 5, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.16 203335 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.16 194759 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.16 194337 (3).aoe2record
invalid mgz file: expected 16 to 16, found 6
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.11 220325 (2).aoe2record
invalid mgz file: expected 5 to 5, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.11 212425 (2).aoe2record
invalid mgz file: expected 5 to 5, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.11 212233 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.11 204606 (2).aoe2record
invalid mgz file: expected 7 to 7, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.09 205818 (2).aoe2record
invalid mgz file: unspecified builtin map: 123 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.07 135830 (2).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.07 000618 (1).aoe2record
invalid mgz file: expected 5 to 5, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.06 221747 (4).aoe2record
invalid mgz file: expected 5 to 5, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35584.0 @2020.03.06 221542 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.26 230408 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.25 221623 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.25 214027 (2).aoe2record
invalid mgz file: expected 4 to 4, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.22 221025 (2).aoe2record
invalid mgz file: expected 4 to 4, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.21 231528 (2).aoe2record
invalid mgz file: expected 4 to 4, found 0
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.21 223418 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.19 220025 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.19 211733 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.19 204808 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.17 213752 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.17 212648 (1).aoe2record
invalid mgz file: expected 524288 to 524288, found 0
    (parsing) -> map_info | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.17 202110 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.16 213424 (2).aoe2record
invalid mgz file: unspecified builtin map: 87 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.16 210812 (4).aoe2record
invalid mgz file: unspecified builtin map: 113 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.15 225124 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.15 223755 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.15 222900 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.35209.0 @2020.02.15 211234 (3).aoe2record
could not find bytes | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 225454 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 222658 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 220107 (4).aoe2record
invalid mgz file: could not read enough bytes, expected 65535, found 18370
    (parsing) -> scenario -> messages -> loss_cin | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 212021 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 202119 (2).aoe2record
invalid mgz file: unspecified builtin map: 87 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.11 193442 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.09 225322 (3).aoe2record
invalid mgz file: unspecified builtin map: 87 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.09 215419 (2).aoe2record
invalid mgz file: no decoding mapping for 1112497874
    (parsing) -> scenario -> game_settings | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.07 231948 (3).aoe2record
could not find bytes | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.07 225637 (3).aoe2record
invalid mgz file: expected 6 to 6, found 4
    (parsing) -> initial | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.07 224213 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.02.02 161539 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.31 234141 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.31 230442 (3).aoe2record
invalid mgz file: expected 16 to 16, found 9
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.31 212732 (3).aoe2record
invalid mgz file: expected 16 to 16, found 5
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.29 234114 (2).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.28 220247 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.28 214124 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34793.0 @2020.01.28 204932 (5).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34699.0 @2020.01.21 224630 (4).aoe2record
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34699.0 @2020.01.21 215009 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.20 000238 (4).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.17 212740 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.17 202857 (4).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.17 195636 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.15 214609 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.15 214409 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.15 210531 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.15 210208 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.15 202904 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.13 220135 (6).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.13 212037 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.13 205137 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.13 204834 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34397.0 @2020.01.13 201748 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2020.01.09 203319 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2020.01.09 194407 (4).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.29 173445 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.29 172933 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.29 165723 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.29 164933 (1).aoe2record
invalid mgz file: unspecified builtin map: 83 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.28 163654 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.28 161432 (2).aoe2record
invalid mgz file: expected 16 to 16, found 8
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.28 151424 (4).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.28 141219 (4).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.25 222435 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.25 212900 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.25 212637 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.24 215021 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.24 211641 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.23 211218 (3).aoe2record
invalid mgz file: no decoding mapping for 33554944
    (parsing) -> scenario -> game_settings | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.23 205842 (3).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.23 200429 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.23 194944 (1).aoe2record
invalid mgz file: expected 16 to 16, found 5
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.23 191807 (2).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.22 200623 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.22 195304 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.22 185807 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.22 182936 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34223.0 @2019.12.22 175306 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34055.0 @2019.12.17 204817 (1).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.34055.0 @2019.12.17 204624 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.15 174004 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.15 173718 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.15 164635 (3).aoe2record
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.11 234840 (3).aoe2record
invalid mgz file: unspecified builtin map: 113 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.11 220108 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.05 212939 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33315.0 @2019.12.05 205800 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.25 210115 (3).aoe2record
invalid mgz file: expected 16 to 16, found 4
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.25 205453 (3).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.25 202514 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.25 192028 (3).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.25 191007 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.23 140513 (2).aoe2record
invalid mgz file: expected 8 to 8, found 0
    (parsing) -> ai -> yep -> ais | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.23 131958 (1).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.23 131623 (2).aoe2record
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.23 131053 (2).aoe2record
invalid mgz file: unspecified builtin map: 0 | path: C:\Users\Connor\Downloads\savegame\MP Replay v101.101.33164.0 @2019.11.23 130828 (2).aoe2record
88 / 197 parsed without errors

Version 101.101.33164.0 (0 / 10) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 4     (parsing) -> scenario -> players
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
     invalid mgz file: unspecified builtin map: 0
Version 101.101.33315.0 (2 / 9) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
     invalid mgz file: unspecified builtin map: 113
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
Version 101.101.34055.0 (0 / 2) parsed without errors
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
Version 101.101.34223.0 (4 / 29) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 83
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 8     (parsing) -> scenario -> players
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: no decoding mapping for 33554944     (parsing) -> scenario -> game_settings
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 16 to 16, found 5     (parsing) -> scenario -> players
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
Version 101.101.34397.0 (3 / 17) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
Version 101.101.34699.0 (6 / 8) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 0     (parsing) -> scenario -> players
Version 101.101.34793.0 (22 / 41) parsed without errors
     could not find bytes
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: could not read enough bytes, expected 65535, found 18370     (parsing) -> scenario -> messages -> loss_cin
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 87
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 87
     invalid mgz file: no decoding mapping for 1112497874     (parsing) -> scenario -> game_settings
     could not find bytes
     invalid mgz file: expected 6 to 6, found 4     (parsing) -> initial
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 9     (parsing) -> scenario -> players
     invalid mgz file: expected 16 to 16, found 5     (parsing) -> scenario -> players
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
Version 101.101.35209.0 (34 / 52) parsed without errors
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 4 to 4, found 0     (parsing) -> initial
     invalid mgz file: expected 4 to 4, found 0     (parsing) -> initial
     invalid mgz file: expected 4 to 4, found 0     (parsing) -> initial
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 524288 to 524288, found 0     (parsing) -> map_info
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 87
     invalid mgz file: unspecified builtin map: 113
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
Version 101.101.35584.0 (17 / 29) parsed without errors
     invalid mgz file: expected 5 to 5, found 0     (parsing) -> initial
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 16 to 16, found 6     (parsing) -> scenario -> players
     invalid mgz file: expected 5 to 5, found 0     (parsing) -> initial
     invalid mgz file: expected 5 to 5, found 0     (parsing) -> initial
     invalid mgz file: unspecified builtin map: 0
     invalid mgz file: expected 7 to 7, found 0     (parsing) -> initial
     invalid mgz file: unspecified builtin map: 123
     invalid mgz file: expected 8 to 8, found 0     (parsing) -> ai -> yep -> ais
     invalid mgz file: expected 5 to 5, found 0     (parsing) -> initial
     invalid mgz file: expected 5 to 5, found 0     (parsing) -> initial

here's what I found out:

(btw, you probably already know this, but FYI for others: Probe(x) is useful for dumping the next x bytes of the buffer. put that just before an unknown portion of the data and it'll help you debug)

1) expected 8 to 8, found 0 (parsing) -> ai -> yep -> ais (ai scripts)

https://github.com/happyleavesaoc/aoc-mgz/blob/master/mgz/header/ai.py#L38

I think the ai part always fails. How do you determine what the fields should be?

image

2) unsane min -1 and max -1

This error didn't say which component caused it, but I narrowed it down to "achievements". Seems that replay.num_players is 0.

image

This should work for DE:

achievements = Array(lambda ctx: ctx.de.num_players, player_achievements)

3) unspecified builtin map

https://github.com/happyleavesaoc/aoc-mgz/blob/master/mgz/summary/map.py#L94

If getting all the values isn't straightforward, could just fall back to "unknown" instead of erroring.

4) (parsing) -> initial

Something wrong with the object parsing. Parser continues to work if can_parse_objects returns False.

5) expected 16 to 16, found 6 (parsing) -> scenario -> players

the data seems to go bad somewhere between initial.players[0] and players[1]

6) expected 524288 to 524288, found 0 (parsing) -> map_info

didn't look at this yet

happyleavesaoc commented 4 years ago
  1. AI parsing is definitely broken (DE made changes to that structure). Determining what the fields should be is the hard part! I normally stare at the bytes for awhile and try to see patterns.
  2. The other errors are sometimes reporting the right problem, but usually not. The parser is basically a big fancy struct that overlays the bytes, right? So if something is inaccurate, the parser may continue to "work" for awhile until we try to use the output (for example, using an array length or computing an enum) with a wrong value. The parser dies at this point, but the real problem is upstream. The way I troubleshoot these is by incrementally parsing each section and validating that the data looks good (just eyeballing), until I hit a block that seems misaligned. That's the location for further investigation. Once properly aligned, the downstream problem goes away.
connorjclark commented 4 years ago

I normally stare at the bytes for awhile and try to see patterns

I was afraid you'd say that! I'll give it a shot too.

I'm trying to debug why most of my DE recordings don't have any achievements. I've noticed that only restored games have non-zero values in the achievements struct, which is strange. Is that expected? I see it should come from the "postgame" action, but I don't have that in many of my recordings. I see that the last action is often "Resign", so maybe that makes Postgame not occur ...

You can see the Statistics via the in-game replay mode at any time, which suggests to me that it should be possible to calculate the achievements by processing the actions. I don't see any actions such as "villager collects x resources" - I guess that isn't an explicit action, but rather something that must be simulated using "order" actions and the map data. Is that right?

happyleavesaoc commented 4 years ago

The "achievements" section in the header is a snapshot of the initial state. Normally this is at time 0, so there's not much to see. In restored games, the initial state is obviously some other time, so you'll have some more values there. In summary, yes, this is expected.

There is a second achievements structure in the postgame action. This action is not in DE. It is Userpatch only. I've suggested they add it on the DE forums: https://forums.ageofempires.com/t/suggestion-append-achievements-to-recorded-game-file/64800

The "Statistics" screen you can see in-game is reading that same achievements structure in memory. You can calculate the achievements by processing the actions but only by re-implementing the whole game itself. In fact, that is how the game does it. You'll only see actions that are player input, which mutate the game state according to the game rules. You can think of the recorded game like chess notation - just a list of moves. The outcomes depend on the current state and applicability of the game rules.

happyleavesaoc commented 4 years ago

I'm going to close this in favor of more specific issues (specifying a single rec), and considering the intended scope of the project (competitive multiplayer recs).