Unable to use JSON parser

[NathanFlegg]

The code snippet for the JSON parser doesn't seem to be working. I've got the full parser working but one of the latest DE test files (provided in this repo) is not working with the JSON parser. Here is the error I get:

/<my_absolute_path>/bin/python /<my_absolute_path>/test.py
Traceback (most recent call last):
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 461, in parse
    de = parse_de(header, version, save)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 338, in parse_de
    de_string(data)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 47, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/<my_absolute_path>/test.py", line 31, in <module>
    match = parse_match(h)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/model/__init__.py", line 93, in parse_match
    data = parse(handle)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 469, in parse
    raise RuntimeError("could not parse")
RuntimeError: could not parse

For this file: de-26.16.zip

Is the JSON parser not working yet for this version of DE recordings?

[happyleavesaoc]

Hi, the JSON output relies on the fast parser, which currently is not supported on the main branch. You can check out the fix-fast branch if you want to test the fix.

I'm testing the fix myself right now and will be merging it soon-ish.

[NathanFlegg]

Thanks, I had a go with the fix-fast branch but during the recursive call to serialize's impl function it's failing when it calls hashlib.HASH, saying that this isn't an attribute. Is hashlib.HASH a placeholder or was this meant to work?

/<mypath>/bin/python /<mypath>/test.py
Traceback (most recent call last):
  File "/<mypath>/test.py", line 77, in <module>
    print(json.dumps(serialize(match), indent=2))
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 367, in serialize
    return impl(obj)
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 355, in impl
    return {k:v for k, v in {f.name:impl(getattr(obj, f.name)) for f in dataclasses.fields(obj)}.items() if v is not None}
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 355, in <dictcomp>
    return {k:v for k, v in {f.name:impl(getattr(obj, f.name)) for f in dataclasses.fields(obj)}.items() if v is not None}
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 351, in impl
    return [v for v in [impl(o) for o in obj] if v is not None]
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 351, in <listcomp>
    return [v for v in [impl(o) for o in obj] if v is not None]
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 355, in impl
    return {k:v for k, v in {f.name:impl(getattr(obj, f.name)) for f in dataclasses.fields(obj)}.items() if v is not None}
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 355, in <dictcomp>
    return {k:v for k, v in {f.name:impl(getattr(obj, f.name)) for f in dataclasses.fields(obj)}.items() if v is not None}
  File "/<mypath>/lib/python3.9/site-packages/mgz/model/__init__.py", line 362, in impl
    elif isinstance(obj, hashlib.HASH):
AttributeError: module 'hashlib' has no attribute 'HASH'

Using same file as above

[happyleavesaoc]

I pushed a commit that should fix it. I think this was some sort of incompatibility between python versions.

[NathanFlegg]

Amazing, it seems to be pretty much working now. I've noticed in the replays though that under the "actions" key for these JSON files, all the positions are wrong, e.g.

    {
      "timestamp": "0:00:01.976000",
      "type": "BUILD",
      "payload": {
        "building_id": 70,
        "building": "House"
      },
      "player": 8,
      "position": -7826587719021397205
    }

I'd expect it to be something like x = 61, y = 80 or something, not a random integer. Any idea why this is and whether it's happening for you too? File here: AgeIIDE_Replay_161780038.zip

I used the JSON snippet you've provided (with a couple of file renaming additions but nothing that should affect the output. It seems to be a fast-parser only problem (or maybe JSON only?) because the full parser is producing all of the positions for moves that it should be.

[happyleavesaoc]

Pushed a commit that fixes it. Sorry about that - I don't use the serialization/json myself so sometimes bugs go unnoticed.

[NathanFlegg]

Amazing - I think I'm only going to be using the json format for what I need it for so that's why I noticed ;) Thank you so much!