Adapt StellarMap's save game parser

[MichaelMakesGames]

Hey, I recently wrote a new parser (also in Rust) for my own project stellarmaps. I think Stellaris Dashboard's parsing could get a big speed improvement, and additionally it might solve some of the errors you mentioned in https://github.com/eliasdoehne/stellaris-dashboard/issues/127#issuecomment-1872605352

The main benefit of this parser is the ability to filter to only the parts of the save you're interested in. The filter is a json object like:

{
  "planets": {
    "planet": {
      "*": {
        "name": true,
        "class": true
      }
    }
  }
}

The above example would only give you the db of planets, and only the name and class for each planet. The parts that are skipped are only minimally parsed, and, more importantly, are not converted to python objects at all. In a quick and dirty implementation for Stellaris dashboard, I saw a ~6x improvement in parsing speed (from 3s to 0.5s)

(I suspect in reality, it might be a smaller but still substantial improvement, because there are probably some errors in my filter causing some data to be skipped. The filter I'm using now was auto-generated by spying on items accessed in _gamestate_dict. Here's the generated filter: filter.json)

Regarding the errors you mentioned in https://github.com/eliasdoehne/stellaris-dashboard/issues/127#issuecomment-1872605352, I believe those are caused by ambiguity around parsing empty objects as maps vs lists. In this parser, the filter also helps to disambiguate that situation.

Another thing I should bring: this parser handles duplicate keys differently. The current parser turns them into lists, This parser instead adds a special property $multiKeys, to hold the duplicate entries. The gamestate processors would need to be updated to account for this.

Thoughts? It's a pretty fundamental change, so there would be bugs along the way. But 6x faster is pretty big

For reference: https://github.com/MichaelMakesGames/stellarmaps/blob/main/src-tauri/src/lexer.rs https://github.com/MichaelMakesGames/stellarmaps/blob/main/src-tauri/src/parser.rs

[eliasdoehne]

I think that's a great idea and makes a lot of sense for the application. For my workflow it was quite important to parse the entire save object to inspect it in the debugger to understand the save structure and figure out breaking changes due to game updates, so I never really considered that. But a 6x improvement in the final application is definitely worth it.

[MichaelMakesGames]

Sounds good! I'll look into getting this fully implemented when I have time (and verify what the savings actually are)

parse the entire save object to inspect it in the debugger

You can pass it a filter of simply true to parse all keys (recursively). I often do this when working on new features for StellarMaps