Rust parsing error on saves with accented names; also with C.A.R.E. country

[chennin]

Commit a8bab1ec402fabb7c905fcda9ab5a00234f833d3 breaks parsing a couple saves of mine, in the Rust portion.

1. Can't handle á in names
2. Can't handle a name of = (see first comment)

3.10.4 save (Linux; from Observer mode)(rename to .sav):

kaanvisamfriendship3_2508.07.01.zip

Old save (original report; I don't believe it was in Observer mode):

nexitronawareness4_2948.03.14.zip:

Repro - using docker for isolation

docker run -it forbiddenlake/stellaris-dashboard:v6-debian-bookworm-py3.11 bash

In this container, run:

pip install -e .
curl -sSf https://sh.rustup.rs | sh -s -- -y && . "$HOME/.cargo/env"
rm test/saves/nexitronawareness_1329922464/2565.06.02.sav
git checkout a8bab1ec402fabb7c905fcda9ab5a00234f833d3

Outside the container (in another window), copy the offending save in (remember to rename it) with something like the following, where 98a4b7dd6cb4 is the id of the container you just ran:

docker cp /syncthing/stellaris-saves/nexitronawareness4_-920922254/2948.03.14.sav 98a4b7dd6cb4:/home/stellaris/stellaris-dashboard/test/saves/nexitronawareness_1329922464/

Back in the container, run:

pytest test/parser_test.py -k "test_real_save" -s

This fails with:

FAILED test/parser_test.py::test_real_save - ValueError: Failed to parse /home/stellaris/stellaris-dashboard/test/saves/nexitronawareness_1329922464/2948.03.14.sav: Failed to parse save gamestate

Next, check out the previous commit and rebuild the rust parser:

git checkout 00b4f03b19c6bb51cf6cfae907a8edbf1c875c81
cd stellarisdashboard/parsing/rust_parser
env VIRTUAL_ENV=$(python3 -c 'import sys; print(sys.base_prefix)') maturin develop -r
cd -

And rerun:

pytest test/parser_test.py -k "test_real_save" -s

The parsing succeeds.

Aside: A later commit (7cabf081af69bb8367e4ec1cabba551e39851bcc) contains updates that will prevent parsing this save file for a different reason, which is fixed in #124.

I don't know Rust at all to debug this further, so I'm submitting this. cc @MichaelMakesGames

People do run old versions of the game, so I think keeping backwards compatibility is good if feasible.

[chennin]

Unicode error? Also, an "=" error below

thread '<unnamed>' panicked at src/parser.rs:282:24:
byte index 150 is not a char boundary; it is inside 'á' (bytes 149..151) of `"trait_traditional"
                }
                home_planet=7517
                name_data="backatta"
                gender=not_set
        }
        748=
        {
                base_ref=3
                name_list="PLANT"
                name=
                {
                        key="Várokid"
                }
                plural=
                {
                        key="Várokids"
                }
                adjective=
                {
                        key="%ADJECTIVE%"
                        variables=
                        {

In leader names too:

thread '<unnamed>' panicked at src/parser.rs:282:24:
byte index 150 is not a char boundary; it is inside 'á' (bytes 149..151) of `"nec5"
                        texture=3
                        attachment=0
                        clothes=0
                        ruler_title=
                        {
                                key="Watcher"
                                literal=yes
                        }
                        ruler_title_female=
                        {
                                key="Parártima"
                                literal=yes
                        }
                        trait="leader_trait_principled"
                        leader_class="official"
                }

Replacing all the á makes parsing the save work.

---

Second issue. If you replace all the ás in kaanvisamfriendship3_2508.07.01.zip, you will later get another error on the following input:

        50336245=
        {
                name=
                {
                        key="="
                }
                ship_size="starbase_starport"
                section=
                {
                        template="STARPORT_STARBASE_SECTION"

Yes, that's a name of =. On a autogenerated starport design for C.A.R.E. (planetary_mechanocalibrator) which isn't even a player country. This save is entirely vanilla let run in observer mode. The name is generated every time the C.A.R.E. country is.

[MichaelMakesGames]

I should have time to take a look at this sometime in the next few days

[MichaelMakesGames]

Looks like the unicode error was just a bug in the debugging code, which tries to print the first 150 bytes without considering char boundaries. So that isn't an issue in the release, which has that debugging disabled. Fixed by making the slice char-boundary sensitive

key="=" was the real issue. A strange interaction between the new more-permissive unquoted string parsing and some code that handles an edge-case with value-less keys. I added " to the chars not allowed in unquotes strings