Default language breaks non-ascii and special characters

[ahyangyi]

Hi,

I noticed that non-ascii characters and special characters (such as {SILVER}) don't work for the default language file, no matter what language it is.

For example, if I have Chinese and English language files, if I set Chinese to the default, the Chinese strings will be broken; and vice versa.

By "broken" I mean that all non-ascii characters and special characters are shown escaped.

[ahyangyi]

A minimal example:

#!/usr/bin/env python
import grf

def get_string_manager():
    s = grf.StringManager()
    s.import_lang_dir("lil/lang", default_lang_file="english-uk.lng")

    return s

def gen():
    s = get_string_manager()
    g = grf.NewGRF(
        grfid=b"MNMM",
        name=s["STR_GRF_NAME"],
        description="",
        strings=s,
    )

    g.write("minimum.grf")

if __name__ == "__main__":
    gen()

then the lang files:

lang/chinese.lng

##grflangid 0x56
STR_GRF_NAME                    :最{WHITE}小{RED}复{GOLD}现

lang/english-uk.lng

##grflangid 0x01
STR_GRF_NAME                    :Minimal {WHITE}Re{RED}producible {GOLD}Example

In this case the Chinese works but the English (UK) breaks because the latter is the default.

[ahyangyi]

The "default language" strings are in Action 8, the other strings in Action 14. Perhaps this is the the reason of the difference...

[ahyangyi]

Oh, I think I found it: the call to __str__ drops the special symbols,

https://github.com/citymania-org/grf-py/blob/main/grf/grf.py#L966

Or, perhaps I should say that it didn't call grf_compile_string and left the escape strings there.

[ldpl]

Can you please check and confirm that https://github.com/citymania-org/grf-py/commit/f320d9a5a74e93fa57717f0eddbec54c2a0fc881 fixes this?

[ahyangyi]

Yes, I tested and can confirm that https://github.com/citymania-org/grf-py/commit/f320d9a5a74e93fa57717f0eddbec54c2a0fc881 fixes this.