[Discussion] UnicodeDecodeError with vdf.load

We encountered an error where vdf.load(open("config.vdf")) would crash when the vdf file contains an invalid UTF-8 character. This was reported a few times now. See https://github.com/DavidoTek/ProtonUp-Qt/issues/424 for details.

The error occurs in this line of vdf/__init__.py:

  File "/app/lib/python3.11/site-packages/pupgui2/steamutil.py", line 221, in get_steam_global_ctool_name
    d = get_steam_vdf_compat_tool_mapping(vdf.load(open(config_vdf_file)))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 211, in load
    return parse(fp, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 97, in parse
    for lineno, line in enumerate(fp, 1):
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2170: invalid start byte

We are not sure how often the error occurs and were not able to reproduce it. It is likely caused by a misconfigured system locale on Linux. A similar issue was reported and fixed for vdf.binary_loads in https://github.com/ValvePython/vdf/issues/20.

The fix we implemented looks like this (see ProtonUp-Qt PR #455). Invalid UTF-8 characters are simply replaced by "�".

with open("config.vdf", "r", encoding="utf-8", errors="replace") as f:
    x = vdf.loads(f.read())

# Something like this would probably also work - untested:
vdf.load( open("config.vdf", "r", encoding="utf-8", errors="replace") )

I don't think this is something that can be fixed from your side as vdf.load only accepts the file object and the Unicode-decoding and error handling is part of open() which is called by the user. I just thought it might be interesting to share in case others also have a similar issue (this could go into "GitHub Discussions").

solsticegamestudios / vdf

[Discussion] UnicodeDecodeError with vdf.load #2