Running into non-fatal error when parsing some rtf readmes

[ProjectSynchro]

When installing: https://deadlystream.com/files/file/312-gaffi-stick-improvement/

It looks like some of the bits of the rtf README are throwing parsing errors in HoloPatcher, this is benign as far as I can tell, but interrupts an automated install (like with KOTORModSync).

Here's the stack trace: let me know if I can give you anything else :)

An unexpected error occurred while loading the patcher namespace. Exception ''utf-8' codec can't decode byte 0xbb in position 36: invalid start byte' of type '<class 'UnicodeDecodeError'>' occurred.
Stack Trace Variables:

Function 'on_namespace_option_chosen' at /tmp/_MEI9fHypt/__main__.py:863:
  self = App(
    browse_button=<tkinter.ttk.Button object .!frame.!button>,
    exit_button=<tkinter.ttk.Button object .!frame4.!button>,
    expand_namespace_description_button=<tkinter.ttk.Button object .!frame.!button2>,
    gamepaths=<tkinter.ttk.Combobox object .!frame.!combobox2>,
    gamepaths_browse_button=<tkinter.ttk.Button object .!frame.!button3>,
    install_button=<tkinter.ttk.Button object .!frame4.!button2>,
    install_running=False,
    log_file_path=PosixPath(/home/synchro/Games/Modding/kotormods...<truncated>
  event = Event(
    <tkinter.Event object at 0x7fa0328a3aa0>
  config_reader = ConfigReader(
    config=PatcherConfig(
        confirm_message='N/A',
        game_number=1,
        ignore_file_extensions=False,
        install_list=[],
        log_level=<LogLevel.WARNINGS: 3>,
        patches_2da...,
    ini=ConfigParser(
        BOOLEAN_STATES={'1': True, 'yes': True, 'true': True, 'on': True, '0': False, 'no': False, 'false': False, 'off': False},
        NONSPACECRE=re.compile('\\S'),
        OPTCR...,
    log=PatchLogger(
        all_logs=[<pykotor.tslpatcher.logger.PatchLog object at 0x7fa0328a3b90>],
        error_observable=Observable(
            callbacks=[]
        ),
        errors=[],
        n...,
    mod_path=CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/Gaffi Stick Improvement/Gaffi Stick Improvement/tslpatchdata"),
    previously_parsed_sections={'Settings'}
)
  namespace_option = PatcherNamespace(
    DEFAULT_INFO_FILENAME='info.rtf',
    DEFAULT_INI_FILENAME='changes.ini',
    data_folderpath=PurePosixPath(.),
    description='',
    info_filename='info.rtf',
    ini_filename='changes.ini',
    name='Gaffi Stick Improvement - KotOR 1',
    namespace_id=''
)
  changes_ini_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/Gaffi Stick Improvement/Gaffi Stick Improvement/tslpatchdata/changes.ini")
  reader = ConfigReader(
    config=PatcherConfig(
        confirm_message='N/A',
        game_number=1,
        ignore_file_extensions=False,
        install_list=[],
        log_level=<LogLevel.WARNINGS: 3>,
        patches_2da...,
    ini=ConfigParser(
        BOOLEAN_STATES={'1': True, 'yes': True, 'true': True, 'on': True, '0': False, 'no': False, 'false': False, 'off': False},
        NONSPACECRE=re.compile('\\S'),
        OPTCR...,
    log=PatchLogger(
        all_logs=[<pykotor.tslpatcher.logger.PatchLog object at 0x7fa0328a3b90>],
        error_observable=Observable(
            callbacks=[]
        ),
        errors=[],
        n...,
    mod_path=CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/Gaffi Stick Improvement/Gaffi Stick Improvement/tslpatchdata"),
    previously_parsed_sections={'Settings'}
)
  game_number = 1
  game = <Game.K1: 1>
  info_rtf_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/Gaffi Stick Improvement/Gaffi Stick Improvement/tslpatchdata/info.rtf")
  info_rte_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/Gaffi Stick Improvement/Gaffi Stick Improvement/tslpatchdata/info.rte")
  data = b'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\xbb Gaffi Stick Improvement ReadMe\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\nAUTHOR: Fallen Guardian\r\nNAME: Gaffi Stick Improvement\r\nTYPE: Modification\r\nV...
  e = UnicodeDecodeError('utf-8', b'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\xbb Gaffi Stick Improvement ReadMe\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\nAUTHOR: Fallen Guardian\r\nNAME: Gaffi Stick Improvemen...

Function 'decode_bytes_with_fallbacks' at /tmp/_MEI9fHypt/pykotor/tools/encoding.pyc:61:
  byte_content = b'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\xbb Gaffi Stick Improvement ReadMe\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\nAUTHOR: Fallen Guardian\r\nNAME: Gaffi Stick Improvement\r\nTYPE: Modification\r\nV...
  errors = 'strict'
  encoding = None
  lang = None
  only_8bit_encodings = False
  provided_encoding = 'utf-8'
Traceback (most recent call last):
  File "__main__.py", line 863, in on_namespace_option_chosen
  File "pykotor/tools/encoding.py", line 61, in decode_bytes_with_fallbacks
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 36: invalid start byte

[ProjectSynchro]

Here's another example with a trace: Mod to reproduce with: https://deadlystream.com/files/file/1846-new_lightsaber_blade_model_k1/

An unexpected error occurred while loading the patcher namespace. Exception ''utf-8' codec can't decode byte 0xdc in position 797: invalid continuation byte' of type '<class 'UnicodeDecodeError'>' occurred.
Stack Trace Variables:

Function 'on_namespace_option_chosen' at /tmp/_MEIT0bjNK/__main__.py:863:
  self = App(
    browse_button=<tkinter.ttk.Button object .!frame.!button>,
    exit_button=<tkinter.ttk.Button object .!frame4.!button>,
    expand_namespace_description_button=<tkinter.ttk.Button object .!frame.!button2>,
    gamepaths=<tkinter.ttk.Combobox object .!frame.!combobox2>,
    gamepaths_browse_button=<tkinter.ttk.Button object .!frame.!button3>,
    install_button=<tkinter.ttk.Button object .!frame4.!button2>,
    install_running=False,
    log_file_path=PosixPath(/home/synchro/Games/Modding/kotormods...<truncated>
  event = Event(
    <tkinter.Event object at 0x7fa4f79cb380>
  config_reader = None
  namespace_option = PatcherNamespace(
    DEFAULT_INFO_FILENAME='info.rtf',
    DEFAULT_INI_FILENAME='changes.ini',
    data_folderpath=PurePosixPath(standard),
    description='Adds new blade models and environmental glow to lightsabers.',
    info_filename='standardinfo.rtf',
    ini_filename='standardchanges.ini',
    name='Modify Standard Lightsaber (Vanilla sabers)',
    namespace_id='standard'
)
  changes_ini_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/New_Lightsaber_Blades_K1_v_1/New_Lightsaber_Blades_K1_v_1/tslpatchdata/standard/standardchanges.ini")
  reader = ConfigReader(
    config=PatcherConfig(
        confirm_message='N/A',
        game_number=2,
        ignore_file_extensions=False,
        install_list=[],
        log_level=<LogLevel.WARNINGS: 3>,
        patches_2da...,
    ini=ConfigParser(
        BOOLEAN_STATES={'1': True, 'yes': True, 'true': True, 'on': True, '0': False, 'no': False, 'false': False, 'off': False},
        NONSPACECRE=re.compile('\\S'),
        OPTCR...,
    log=PatchLogger(
        all_logs=[<pykotor.tslpatcher.logger.PatchLog object at 0x7fa4f79cac90>],
        error_observable=Observable(
            callbacks=[]
        ),
        errors=[],
        n...,
    mod_path=CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/New_Lightsaber_Blades_K1_v_1/New_Lightsaber_Blades_K1_v_1/tslpatchdata/standard"),
    previously_parsed_sections={'Settings'}
)
  game_number = 2
  game = <Game.K2: 2>
  info_rtf_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/New_Lightsaber_Blades_K1_v_1/New_Lightsaber_Blades_K1_v_1/tslpatchdata/standard/standardinfo.rtf")
  info_rte_path = CaseAwarePath("//home/synchro/Games/Modding/kotormods/mods/New_Lightsaber_Blades_K1_v_1/New_Lightsaber_Blades_K1_v_1/tslpatchdata/standard/standardinfo.rte")
  data = b"{\\rtf1\\ansi\\deff0\\adeflang1025\r\n{\\fonttbl{\\f0\\froman\\fprq2\\fcharset0 Times New Roman;}{\\f1\\froman\\fprq2\\fcharset2 Symbol;}{\\f2\\fswiss\\fprq2\\fcharset0 Arial;}{\\f3\\fswiss\\fprq2\\...
  e = UnicodeDecodeError('utf-8', b"{\\rtf1\\ansi\\deff0\\adeflang1025\r\n{\\fonttbl{\\f0\\froman\\fprq2\\fcharset0 Times New Roman;}{\\f1\\froman\\fprq2\\fcharset2 Symbol;}{\\f2\\fswiss\\fprq2\\fcharset0 A...

Function 'decode_bytes_with_fallbacks' at /tmp/_MEIT0bjNK/pykotor/tools/encoding.pyc:61:
  byte_content = b"{\\rtf1\\ansi\\deff0\\adeflang1025\r\n{\\fonttbl{\\f0\\froman\\fprq2\\fcharset0 Times New Roman;}{\\f1\\froman\\fprq2\\fcharset2 Symbol;}{\\f2\\fswiss\\fprq2\\fcharset0 Arial;}{\\f3\\fswiss\\fprq2\\...
  errors = 'strict'
  encoding = None
  lang = None
  only_8bit_encodings = False
  provided_encoding = 'utf-8'
Traceback (most recent call last):
  File "__main__.py", line 863, in on_namespace_option_chosen
  File "pykotor/tools/encoding.py", line 61, in decode_bytes_with_fallbacks
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 797: invalid continuation byte

[th3w1zard1]

iirc the bundled holopatcher in KOTORModSync is a version created before #30 was merged.

If you can verify this by checking the version of HoloPatcher bundled with KOTORModSync, or trying out one of the more recent HoloPatcher versions, it would be appreciated. You can drop a more recent version of HoloPatcher into the Resources folder of KOTORModSync.

[th3w1zard1]

Actually that stack trace looks like it's using the safe_repr function I implemented after #30 was merged.

Is this happening with 1.6.0b3? if so did you build it yourself or are you using the pre-compiled binary? If you built it yourself you most likely did not install charset_normalizer which is listed as an optional package.

Edit: Can confirm from the source that this error only happens when charset_normalizer is not installed. Fixed in https://github.com/th3w1zard1/PyKotor/commit/b3297ec37756d460535940b3ec980872a7066a71.

[ProjectSynchro]

iirc the bundled holopatcher in KOTORModSync is a version created before #30 was merged.

If you can verify this by checking the version of HoloPatcher bundled with KOTORModSync, or trying out one of the more recent HoloPatcher versions, it would be appreciated. You can drop a more recent version of HoloPatcher into the Resources folder of KOTORModSync.

Ah my mistake, forgot to mention I swapped out the release that was in there with 1.6.0b3. it may be missing from the precompiled binaries, I'll have to try building it myself and see if I can reproduce if it's fixed.

[ProjectSynchro]

Can confirm this fixes the issue I was seeing. The fallback also works as intended as well.

I followed this snippet of the readme, but it looks like that requirements.txt file it references is empty:

Once 'install_python_venv.ps1' finishes, you can run any of the provided tools, such as HoloPatcher, KotorDiff, or the Toolset, like this:

pip install -r Tools/HoloPatcher/requirements.txt --prefer-binary
python Tools/HoloPatcher/src/holopatcher/__main__.py
pip install -r Tools/HolocronToolset/requirements.txt --prefer-binary
python Tools/HolocronToolset/src/toolset/__main__.py
python Tools/KotorDiff/src/kotordiff/__main__.py

Should recommended.txt be merged into requirements.txt? or do you still want to requirements to be minimal (or in this case nil :smile: ) it also seems like the release build environment might be missing that module, since that is what I was using when I saw the issue.

[th3w1zard1]

Should recommended.txt be merged into requirements.txt?

No, @NickHugi 's goal is to keep the amount of dependencies as small as possible.

The bug posted here was caused by a missing charset_normalizer package but it should have been falling back to using errors=replace in any case. So what I fixed is now rtf's will load regardless of whether that package is installed or not.

Cheers.