search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.54k stars 274 forks source link

Load models' tokenizer.json file with utf-8 encoding. #194

Closed VldmrB closed 10 months ago

VldmrB commented 10 months ago

Otherwise, json sometimes fails to parse the contents e.g.

  File "E:\exllamav2_ui\exllamav2\model_init.py", line 101, in init
    tokenizer = ExLlamaV2Tokenizer(config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\exllamav2_ui\exllamav2\tokenizer.py", line 77, in __init__
    tokenizer_json = json.load(f)
                     ^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
                 ^^^^^^^^^
UnicodeDecodeError: 'cp932' codec can't decode byte 0x81 in position 812: illegal multibyte sequence

Not sure if this could be necessary anywhere else, didn't have any other files fail to load.

turboderp commented 10 months ago

This doesn't seem to break anything on Linux, so if it fixes stuff in Windows for whatever reason, why not. :+1: