Support UTF-16 text files

[Lumerica]

Hi, I'm using Windows 10, all the text file i created are utf-16 using latest SD 1.10.1. Python 3.10.10 I am aware that this is not related to gradio or SD, as I had this problem since more than 1 year ago. But today I thought I should give another try with a fresh look to see if I can find something.

It seems to be pointing that if a UTF16 text file exist you will get error. if I change the text file format to UTF-8, I get no error in console nor in webui and the text file can be displayed.

Is there a way to support these UTF-16?

Traceback (most recent call last):
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "S:\GIT\stable-diffusion-webui\venv_Torch201\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "S:\GIT\stable-diffusion-webui\extensions\sd-model-preview-xd\scripts\modelpreview.py", line 941, in show_lora_preview
    return show_preview(modelname, get_lora_dirs(), "loras")
  File "S:\GIT\stable-diffusion-webui\extensions\sd-model-preview-xd\scripts\modelpreview.py", line 967, in show_preview
    for line in file:
  File "C:\Users\HelF\AppData\Local\Programs\Python\Python310\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

[CurtisDS]

is there a particular reason you are using utf16? The gradio page is encoded as utf8... im not sure what will happen if I tried to put raw utf16 text on the page.

[Lumerica]

Kinda late response here. Basically I've been using UTF16 for years as default encoding. The answer of the question would be not so related to the extension,

Reason is because i like to save metadata of many various things (copy pasting etc.) But in general its not just limited to asian characters but also emojis. Saving them in a UTF-8 text file will mean its lost.

That is my reason that I've always used UTF-16 for my text files except when I need to write something in bat file or stuff then I'm forced to save it as UTF-8 lol which makes sense in that case.

[CurtisDS]

I did some looking into this... and from what i can tell... I dont see an easy way to allow this. Everything I see that talks about UTF16 for websites says DONT DO IT lol. If someone wants to add support and do a pull request Id be happy to accept it. But unfortunately I dont know how to add support without a lot of work and more research. So for now Im going to close this as not planned.