[BUG] Failed to start a server — UnicodeDecodeError

[antshar]

Using v2024.1.10961014, the extension crashes immediately on start up due to server initialization failure.

[info] [Error - 1:34:18 PM] Restarting server failed
[info]   Message: Pending response rejected since connection got disposed
  Code: -32097 
[info] Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 617, in <module>
    main()
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 604, in main

[info]     known_paths = addsitepackages(known_paths)
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 387, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 226, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 179, in addpackage
    for n, line in enumerate(f):
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\codecs.py", line 322, in decode

[info]     (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7 in position 3: invalid continuation byte

I suppose it's due to some non latin characters in somewhere in the path, however the encoding is utf-8

[karthiknadig]

@antshar this seems like an issue with python itself. I don't see any of the extension python code on stack at all.

Try this in a terminal and see if that runs for you. We set PYTHONUTF8=1 before starting the server script, so you might have to do the same.

> cd <dir with non-latin char>
> set PYTHONUTF8=1
> <python3.10 path> <some_script.py>

[antshar]

I tried the proposed steps and yes, set PYTHONUTF8=1 reproduces the very same error indeed. Moreover, even if a path doesn't contain non-latin character, the error still shows up. I tried C:\test\foo.py.

Can it be related to the fact, my window's language is cyrillc based? Is there any way I can fix the issue (I'm ok with any possible workaround)?

[karthiknadig]

@antshar Does it run fine without the PYTHONUTF8=1? You can unset it by using this command set PYTHONUTF8=. Can you share your environment variables? I would particularly like to see the language env variables.

[antshar]

Unsetting PYTHONUTF8 makes C:\test\foo.py to run perfectly, but I don't know how to unset it for autopep8. Here are all env variables I get https://pastebin.com/y2g9HEzg (I don't see any language variables here, though)

[antshar]

I've just noticed that the following

> cd C:\test
> set PYTHONUTF8=1
> python foo.py

leads to an error only when running from cmd.exe while in powershell and git bash, set PYTHONUTF8=1 doesn't cause the exception.

[karthiknadig]

@antshar also can you launch the python in PowerShell (where it works) and run the following commands:

Can you also share how you installed the python? from windows store or python.org etc?

[antshar]

I'm pretty sure I installed python from python.org

[karthiknadig]

I have asked someone with more experience in this area for input on this issue. Meanwhile I think you need to ask this on python.org. Since this is related to running Python itself and not really limited to the extension.

[antshar]

Yes yes, I think the issue can be closed as not related to extension. Thank you for helping me find the cause in the first place, I appreciate that.

While I'll be finding a solution to this, I removed "PYTHONUTF8": "1" everywhere in sources HOME\.vscode\extensions\ms-python.autopep8-2024.0.0 as a dirty fix, so I can use the extension.

[karthiknadig]

@antshar i heard back from someone. You might have a .pth file somewhere that I has the character that might be causing this.

[antshar]

I made a search with Everything which found ten-ish .pth files, mostly in %PYTHONPATH%. I checked each of them and yes, there was a single one that contained non-latin characters. It was easy-install.pth with a single line — path to jupyterlab_sublime-master.

Once I removed it, everything started to work with PYTHONUTF8=1. That's an impressive guess with .pth files! I must admit I spent a couple of hours trying to find a solution, but I found nothing even close to this.

I'm really grateful for your help, especially because it went far beyond autopep8 scope, so you could just say "unrelated".

[zooba]

That's an impressive guess with .pth files!

Allow me to spoil the magic 😄 Open the file from the second last line of your traceback and look around the for loop that broke:

  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 387, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 226, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\site.py", line 179, in addpackage
    for n, line in enumerate(f):
  File "c:\Users\antshar\AppData\Local\Programs\Python\Python310\lib\codecs.py", line 322, in decode