UnicodeEncodeError when Python program outputs unicode

Laiteux commented 1 year ago

Version

3.6.4

Platform

.NET 7 / Windows 11

Steps to reproduce

Install https://github.com/jsavargas/zspotify (I used pip, not sure if using Docker would make a difference)

Then run this code:

await Cli.Wrap("zspotify")
    .WithStandardErrorPipe(PipeTarget.ToDelegate(Console.WriteLine))
    .ExecuteAsync();

Which will throw:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Matt\AppData\Local\Programs\Python\Python311\Scripts\zspotify.exe\__main__.py", line 7, in <module>
  File "C:\Users\Matt\AppData\Local\Programs\Python\Python311\Lib\site-packages\zspotify\__main__.py", line 957, in main
    zs.start()
  File "C:\Users\Matt\AppData\Local\Programs\Python\Python311\Lib\site-packages\zspotify\__main__.py", line 881, in start
    self.splash()
  File "C:\Users\Matt\AppData\Local\Programs\Python\Python311\Lib\site-packages\zspotify\__main__.py", line 263, in splash
    print(
  File "C:\Users\Matt\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 6-12: character maps to <undefined>

Details

The unicode header ("splash screen") they're printing seems to be the issue: https://github.com/jsavargas/zspotify/blob/master/zspotify/__main__.py#L263

I have tried setting different output encodings but without success.

Checklist

[X] I have looked through existing open and closed issues to make sure that this bug has not been reported before
[X] I have provided a descriptive title for this issue
[X] I have made sure that that this bug is reproducible on the latest version of the package
[X] I have provided all the information needed to reproduce this bug as efficiently as possible
[ ] I have sponsored this project

Laiteux commented 1 year ago

Is that the answer to this issue as well? https://github.com/Tyrrrz/CliWrap/issues/63#issuecomment-626374243

Laiteux commented 1 year ago

Well, I tried the chcp wrapping workaround (https://github.com/Tyrrrz/CliWrap/issues/196#issuecomment-1445190176) which didn't help.

Laiteux commented 1 year ago

Got it to work by adding PYTHONIOENCODING:UTF-8 to environment variables.

Not sure if I should leave this open.

Tyrrrz commented 1 year ago

Got it to work by adding PYTHONIOENCODING:UTF-8 to environment variables.

Not sure if I should leave this open.

My guess is that the program can't infer the correct encoding (because there is no terminal), so it falls back to cp1252, while also at the same time trying to produce Unicode characters to the console. I suspect you could also work around this by wrapping an actual terminal (i.e. cmd) and running the program through there.

I'd say it's an error on their end, at least based on what you've described. So I'm going to close the issue. Hopefully, your findings (the environment variable) may also be useful to someone in the future.

Tyrrrz / CliWrap