[Bug Report] cp950 encoding and decoding issue on Windows

[tai271828]

Description

When running mail_handler on windows, we have the following error message:

UnicodeDecodeError: 'cp950' codec can't decode byte 0xe5 in position 49: illegal multibyte sequence

We have tried https://github.com/pycontw/mail_handler/pull/10 . Unfortunately, https://github.com/pycontw/mail_handler/pull/10 did not resolve the issue.

Environment

mail_handler version => 0.1.0 windows version => Microsoft Windows 10 Home Edition 10.0.19042 python version => python39 python environment => C:\Python39\python.exe

Steps to Reproduce

Steps to reproduce the behavior:

1. clone the source
2. follow README to run sponsorship 2020 example to render emails

Expected Behavior

Emails are rendered.

Actual Behavior

Traceback (most recent call last):
  File "c:\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\render_mail.exe\__main__.py", line 7, in <module>
  File "c:\python39\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\python39\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\python39\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python39\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\python39\lib\site-packages\mail_handler\render_mail.py", line 61, in main
    data = json.load(input_file)
  File "c:\python39\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
UnicodeDecodeError: 'cp950' codec can't decode byte 0xe5 in position 49: illegal multibyte sequence

[tai271828]

@allen91wu , would you mind providing more information for debugging? At the very beginning, we may need the following information:

mail_handler version (pypi package version, or commit hash) windows version python version python environment (which python or the absolute path of your python interpreter)

more system environment variables are welcome.

[tai271828]

From @allen91wu when cloning repo:

remote: Enumerating objects: 683, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 683 (delta 44), reused 75 (delta 24), pack-reused 572
Receiving objects: 100% (683/683), 773.70 KiB | 4.05 MiB/s, done.
Resolving deltas: 100% (328/328), done.
error: unable to create file tests/data/with-separator/tai271828+sponsorship.testemail.4@gmail.com, tai271828+sponsorship.testemail.4.1@pycon.tw, taihsiangho+sponsorship.testemail.4.2@gmail.com,tai271828+sponsorship.testemail.4.3@solvcon.net - 韐蝯葫閰▒04▒▒▒▒: Filename too long
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

[tai271828]

@allen91wu , would you mind providing more information for debugging? At the very beginning, we may need the following information:

mail_handler version (pypi package version, or commit hash) windows version python version python environment (which python or the absolute path of your python interpreter)

more system environment variables are welcome.

git version may be also useful. @allen91wu would you mind giving git config --system core.longpaths true a try?

See https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

[allen91wu]

git version may be also useful. @allen91wu would you mind giving git config --system core.longpaths true a try? See https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

@tai271828 I try it, and it is successfully for me to use git clone repo in terminal. Thanks for your helping.

[allen91wu]

@allen91wu , would you mind providing more information for debugging? At the very beginning, we may need the following information: mail_handler version (pypi package version, or commit hash) windows version python version python environment (which python or the absolute path of your python interpreter)

mail_handler version => 0.1.0 windows version => Microsoft Windows 10 Home Edition 10.0.19042
python version => python39 python environment => C:\Python39\python.exe

[tai271828]

git version may be also useful. @allen91wu would you mind giving git config --system core.longpaths true a try? See https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

@tai271828 I try it, and it is successfully for me to use git clone repo in terminal. Thanks for your helping.

Neat! My gut feeling shows the traceback of this issue is caused by the same limitation of windows. By changing a file name like tai271828+sponsorship.testemail.4@gmail.com, tai271828+sponsorship.testemail.4.1@pycon.tw, taihsiangho+sponsorship.testemail.4.2@gmail.com,tai271828+sponsorship.testemail.4.3@solvcon.net in tests/data and/or the mail you are going to render may help.

[tai271828]

Additional information:

I found CPython has been handling cp950 already, so the issue is not likely caused by cpython (and its standard library).

https://github.com/python/cpython/blob/master/Lib/encodings/cp950.py

[allen91wu]

I solve the problem in PR #23. In PR #10 , I don't have absolutely fix default encoding to utf-8. For windows user, you can also consider to use WSL for you to use Ubuntu terminal with development or using mail_handler .