`pretty_format_json` crashes when run as a `pre-commit` hook, but not when run alone with a similar config

[aatango]

SETUP

1. Change to a (new) repository on Windows;

2. use the following .pre-commit-config.yaml configuration:

   -   repo: 'https://github.com/pre-commit/pre-commit-hooks'
   rev: 'v4.5.0'
   hooks:
   -   id: 'pretty-format-json'
       args: ['--no-sort-keys', '--indent', '2', '--autofix']

3. add a new JSON file with a non-ascii filename, for example: 漢字.json, or עִבְרִית‎.json; and

4. attempt to commit both the pre-commit configuration and the JSON file to your repository.

EXPECTED

The newly added JSON file is properly handled both when running git commit, or the individual pretty_format_json.py file.

ACTUAL

• When running pretty_format_json.py $FILENAME --no-sort-keys --indent 2 --autofix, everything works as expected.
• However, when pretty_format_json runs as part of a hook, the following error is thrown:

  
  pretty format json........................................................................Failed

• hook id: pretty-format-json
• exit code: 1 Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "%USERPROFILE%.cache\pre-commit\repoq17iut6z\py_env-python3.10\Scripts\pretty-format-json.EXE__main__.py", line 7, in File "%USERPROFILE%.cache\pre-commit\repoq17iut6z\py_env-python3.10\lib\site-packages\pre_commit_hooks\pretty_format_json.py", line 122, in main _autofix(json_file, pretty_contents) File "%USERPROFILE%.cache\pre-commit\repoq17iut6z\py_env-python3.10\lib\site-packages\pre_commit_hooks\pretty_format_json.py", line 34, in _autofix print(f'Fixing file {filename}') File "C:\Program Files\Python310\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 12-19: character maps to

NOTE

The error can be reproduced on Windows 10, but not on Ubuntu. Both have Python 3.10.

[asottile]

what codepage have you confused on Windows? have you tried enabling utf8 mode?

[aatango]

Activating the (still beta) support for UTF-8 mode in Windows appears to bypass the issue. In this mode, executing the script within the context of a git commit does not throw an error; a JSON file is properly formatted.

However, this is still an experimental support from Windows, and activating it causes problems elsewhere.

[asottile]

how about python's utf8 mode?

[aatango]

Setting the environment variable PYTHONUTF8=1 did solve the issue. Thank you for the help.