[BUG] UnicodeEncodeError when trying to print utf-8 characters despite setting correct encoding env variable on Windows.

[jakub-deka]

• [X] I've checked docs and closed issues for possible solutions.
• [X] I can't find my issue in the FAQ.

Describe the bug

UnicodeEncodeError when trying to print utf-8 characters despite setting correct encoding env variable on Windows.

To replicate this you can try this.

from rich.console import Console
import os

log_file = 'log_file.log'
test_messages = [
    'This is a test',
    'çç',
    '\u2248',
    '\u0104',
]

os.makedirs(os.path.dirname(log_file), exist_ok=True)

with open(log_file, "a") as f:
    console = Console(file=f, width=200, )

    console.print(os.getenv('PYTHONIOENCODING'))

    for m in test_messages:
        console.print(m)

Conda env has been set up like this:

name: test-env
dependencies:
  - python==3.11.7
  - numpy
  - pandas
  - pip
  - ipykernel
  - pip:
    - langroid
    - crewai
    - crewai[tools]
variables:
  PYTHONIOENCODING: utf-8
  TESTVAR: this_is_a-test

When running this on my machine I get following error.

Current working directory is C:\Users\user
Traceback (most recent call last):
  File "logging_demo.py", line 21, in <module>
    console.print(m)
  File "C:\Users\user\anaconda3\envs\test-env\Lib\site-packages\rich\console.py", line 1673, in print
    with self:
  File "C:\Users\user\anaconda3\envs\test-env\Lib\site-packages\rich\console.py", line 865, in __exit__
    self._exit_buffer()
  File "C:\Users\user\anaconda3\envs\test-env\Lib\site-packages\rich\console.py", line 823, in _exit_buffer
    self._check_buffer()
  File "C:\Users\user\anaconda3\envs\test-env\Lib\site-packages\rich\console.py", line 2039, in _check_buffer
    write(text)
  File "C:\Users\user\anaconda3\envs\test-env\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u2248' in position 0: character maps to <undefined>
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***

Platform

Windows 11 running in conda env.

``` └$ & C:/Users/user/anaconda3/envs/test-env/python.exe -m rich.diagnose ╭───────────────────────── ─────────────────────────╮ │ A high level console interface. │ │ │ │ ╭──────────────────────────────────────────────────────────────────────────────╮ │ │ │ │ │ │ ╰──────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ color_system = 'truecolor' │ │ encoding = 'utf-8' │ │ file = <_io.TextIOWrapper name='' mode='w' encoding='utf-8'> │ │ height = 81 │ │ is_alt_screen = False │ │ is_dumb_terminal = False │ │ is_interactive = True │ │ is_jupyter = False │ │ is_terminal = True │ │ legacy_windows = False │ │ no_color = False │ │ options = ConsoleOptions( │ │ size=ConsoleDimensions(width=137, height=81), │ │ legacy_windows=False, │ │ min_width=1, │ │ max_width=137, │ │ is_terminal=True, │ │ encoding='utf-8', │ │ max_height=81, │ │ justify=None, │ │ overflow=None, │ │ no_wrap=False, │ │ highlight=None, │ │ markup=None, │ │ height=None │ │ ) │ │ quiet = False │ │ record = False │ │ safe_box = True │ │ size = ConsoleDimensions(width=137, height=81) │ │ soft_wrap = False │ │ stderr = False │ │ style = None │ │ tab_size = 8 │ │ width = 137 │ ╰──────────────────────────────────────────────────────────────────────────────────╯ ╭── ───╮ │ Windows features available. │ │ │ │ ╭─────────────────────────────────────────────────╮ │ │ │ WindowsConsoleFeatures(vt=True, truecolor=True) │ │ │ ╰─────────────────────────────────────────────────╯ │ │ │ │ truecolor = True │ │ vt = True │ ╰─────────────────────────────────────────────────────╯ ╭────── Environment Variables ───────╮ │ { │ │ 'TERM': None, │ │ 'COLORTERM': 'truecolor', │ │ 'CLICOLOR': None, │ │ 'NO_COLOR': None, │ │ 'TERM_PROGRAM': 'vscode', │ │ 'COLUMNS': None, │ │ 'LINES': None, │ │ 'JUPYTER_COLUMNS': None, │ │ 'JUPYTER_LINES': None, │ │ 'JPY_PARENT_PID': None, │ │ 'VSCODE_VERBOSE_LOGGING': None │ │ } │ ╰────────────────────────────────────╯ platform="Windows" ```

[github-actions[bot]]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

[TomJGooding]

I suspect you'd see the same error with plain old print?

with open("out.txt", "w") as f:
    print("\u2248", file=f)

[willmcgugan]

Add encoding="utf-8" to your call to open.

[github-actions[bot]]

I hope we solved your problem.

If you like using Rich, you might also enjoy Textual