[BUG] UnicodeEncodeError: 'charmap' codec can't encode character

[woctezuma]

Description I want to export the list of apps to a text file using the Windows console, and there is an UnicodeEncodeError error.

Steps to Reproduce the behavior

Run: steamctl apps list > apps_list.txt

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main
    rcode = cmd_func(args=args)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list
    print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id)))
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to <undefined>

Expected behavior The software should run without any error, and the text file should contain the lists of appIDs along the app names.

What actually happend

$ head apps_list.txt

5 Dedicated Server
7 Steam Client
8 winui2
10 Counter-Strike
20 Team Fortress Classic
30 Day of Defeat
40 Deathmatch Classic
50 Half-Life: Opposing Force
60 Ricochet
70 Half-Life

$ tail apps_list.txt

208090 Unknown App 208090
208140 ENDLESS▒ Space - Definitive Edition
208157 ENDLESS▒ Space - Amplitude Badge + GAMES2GETHER Points
208286 Train Simulator: Sherman Hill Route Add-On
208540 Dungeon Defenders Lucky Community Costume Pack
208546 Dungeon Denders - Jester Hero
208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARS▒ Knights of the Old Republic▒ II: The Sith Lords▒
208610

The error happens for appID=208610 which is Skullgirls ∞Endless Beta∞. I believe the culprit is the ∞ character.

Logs

steamctl -l debug

``` [DEBUG] steamctl: Parsed args: {'versions_report': None, 'log_level': 'debug', 'anonymous': False, 'user': None, 'password': None, 'command': 'apps', '_cmd_func': 'steamctl.commands.apps.gcmds:cmd_apps_list', 'subcommand': 'list', 'all': False} [DEBUG] SteamClient: Reading CM servers from 'C:\\Users\\Wok\\AppData\\Local\\steamctl\\steamctl\\client\\cm_servers.json' [DEBUG] CMServerList: Added 80 new CM addresses. [DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser [INFO] SteamClient: Reusing previous username: XXX [INFO] SteamClient: Hint: use 'steamctl --user ...' to change [DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser [INFO] SteamClient: Attempting login with remembered credentials [DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\XXX.key [DEBUG] SteamClient: Attempting login [DEBUG] SteamClient: Connect initiated. [DEBUG] Connection: Attempting connection to ('162.254.192.75', 27017) [DEBUG] Connection: Connected. [DEBUG] SteamClient: Emit event: 'connected' [DEBUG] SteamClient: Incoming: | ChannelEncryptRequest)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Securing channel [DEBUG] SteamClient: Outgoing: | ChannelEncryptResponse)> [DEBUG] SteamClient: Incoming: | ChannelEncryptResult)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Channel secured [DEBUG] SteamClient: Emit event: 'channel_secured' [DEBUG] SteamClient: Outgoing: | CMsgClientLogon)> [DEBUG] SteamClient: Incoming: | CMsgMulti)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Multi: Unpacking [DEBUG] SteamClient: Multi: Decompressing payload (655 -> 1207) [DEBUG] SteamClient: Incoming: | not parsed)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Incoming: | not parsed)> [...] [DEBUG] SteamClient: Incoming: | CMsgMulti)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Multi: Unpacking [DEBUG] SteamClient: Multi: Decompressing payload (20905 -> 68577) [DEBUG] SteamClient: Incoming: | not parsed)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Emit event: 'job_4' [DEBUG] SteamClient: Incoming: | CMsgMulti)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Multi: Unpacking [DEBUG] SteamClient: Multi: Decompressing payload (24145 -> 72215) [DEBUG] SteamClient: Incoming: | not parsed)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Emit event: 'job_4' [DEBUG] SteamClient: Incoming: | CMsgMulti)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Multi: Unpacking [DEBUG] SteamClient: Multi: Decompressing payload (3153 -> 9775) [DEBUG] SteamClient: Incoming: | not parsed)> [DEBUG] SteamClient: Emit event: [DEBUG] SteamClient: Emit event: 'job_4' [DEBUG] Connection: Disconnected. [DEBUG] SteamClient: Emit event: 'disconnected' Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main rcode = cmd_func(args=args) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id))) File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to ```

Versions Report

steamctl --versions-report

```yaml steamctl: 0.9.5 Dependencies: steam: 1.4.3 appdirs: 1.4.4 argcomplete: 2.0.0 tqdm: 4.64.1 arrow: 1.2.3 pyqrcode: 1.2.1 beautifulsoup4: 4.11.1 vpk: 1.4.0 vdf: 3.4 gevent-eventemitter: 2.1 gevent: 22.10.2 greenlet: 2.0.1 pyyaml: Not Installed pycryptodomex: 3.15.0 protobuf: 3.20.3 Python runtime: executable: C:\Users\Wok\.local\pipx\venvs\steamctl\Scripts\python.exe version: 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] platform: win32 System info: system: Windows machine: AMD64 release: 10 version: 10.0.19044 ```

[woctezuma]

There is no error if I run:

 steamctl apps list

I can even get the ∞ character to be rendered correctly:

208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARS™ Knights of the Old Republic™ II: The Sith Lords™
208610 Skullgirls ∞Endless Beta∞
208620 Midnight Mysteries: Salem Witch Trials
208630 Midnight Mysteries 3: Devil on the Mississippi
208640 Midnight Mysteries 4: Haunted Houdini

However, the list is really long, so it gets truncated by the Windows console, and it would be handier to save it to a text file.

I remember encountering this issue in the past, without reporting it, and then editing the code to save the output to a file. I wonder if there is a better fix.

On a side-note, I am not sure if the issue only happens on Windows.

[woctezuma]

The error can be reproduced with a Python file scratch_1.py, unrelated to steamctl:

print('∞')

---

python scratch_1.py

Output:

∞

---

python scratch_1.py > temp.txt

Output:

Traceback (most recent call last):
  File "C:\Users\Wok\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_1.py", line 1, in <module>
    print('∞')
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 0: character maps to <undefined>

[rossengeorgiev]

Use a terminal that supports utf8 and is configured to use utf8

[woctezuma]

Use a terminal that supports utf8 and is configured to use utf8

That is the case. The issue is with the redirection to a file, which does not seem to work properly on Windows.

For instance, in the picture below, there is no error, but an inconsistency.

echo ∞
echo ∞ > temp.txt
type temp.txt

I can fix the inconsistency with chcp 65001, but that does not fix the error when redirecting the output of a Python script.

chcp 65001
echo ∞
echo ∞ > temp.txt
type temp.txt

For the error with Python, I have a small workaround.