caksoylar / keymap-drawer

Visualize keymaps that use advanced features like hold-taps and combos, with automatic parsing
https://caksoylar.github.io/keymap-drawer
MIT License
660 stars 57 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte #111

Closed eristocrates closed 4 weeks ago

eristocrates commented 4 weeks ago

I'm on windows 10, instillation with pipx worked fine and all that, and parsing qmk. I'm having an issue with unicode (maybe multibyte in general?) characters. At first i thought in my config.yaml that trans_legend: t: �� type: trans was the culprit. I even tried changing that line to ▽▽ and even '' and the error (log below) persisted. It's almost like my config isn't applying it? I'll doublecheck how i'm trying to use my config, but to add to the strangeness, my actual keymap.yaml has lines like

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "c:\users\reill.local\bin\keymap.exe__main.py", line 7, in File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\keymap_drawer__main.py", line 212, in main draw(args, config.draw_config) File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\keymap_drawer__main__.py", line 22, in draw yaml_data = yaml.safe_load(args.keymap_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\init__.py", line 125, in safe_load return load(stream, SafeLoader) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\init.py", line 79, in load loader = Loader(stream) ^^^^^^^^^^^^^^ File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\loader.py", line 34, in init Reader.init(self, stream) File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\reader.py", line 85, in init__ self.determine_encoding() File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\reader.py", line 124, in determine_encoding self.update_raw() File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\yaml\reader.py", line 178, in update_raw data = self.stream.read(size) ^^^^^^^^^^^^^^^^^^^^^

caksoylar commented 4 weeks ago

Hey, unfortunately when I experimented in Windows I ran into similar issues. I couldn't find a set of settings to resolve it, so I documented a workaround here: https://github.com/caksoylar/keymap-drawer?tab=readme-ov-file#%E2%84%B9%EF%B8%8F-windows-command-line-issues

tl;dr is to use -o output instead of stdout redirecting like >output.

caksoylar commented 4 weeks ago

Related: #89

eristocrates commented 4 weeks ago

Yep that took care of it! thank you very much for the workaround. Shame about windows, but that's really on me for dragging my feet on leaving windows lmao

eristocrates commented 4 weeks ago

actually, i had another related issue and figured it'd be better to keep them on the same issue. Is there any way to have japanese kana drawn? i just tried and even with the -o flag and it didn't work.

caksoylar commented 4 weeks ago

Can you give an example of what you're trying? I can try to reproduce here.

(I actually use Windows full time but all dev work happens in WSL.)

eristocrates commented 4 weeks ago

just a small snippet of a single layer from keymap.yaml. keymap draw works without issue in this example

  - ['X', 'V', 'G', 'M', 'P', {t: '*', s: Arcane}, {t: '%', s: Arcane}, 'U', 'O', 'Y', 'B', 'Z']
  - ['J', 'K', 'S', 'N', 'D', {t: '=', s: Arcane}, {t: ':', s: Arcane}, 'A', 'E', 'I', 'H', {t: QU, s: Q}]
  - [{t: Leader, s: Core}, 'W', 'F', 'L', 'C', {t: '\', s: Arcane}, {t: '/', s: Arcane}, (, {t: ',', s: Adaptive, h: OSM Shift}, {t: '.', s: Adaptive, h: Capsword}, ), {t: Vim, s: Genuine, h: Emulation}]
  - [{t: Num Pad, h: MO Layer}, {t: R, h: CTRLR}, {t: Space, h: SML}, {t: Space, h: SMR}, {t: T, h: ALTT}]

however, if i attempt to replace X with た, like so - ['た', 'V', 'G', 'M', 'P', {t: '*', s: Arcane}, {t: '%', s: Arcane}, 'U', 'O', 'Y', 'B', 'Z'] it fails with the error below. it seems kana is not recognized as a string?

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "c:\users\reill.local\bin\keymap.exe__main.py", line 7, in File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\keymap_drawer__main.py", line 212, in main draw(args, config.draw_config) File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\keymap_drawer\main__.py", line 43, in draw drawer = KeymapDrawer( ^^^^^^^^^^^^^ File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\keymap_drawer\draw\draw.py", line 24, in init self.keymap = KeymapData(config=config, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\reill\AppData\Local\pipx\pipx\venvs\keymap-drawer\Lib\site-packages\pydantic\main.py", line 193, in init self.pydantic_validator__.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 1 validation error for KeymapData layers.t Input should be a valid string [type=string_type, input_value={'t': 'Num Pad', 'h': 'MO Layer'}, input_type=dict] For further information visit https://errors.pydantic.dev/2.8/v/string_type

even more strangely, on inspecting the keymap.yaml after running draw, there seems to be duplication on some {t:} keys. note the {t: {t: Num Pad below

     - ['た', 'V', 'G', 'M', 'P', {t: '*', s: Arcane}, {t: '%', s: Arcane}, 'U', 'O', 'Y', 'B', 'Z']
  - ['J', 'K', 'S', 'N', 'D', {t: '=', s: Arcane}, {t: ':', s: Arcane}, 'A', 'E', 'I', 'H', {t: QU, s: Q}]
  - [{t: Leader, s: Core}, 'W', 'F', 'L', 'C', {t: '\', s: Arcane}, {t: '/', s: Arcane}, (, {t: ',', s: Adaptive, h: OSM Shift}, {t: '.', s: Adaptive, h: Capsword}, ), {t: Vim, s: Genuine, h: Emulation}]
  - [{t: {t: Num Pad, h: MO Layer}, h: MO Layer}, {t: R, h: CTRLR}, {t: Space, h: SML}, {t: Space, h: SMR}, {t: T, h: ALTT}]
caksoylar commented 4 weeks ago

I am guessing this is an encoding issue with the input file. keymap-drawer always expects a utf-8 encoded file and Windows text editors might not default to that.

I can get this file working here, uploaded verbatim (had to change extension to txt): kana.txt

When you open this in notepad (or notepad++), you can see it shows utf-8 encoding: image

If I open a new tab in notepad and paste this in, it also seems to work because it defaults to utf-8 in my case. But I am guessing the file you are trying isn't. You can try figuring out what it is, or upload it here like I did so I can have a look. But the solution will be to save it in utf-8 encoding.

eristocrates commented 4 weeks ago

Ah, remember when i mentioned earlier that I had to use a separate script before calling draw? that was the culprit! I'm 100% taken care of now, tysm for the support o7