bug: normal mode hardcoded with english layout

[whoizit]

this bug is thousands of years old, it is present in vi / vim / nvim / any vim-like editors I have tried and in the post-modern editor it is also present. This is super sad.

It must use key-codes, not key-chars or key-chars must be translated to key-codes on the fly.

Affected any national non-English and non-CJK layouts

[kirawi]

To be fair, won't most programmers already be typing Latin characters? I also think most keyboards support Latin characters. But I'm genuinely interested in why you want this.

[whoizit]

so is this a text editor or a code editor? Constantly switching the layout to get work in default mode is a pain for everyone in the world who speaks more than just English. And who writes more than just code.

[kirawi]

You make a valid point, this should definitely be a feature. However, it's unclear to me why you wouldn't be able to use alternate mappings on vim/neovim/etc. I'd think this is probably the route Helix should go as well.

[itzmjauz]

You make a valid point, this should definitely be a feature. However, it's unclear to me why you wouldn't be able to use alternate mappings on vim/neovim/etc. I'd think this is probably the route Helix should go as well.

Alternate mappings are the sensible solution to this problem(wouldn't call it a bug), the alternative is a very bloated base configuration. Especially given the way crossterm keycode mappings work https://docs.rs/crossterm/0.19.0/crossterm/event/enum.KeyCode.html

[archseer]

As someone that uses an alternate layout too (Norman) I can sympathize with the problem. This needs some changes in crossterm: We'd need to be able to receive raw position-based key codes but crossterm only offers us key chars.

[pickfire]

I also use an alternative layout (dvorak) and use IME to type chinese and japanese. I don't see an issue here besides the invalid IME position. But yeah, maybe non-latin (but non-pictographs) are an issue. What do you suggest?

[Kethku]

I've recently been dealing with this issue on other projects. The good news (once a gui lands) is that winit will very soon have the tools to fix this problem for real (this wasn't the case a few months ago). I'm interested in helping out in this area as its fresh in my mind at least on the gui side

[pickfire]

so is this a text editor or a code editor? Constantly switching the layout to get work in default mode is a pain for everyone in the world who speaks more than just English. And who writes more than just code.

I speak cantonese and mandarin and malay as well but I don't have any issue with the defaults. At least I still use a US keyboard so this must have affected those that use non-us keyboard like in europe.

I think there should be a flag. I still prefer to use key chars rather than keycode. The default keys for dvorak is still good for me to use it.

A temporary solution right now is to use the key mappings configuration. @whoizit you can map out the keys and submit it to contrib/ or something so that others is able to use it. Or maybe have something like the current theme configuration where you can just :keymap <keys>.

[archseer]

showkey --scancodes internals: an ioctl call can set KDSKBMODE to RAW for scancodes (MEDIUMRAW to get keycodes again)

[blaggacao]

If there is a possibility to access scancodes directly past the terminal, it would be possible to avoid having to implement CSIu mappers like this one (you see the extend of the problem) in order get access to some key combinations.

It would be a solution at the root.

Maybe helpful:

• https://docs.rs/evdev/0.11.0/evdev/
• https://docs.rs/evdev/0.11.0/evdev/struct.Key.html

A.f.a.i.r. and as the evdev rust docs say:

The Linux kernel’s “evdev” subsystem exposes input devices to userspace in a generic, consistent way.

It looks like a definite route to obviate CSIu hacks for the century-old ASCII limitations on key combinations.

[whoizit]

I am not (yet) a helix user, I have not read the thread (hard work for me, not english speaker). Look how it works in sway:

Bindings to keysyms are layout-dependent. This can be changed with the --to-code flag. In this case, the keysyms will be translated into the corresponding keycodes in the first configured layout. https://github.com/swaywm/sway/wiki#key-bindings-on-a-dual-usrussian-layout swaywm/sway#5409

so i can do even so:

set $bindsym bindsym --to-code

$bindsym {
...
    $mod+s layout stacking
    $mod+w layout tabbed
    $mod+e layout toggle split   
...
}

all I changed from the default (us-only configuration) is bindsym replaced with bindsym --to-code and began to use $bindsym variable instead bindsym keyword. And it worked no matter what layout I'm on

do not confuse bindsym and bindcode

[blaggacao]

For a text editor to become an extension of your brain through your fingers in 21st century, we need to stop this awful indirection going through the visual cortex (a.k.a. key char aliases) just to make modal keystroke decisions.

I guess a blind person would be a real good "post modern" advisor, but just start by imagining a world where there are no markers on keycaps at all, all blank. :stuck_out_tongue_winking_eye:

In my opinion, the implementation primitives should embrace some representation of "muscle memory" intent-locators instead of chars and make char a higher order abstraction of that. Universal evdev scan codes are the thing that comes closest to that. Only for the modal commands, that is, of course: for regular text input, surely use whatever is configured.

AFAIK key codes are constrained by those awful ASCII terminal constraints

[archseer]

But you're forgetting that not all keyboards have the same physical layout. I find sway's behavior acceptable because I can bind the thumb cluster based on my default layout without having to figure out scancodes for each key.

[whoizit]

by the way, someone may not use the us layout at all, never

[blaggacao]

But you're forgetting that not all keyboards have the same physical layout.

Very true, I really just wanted to complement the discussion with a radically different angle in the hopes that this angle can make helix a radically better editor.

In terms of locators, each finger has it's own coordinate system from the resting position. So the matrix is not a parallel one but a curved one, according to the form of the hand.

Rectangular keyboards are just a very shameful violation of (human) hand ergonomics and should not be the inspiration for our primitives (they are a special "synchronized" case of the "ergonomic" coordinate system). Less chars, which do not have any vector information at all w.r.t. the resting position of any particular finger.

Note that flexing vectors are "ergonomically" cheaper, than stretching vectors. At least for my hand and an ergonomic keyboard, that makes the row below the home row real second-to-prime estate.

Ergonomically, this completely invalidates using char aliases as primitives for modal commands. — albeit, they surely are great helpers for the initial setup (+ maybe an ease in period until muscle memory is sufficiently trained).

I hope I don't sound too radical, though. :smiley_cat:

[blaggacao]

just use classic keyboards by swiping diagonally across the keys and everything will be comfortable

That somewhat works, but it requires elbow movements for smaller hands.

But don't get me wrong, I just want to strengthen the point why char references are bad, so that we can have layout-independent-modal commands as a collateral outcome of a new paradigm. :sweat_smile:

[voroskoi]

Hi,

I'm just being curious: How do You handle the case with other programs? So You set up Your editor to keycodes, but every other application still uses other keys. Do You reconfigure them too? For example vim bindings are quite common, it seems quite a lot work to setup everything.

I use hungarian keyboard, searching(/) is SHIFT+6 for example, which is quite bad, but I haven't changed it as I have to press this for changing dirs and so too, using different keys seems strange. I could use us layout, but then I can not use accent keys (éáű etc.).

I have found https://github.com/samvel1024/kbct which seems a global solution for this problem. OTOH if I remap / to é for example, then I can not type é anymore, so this is just changing layout to us. What I would really use is something that lets me use my accent chars in insert-mode and use the same keys to search and so in normal mode, but with every program. (Actually that still would not work, because I may have directories with accents chars in its name, so typing ~/é would be quite difficult when / is bould to é). Am I over-complicating this?

[gagbo]

Hello,

I'm also interested in looking out how to advance on this subject, I'm using bépo (a french dvorak-like variant), and it'd be really nice if it were possible to have the possibility to use scancodes or some sort of keycode translation to describe default bindings. It would greatly reduce the amount of rebinding I have to do with editors.

For the time being, I only found the implementation in SDL, which apparently is only able to go back from keycode to scancode by manually testing all scancode candidates. Because of this I think that hoping to access scancodes directly is a tough sell, and to avoid perf related issues, I guess the "way to go" would be to:

• allow keycodes to be used in the (Rust) configuration, so that default configuration is still mostly "position based" for all keyboard layouts. (Even with my layout that has ctsr for hjkl, the keycodes for ctsr are still 43..46, just mapped to different keysyms)
• allow users to override bindings in TOML using either
  • "natural" notation which would describe keys just like today
  • "keycode" notation which would take the integer code values as keys for bindings

With this, there'd be no remapping for anyone, and non-qwerty users would have a way to get at least home-row normal mode bindings out of the box. A lot of mnemonic keys would get moved to weird places though (in my examble, yanking becomes ^, finding becomes e...), but I'd rather deal with those on a case by case basis.

It probably has already been proposed but I'm not sure I saw it, if I'm saying obviously bad stuff please correct me,

Regards, Gerry

[kirawi]

I don't know if it's possible, but maybe we could provide that above function within Helix?

[pickfire]

A simpler alternative is rewrite the keymap for specific keybinding.

Even as a dvorak user myself I don't find this useful. Probably need to wait for someone that is interested enough in this to fix it or just create a keymap which is less effort and works well.

[blaggacao]

I do a continue to do custom build for this. thanks to flake.nix that's easy.

[NNBnh]

Kitty's Comprehensive keyboard handling in terminals protocol might be helpful.

[VKondakoff]

Hello! Are there any progress on this topic? Currently it is almost impossible to use Helix on a systems with two (or more) keyboard layouts: when you switch keyboard layout to a non-Latin one, the Normal mode stops to respond. There are several workarounds in Vim/Neovim intended to overcome this issue: langmaps, keymaps, xkb-switching. All of them have pros and cons. What are the Helix devs planning on this issue? (I hope Helix will not become an editor, which supports only Latin keyboard layout). Thanks!

[pickfire]

Let me resend the same message.

A simpler alternative is rewrite the keymap for specific keybinding.

Even as a dvorak user myself I don't find this useful. Probably need to wait for someone that is interested enough in this to fix it or just create a keymap which is less effort and works well.

No progress on this until someone say they are working on this or someone submits a pull request. At least I don't think any of the contributors are very interested in this now, if people think it is important there should already be a patch by now.

Which is also why I labeled the issue as E-help-wanted.

[whoizit]

@pickfire I guess you still not understand issue, dvorak it's english layout, but this issue about non-english layout's like russian, finnish, franch, italiano and thousands others

[VKondakoff]

I've recently been dealing with this issue on other projects. The good news (once a gui lands) is that winit will very soon have the tools to fix this problem for real (this wasn't the case a few months ago). I'm interested in helping out in this area as its fresh in my mind at least on the gui side

Can you, please, bring a little bit more light here? What are these tools? Is something landed? Thanks!

[kirawi]

https://github.com/rust-windowing/winit/issues/753?

[pickfire]

@pickfire I guess you still not understand issue, dvorak it's english layout, but this issue about non-english layout's like russian, finnish, franch, italiano and thousands others

Ah, now I get what you meant. Like uk or japanese keyboard (at least these are the ones I have seen) which have extra keys.

[whoizit]

no, it's not about extra keys. You need to switch the layout to English in order to interact with the editor, that's what this is about.

[VKondakoff]

This ⬆.

Here is a short video. I'm pressing a simple key sequence i<Esc>v<Esc>s<Esc> in Normal mode, while my keyboard layout is set to En, then I repeat the same keys when keyboard layout is set to Ru:

https://user-images.githubusercontent.com/230824/142721405-b179dabc-891b-4925-bfbf-bdb24790aa39.mov

[pickfire]

Ah, I get it now. I wonder if scankey getting the keycode is really the way to go for this since my dvorak original keys works just fine.

How did other editors solve this, like vim or kak? Did they add in the default keys for other non-latin keymap or get the key code?

[VKondakoff]

As far as I can say there are three main methods of fixing this issue:

1. Langmaps: set langmaps to be able to enter Normal commands without switching the keyboard layouts. :set langmap=ФИСВУАПРШОЛДЬТЩЗЙКЫЕГМЦЧНЯ;ABCDEFGHIJKLMNOPQRSTUVWXYZ, фисвуапршолдьтщзйкыегмцчня;abcdefghijklmnopqrstuvwxyz
2. Keymaps: switch between keyboards independently from a system-wide keyboard settings. :set keymap=russian-jcukenmac
3. Auto-switch the system keyboard to English layout when leaving Insert mode and switch back to the layout which was used when you enter Insert mode again.

[NNBnh]

3. Auto-switch the system keyboard to English layout when leaving Insert mode and switch back to the layout which was used when you enter Insert mode again.

Out of these three, this is the only option that work with Chinese, Japanese and Vietnamese.

[pickfire]

Out of these three, this is the only option that work with Chinese, Japanese and Vietnamese. (and Korean)

But it needs integration with IME like https://github.com/vim-scripts/fcitx.vim and it's not usually the default for CJK. Also, non CJK users probably don't use an IME like in the case of russian which makes it not so useful for non-CJK cases. Sadly these plugins are not built-in, both vim and kak needs to install a separate plugin for it to work even though it's just a few lines of code.

[eugenesvk]

5. [Auto-switch the system keyboard to English layout]

Out of these three, this is the only option that work with Chinese, Japanese and Vietnamese.

but unfortunately it introduces some mess in all the others, if I'm using several apps and switch to a non-English layout to copy/edit/paste some text in Helix and a browser, I wouldn't want Helix to mess up my system layout on context switches, because then I'd lose the predictable state in my browser (also the flashes of the language indicator aren't great)

[pickfire]

but unfortunately it introduces some mess in all the others, if I'm using several apps and switch to a non-English layout to copy/edit/paste some text in Helix and a browser, I wouldn't want Helix to mess up my system layout on context switches, because then I'd lose the predictable state in my browser (also the flashes of the language indicator aren't great)

IME usually have the option to set state per application, so if you use a language in terminal, you can use different language in browser.

[eugenesvk]

IME usually have the option to set state per application

Ok, so the inelegant mess is only introduced to those users that don't have this setting enabled :) (I don't as I found it more confusing than helpful)

Also, I have many other tabs in the terminal, so if I'm typing in a non-English Helix Insert mode and switch to another tab, the language would presumably not switch back to English as I haven't quit the Insert mode yet, right?

[pickfire]

Also, I have many other tabs in the terminal, so if I'm typing in a non-English Helix Insert mode and switch to another tab, the language would presumably not switch back to English as I haven't quit the Insert mode yet, right?

Yes, it won't but I never had the need to do that.

[whoizit]

https://github.com/aveplen/ruscmd.nvim

[nanoqsh]

Helix is a great editor and I'm sure it's the future. But I'm saddened by the non-support of different keyboard layouts out of the box. Although vim has the same problem, I expect a more modern editor to fix this. Similar to how Rust supports utf-8 compared to C and I no longer have to think about special handling of non-latin texts.

But how to do that? As I understand it, supporting different layouts requires different solutions. You can give the ability to set keys by specifying keycodes, but how convenient will it be for the case of different language layouts?

[keys.normal]
"34" = { "1c" = "code_action" }

Users just want to specify mappings like C-s and have it just work. At the same time, without duplicating it like this:

[keys.normal]
"C-s" = ":w"
"C-ы" = ":w"

So why not do something like vim's "langmaps"? When I press a key (not in insert mode) the editor will try to resolve the input to the given "lang" mappings:

[langmapping]
"ы" = "s"
"Ы" = "S" # Perhaps this duplication can be avoided
"ф" = "a"
# ..

So (almost) all previously set mappings will work. For most users this feature will be enough until key codes supporting is implemented. For many languages a default configuration can be provided, which will only need to be enabled: langmapping = "cyrillic". Make a directory from where the corresponding .toml file will be imported, how it works with themes now.

Unfortunately, this solution is not absolute. Different layouts don't match not only in letters, but also in symbols too. For example, the Shift + 2 enters either @ or " depending on the layout. So this problem is much more fundamental and rests on the initially crookedly designed layout.

That's pretty much how I see it. Perhaps this feature should come up with a more descriptive name.

[whoizit]

All existing solutions offered in (n)vim editor are crutches, they are either incomplete and still make you uncomfortable, or do not work in wayland.

Also, entering command mode should always switch the layout to English, because all commands in command mode are entered in English, always

In the Windows operating system, this problem did not exist for me at all. All hotkeys always worked the same in both languages in any program. But I haven't used Windows for 15 years and it's a matter of principle.

But Linux in the 21st century still has great programs that only usable for those who have one layout, English (and it looks like for CJK-guys). This programs like (n)vim and mplayer/mpv. And all the 15 years I have been using Linux I have suffer when using this programs.

We simply have to get rid of the steamy inheritance and rewrite these programs

[pickfire]

@whoizit I believe you can try solving this first by using adding a contrib file to map the keys.

I wonder if it is good to have a set of translation, since it may not always be a one-on-one translation for the keys, if a language have lesser keys than latin then it means that some keys could not be pressed, so I think it might be good to have the keys being remapped completely.

But still, haven't think it is a good idea to put the keys by default because the infobox on the bottom right will still display the keys to be pressed, then non-latin keys come out, it will be confusing for those people not using the keys.

[whoizit]

good solution from Sway WM:

• https://wiki.archlinux.org/title/Sway#Keybindings_and_keyboard_layouts
• https://github.com/swaywm/sway/pull/3058

[SoraTenshi]

Another approach could be to provide some kind of matrix, which then internally will remap the keys based on the location within the matrix. Unfortunately i am not sure how to express this in toml but in general i mean something like this:

[editor.keys.matrix]
matrix = [
"qwfpgjluy;[]",
"arstdhneio'",
"zxcvbkm,./"
]

something like this or alike. this could resolve this issue for other people (like me, using colemak) in a rather simple way.

[VKondakoff]

At least right now trying to add Cyrillic letter to 'config.toml' fires an error Bad config: unexpected character found: `\u{444}` at line 2 column 1 Press <ENTER> to continue with default config

Here is a test config:

[keys.normal]
ф = "move_char_left" # Maps the 'a' key to the move_char_left command
ц = "move_line_up" # Maps the 'w' key move_line_up
[keys.insert]
j = { k = "normal_mode" }

[pickfire]

Try with " on the key?

[VKondakoff]

Thanks, it works with the quotes like this:

[keys.normal]
"р" = "move_char_left" # Maps the 'р' key to the move_char_left command
"л" = "move_line_up" # Maps the 'л' key to move_line_up command
"д" = "move_char_right" # Maps the 'д' key to the move_char_right command
"о" = "move_line_down" # Maps the 'о' key to move_line_down command
[keys.insert] 
"о" = { "л" = "normal_mode" }
"j" = { "k" = "normal_mode" }

BTW, there is nothing about quotes in the docs:

# At most one section each of 'keys.normal', 'keys.insert' and 'keys.select'
[keys.normal]
C-s = ":w" # Maps the Control-s to the typable command :w which is an alias for :write (save file)
C-o = ":open ~/.config/helix/config.toml" # Maps the Control-o to opening of the helix config file
a = "move_char_left" # Maps the 'a' key to the move_char_left command
w = "move_line_up" # Maps the 'w' key move_line_up
"C-S-esc" = "extend_line" # Maps Control-Shift-Escape to extend_line
g = { a = "code_action" } # Maps `ga` to show possible code actions
"ret" = ["open_below", "normal_mode"] # Maps the enter key to open_below then re-enter normal mode

[keys.insert]
"A-x" = "normal_mode" # Maps Alt-X to enter normal mode
j = { k = "normal_mode" } # Maps `jk` to exit insert mode

[koiuo]

BTW, there is nothing about quotes in the docs:

you are looking at the wrong docs https://toml.io/en/v1.0.0#keys

Bare keys may only contain ASCII letters, ASCII digits, underscores, and dashes (A-Za-z0-9_-).

Quoted keys follow the exact same rules as either basic strings or literal strings and allow you to use a much broader set of key names.

[HealsCodes]

Just another voice for a properly implemented langmap-like feature. Contrary to the comments about it being a crutch.. it's not if you use it correctly and it works on any WM and any platform that has a vim.

langmap is accompanied by an additional command called langnoremap in vim/nvim. The later being responsible for preventing keys issued by plugins to be remapped. With that you can very easily fix your non-insert-mode to work for most layouts without breaking functionality.

" swap j/k/l keys with d/r/n to get movement keys on German Dvorak
set langmap=dj,rk,nl,DJ,RK,NL,jd,kr,ln,JD,KR,LN
" don't override mappings for plugin generated input
set langnoremap

[vwkd]

+1

VSCode gets this right.

Currently, helix needs an additional two keystrokes for every input to switch to / from foreign language.