zyedidia / micro

A modern and intuitive terminal-based text editor
https://micro-editor.github.io
MIT License
24.38k stars 1.16k forks source link

Binding raw escape sequences doesn't work #2950

Open friendly-bits opened 9 months ago

friendly-bits commented 9 months ago

Description of the problem or steps to reproduce

I'm using MobaXterm on Windows to connect to my Debian machine via SSH. The default key bindings for SelectWordLeft and SelectWordRight (CtrlShiftLeft and CtrlShiftRight) don't work. I checked in the raw mode (in micro) what it receives when I'm pressing these combinations and it says: EventKey: Left: "\x1bOD" EventKey: Right: "\x1bOC"

So following the instructions for binding raw escape sequences, I added this to my ~/.config/micro/bindings.json: "\x1bOC": "SelectWordRight", "\x1bOD": "SelectWordLeft",

But after that, when launching micro, it says "Error reading bindings.json: invalid character 'x' in string escape code". Exactly like in this older issue #1863 .

Specifications

Commit hash: c2cebaa3 OS: Debian Stretch (connecting from MobaXterm on Windows, through SSH) Terminal: MobaXterm

friendly-bits commented 9 months ago

My complete bindings.json file:

{
    "Alt-/": "lua:comment.comment",
    "\x1bOC": "SelectWordRight",
    "\x1bOD": "SelectWordLeft",
    "CtrlUnderscore": "lua:comment.comment"
}
JoeKar commented 9 months ago

We need to check, if the prefix of x1b in bindings.go#L155 is sufficient, since @zyedidia mentioned in keybindings.md the following:

The terminal sends these events in the form of escape sequences often (but not always) starting with 0x1b.

After some local hacking I assume, that the "\x1b" at bindings.go#L155 and inside the bindings.json needs the escape of \ and thus needs to be change to "\\x1b".

friendly-bits commented 9 months ago

@JoeKar I tried to change the lines to:

    "\\x1bOC": "SelectWordRight",
    "\\x1bOD": "SelectWordLeft",

Now micro says at launch:

\x1bOC is not a bindable event
\x1bOD is not a bindable event

I'm not sure but maybe the cause is that micro already assigns these events to the Left and Right keys? Just guessing but this is what I gather from the initial check in raw mode which produced this output when I pressed the aforementioned key combinations:

EventKey: Left: "\x1bOD"
EventKey: Right: "\x1bOC"

Edit: Maybe I misunderstood you and you meant that something needs to be changed in the program code? If so, I'll wait for something I can test. Unfortunately, I don't know Go and I'm not acquainted with the code here, so I can't really fix this in the code.

JoeKar commented 9 months ago

@JoeKar I tried to change the lines to:

    "\\x1bOC": "SelectWordRight",
    "\\x1bOD": "SelectWordLeft",

Now micro says at launch:

\x1bOC is not a bindable event
\x1bOD is not a bindable event

Ah, you're right...so while changing the stuff locally I didn't realized, that escaping the backslash in the bindings.json is already enough to forward the correct string from the parsed json to micro. Now the trick is, as already mentioned above (comment updated), that we need to adapt the bindings.go#L155 otherwise the strings.HasPrefix() fails due to unmatched prefix.

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

When I understood tcell right, then it will try to handle the escaped sequence first, after no other event handling has matched. In case you now try to use more or less "known" key events then tcell will not create his raw event and thus micro wouldn't even receive a possible registered raw event.

friendly-bits commented 9 months ago

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

Well, Left and Right are the arrow keys. I need to use them with the default actions to move around the text. My current workaround is to use AltRight and AltLeft for actions SelectWordRight and SelectWordLeft. So it's not a critical issue for me. But of course, I would prefer to use CtrlShiftRight etc because these are the defaults in most text editing applications.

friendly-bits commented 9 months ago

When I understood tcell right, then it will try to handle the escaped sequence first, after no other event handling has matched. In case you now try to use more or less "known" key events then tcell will not create his raw event and thus micro wouldn't even receive a possible registered raw event.

Sorry, I didn't get what you mean by that last paragraph and I don't even know what tcell is.

JoeKar commented 9 months ago

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

Well, Left and Right are the arrow keys. I need to use them with the default actions to move around the text. My current workaround is to use AltRight and AltLeft for actions SelectWordRight and SelectWordLeft. So it's not a critical issue for me. But of course, I would prefer to use CtrlShiftRight etc because these are the defaults in most text editing applications.

Ok, check, understood. Didn't read your description carefully enough, sorry for that.

Sorry, I didn't get what you mean by that last paragraph and I don't even know what tcell is.

No worries. According to my current understanding it looks like a bug of micro (doesn't treat the hard coded escape prefix as escaped escape) and tcell (which simply converts \x1b into a string instead of adding the same escaped escape as string in front of the sequence), at least in case the raw events shall really transport strings representing the printable escaped sequence. But I'm currently unsure, if this is all, because with my tested raw sequences I had to move the escape sequence handling in front of the normal event handling in tcell.

So long story short, it's a bit tricky to solve this issue.

dmaluka commented 8 months ago

One of the problems here is that JSON does not support the \x notation, so in bindings.json the raw sequence bindings need to be specified slightly differently:

    "\u001bOC": "SelectWordRight",
    "\u001bOD": "SelectWordLeft",

(Yeah, arguably it is a bug: micro does not attempt to properly parse \x-escaped strings, instead it simply relies on the JSON parser which does not support \x either, so the resulting behavior is non-intuitive and undocumented.)

The bad news here is that it is not the only problem. With the above \u notation, micro does not print any errors, and I can see that micro does call screen.Screen.RegisterRawSeq(), however the bindings themselves still don't work (i.e. nothing happens when I press the corresponding keys). I remember it used to work for me some (rather long) time ago, so I think it is a regression, either in micro or in tcell.

friendly-bits commented 8 months ago

@dmaluka I just tried binding this. I'm not getting any error either, and I don't get the assigned function either (SelectWordRight etc). What I get is the same as pressing the Left and Right arrow keys (which is also the behavior without the bindings).

JoeKar commented 8 months ago

@dmaluka

(Yeah, arguably it is a bug: micro does not attempt to properly parse \x-escaped strings, instead it simply relies on the JSON parser which does not support \x either, so the resulting behavior is non-intuitive and undocumented.)

Yep, but this is something which seems to be fixed easily (see below).

With the above \u notation, micro does not print any errors, and I can see that micro does call screen.Screen.RegisterRawSeq(), however the bindings themselves still don't work (i.e. nothing happens when I press the corresponding keys). I remember it used to work for me some (rather long) time ago, so I think it is a regression, either in micro or in tcell.

I was able with \\x to get them forwarded and registered to tcell (which needs a further improvement to not register the same raw event twice or more...) and after little dirty hacking I received the expected callback via raw event (in \\x notation) too. As a result the configured action has been performed. So I already assumed, that this seems to be a bug on both sides (see above) and especially tcell doesn't create the raw events in the actual situation. So in case it was working in the past then the regression has been introduced somehow introduced in tcell. Since the same situation works without double \\ in \u notation we seem to have a problem in json5 too. :(

Already prepared...

...and working to register the \x notated raw sequence. With the newly linked PRs in json5 and tcell I was able to use the raw escape sequence of Right to register the lua comment action. micro was out of scope afterwards.

JoeKar commented 8 months ago

@dmaluka Again I'd appreciate your feedback on this. ;)

dmaluka commented 8 months ago

I'll try to find time to take a close look into it.

In the meantime, I think I like the idea of adding \x support directly to the json parser (although it is non-standard, but hey).

...Actually there is one more thing that is not working: cannot bind raw sequences in micro via bind command.

JoeKar commented 8 months ago

...Actually there is one more thing that is not working: cannot bind raw sequences in micro via bind command.

I will have a look on that too, since I'm more or less comfortable with that topic.

Edit: The \x1b support for bind was the easy part, but allowing the \u notation is much more complex, because we can't simply exchange "\u" by "\u". :(

dmaluka commented 8 months ago

@blunderful-scripts While micro's existing support for raw escape bindings is indeed not quite perfect and could be improved (e.g. with JoeKar's fixes), I think it would not help you with your specific problem. If you see this when pressing CtrlShiftLeft and CtrlShiftRight:

EventKey: Left: "\x1bOD"
EventKey: Right: "\x1bOC"

it means that your problem is that your terminal sends CtrlShiftLeft and CtrlShiftRight using the same escape sequences as for Left and Right. So micro treats them as Left and Right (it has no choice).

So, if these raw bindings worked:

"\x1bOC": "SelectWordRight",
"\x1bOD": "SelectWordLeft",

the result would be: CtrlShiftLeft and CtrlShiftRight select words, but Left and Right also select words, instead of simply moving the cursor. Which is probably not what you want.

So the only advice here: fix your terminal.

dmaluka commented 8 months ago

@JoeKar So IIUC, raw escape bindings generally work, they only don't work if the same escape sequence is also recognized as a known key event (and your PR https://github.com/zyedidia/tcell/pull/25 fixes that), right?

So IIUC it is not a very critical issue (or even just a philosophical question about which of the two bindings should take precedence: the normal binding or the raw binding), since if it is recognized as a known key, the user can just use a normal keybinding. (The purpose of raw keybindings is actually to be a last resort for the cases when a sequence is not recognized as a known key.)

Still, it is probably worth fixing, at least to avoid the user's confusion when the user does use a raw keybinding (even though doesn't need to) and it doesn't work.

dmaluka commented 8 months ago

Quoting myself:

Yeah, arguably it is a bug: micro does not attempt to properly parse \x-escaped strings, instead it simply relies on the JSON parser which does not support \x either, so the resulting behavior is non-intuitive and undocumented.

I was a bit wrong: the documentation in help keybindings does mention the \u notation.

But users don't necessarily carefully read the documentation or are very familiar with json, so adding support for \x notation in bindings.json would still be nice. (Even though micro will forcefully rewrite bindings.json every time, replacing \x with the canonical \u, which is BTW also something users wouldn't expect, but at least keybindings will keep working.)

friendly-bits commented 8 months ago

it means that your problem is that your terminal sends CtrlShiftLeft and CtrlShiftRight using the same escape sequences as for Left and Right

It is not the same sequence though. I get this for LeftArror and Right Arror (checking in raw): EventKey: Left: "\x1b[D" EventKey: Right: "\x1b[C"

This for CtrlSfhitLeft and Right: EventKey: Left: "\x1bOD" EventKey: Right: "\x1bOC"

friendly-bits commented 8 months ago

I really don't understand much about how the interpretation of escape sequences works, But I'd like to ask: maybe the way to go is detect the terminal more carefully and bind different sequences to different terminals? I don't know if there is a way for micro to know that I'm using Mobaxterm, so this may be a moot point, but I thought i'd ask it anyway.

dmaluka commented 8 months ago

It is not the same sequence though. I get this for LeftArror and Right Arror (checking in raw):

Got it. I've been assuming that if the event received by micro from tcell is the same, e.g. Left, it means that the terminal escape sequence that originated it is the same too. Apparently it's not always the case, i.e. tcell may convert different escape sequences (from the same terminal) to the same event.

So yeah, I was wrong, using raw escape keybindings should be a good workaround for this particular problem. (Only that it doesn't work, as you've found. But it should work with @JoeKar's PRs. I haven't tested them myself yet, but I'm going to.)

maybe the way to go is detect the terminal more carefully and bind different sequences to different terminals? I don't know if there is a way for micro to know that I'm using Mobaxterm, so this may be a moot point, but I thought i'd ask it anyway.

It's a moot topic. The tcell library (the library used by micro, in particular, for receiving events from the terminal) distinguishes between terminals via the TERM environment variable, and then it cares a lot to correctly interpret different sequences from different terminals (but it will never be perfect). But that is how it works on Unix systems. On Windows it works very differently, I have no idea how exactly.

friendly-bits commented 8 months ago

It's a moot topic. The tcell library (the library used by micro, in particular, for receiving events from the terminal) distinguishes between terminals via the TERM environment variable, and then it cares a lot to correctly interpret different sequences from different terminals (but it will never be perfect). But that is how it works on Unix systems. On Windows it works very differently, I have no idea how exactly.

Ok, then another question: is there a way to manually communicate to tcell (or micro) the correct terminal so one of them knows to assign correct actions to correct sequences? I know that there is an option to launch micro with --xterm (or something similar) option, so maybe there is a way to add a similar option for other terminals (as the need arises)? or is this too complicated?

friendly-bits commented 8 months ago

I mean, personally I'm fine with figuring out the correct escape sequences and manually changing the config. But then I'm not the only one using this software, and wouldn't it be nice to have this option so people don't need to figure all this out on their own?

JoeKar commented 8 months ago

@JoeKar So IIUC, raw escape bindings generally work, they only don't work if the same escape sequence is also recognized as a known key event (and your PR zyedidia/tcell#25 fixes that), right?

Yes, it was, but...

dmaluka commented 8 months ago

Ok, then another question: is there a way to manually communicate to tcell (or micro) the correct terminal so one of them knows to assign correct actions to correct sequences?

On Linux and other Unix systems, the way you tell micro the correct terminal is by setting the TERM env variable (if the terminal itself hasn't properly set it already), and if a particular terminal is not properly supported by tcell yet, you are free to contribute. On Windows, I don't know.

...I've just realized that your situation is a bit peculiar: micro is running on Linux (via SSH), while the terminal is running on Windows, right? So, micro (and tcell) actually thinks that it is running in a Unix terminal, not a Windows one. And yet micro is working for you, so I guess MobaXTerm behaves like a Unix terminal, so it might be good news: you may not need to care about the fact that it is running on Windows, so you may try to add missing support for MobaXterm to https://github.com/zyedidia/tcell/tree/master/terminfo as a unix terminal, and then accordingly set e.g. TERM=mobaxterm (on your remote Debian machine) to tell micro and tcell that it is running in MobaXterm.

Or maybe the entire problem should be approached differently, e.g. maybe MobaXterm properly emulates xterm (so no need to add special support for MobaXterm) but there is some bug in tcell. So I would try debugging tcell, to figure out why exactly does it e.g. report the same Left key event for both \x1b[D and \x1bOD.

...I don't use Windows, but if I was you, I would maybe start with simple experiments like: try using other Windows terminals and compare the behavior (to see if this is an issue with MobaXterm or with Windows terminals in general), try running micro locally on Windows instead of SSH and compare the behavior, try running apps other than micro (e.g. vim) and compare the behavior, and so on.

friendly-bits commented 8 months ago

if I was you, I would maybe start with simple experiments like: try using other Windows terminals

On Putty, it's exact same behavior as in MobaXterm, and exact same raw escape sequences.

on Kitty, CtrlShiftRight & Left work in micro correctly. Raw output:

(for CtrlShiftLeft & Right) EventKey: Shift-Ctrl-Left: "\x1b[1;6D" EventKey: Shift-Ctrl-Right: "\x1b[1;6C"

(for Left & Right) EventKey: Left: "\x1b[D" EventKey: Right: "\x1b[C"

I'm starting to wonder if the Putty escape sequences for CtrlShiftLeft & Right are specific to Putty and Putty-like terminals (which MobaXterm might be for this matter), in which case it would make sense to just map them to the correct commands in tcell (or in micro).

I can try to look at the tcell code but chances that I can figure this out are quite slim, since I really know nothing about Go. One thing I'm noticing while trying to make some sense of the code is that some/all "import" commands in these files point to non-existing paths. e.g. import "github.com/zyedidia/tcell/v2/terminfo" There is no v2 under tcell/

Edit: When running natively on Windows, CtrlShiftLeft & Right work correctly, although micro doesn't detail escape sequences in raw mode.

Edit2: I see that x1bOC and x1bOD are specifically assigned to Left and Right in https://github.com/zyedidia/tcell/blob/master/terminfo/x/xterm/term.go

I'm wondering why? Checking natively (not through SSH) on my second Linux Mint machine, I'm getting x1b[C and x1b[D for Left and Right arrow buttons (which incidentally fits the ANSI spec for xterm) https://en.wikipedia.org/wiki/ANSI_escape_code#Terminal_output_sequences image .

friendly-bits commented 8 months ago

I did find some references to x1bOC and x1bOD being used by vt100 terminals or similar. References:

https://studmed.uio.no/elaring/voila/venv/lib/python3.9/site-packages/prompt_toolkit/input/ansi_escape_sequences.py http://www.manmrk.net/tutorials/ISPF/XE/xehelp/html/HID00000594.htm https://github.com/SDL-Hercules-390/hyperion/blob/master/hconsole.h https://www.osti.gov/servlets/purl/5889325

I may be wrong but I suspect that someone mistakenly copy-pasted these escape sequences into tcell without checking whether they indeed apply to xterm.

friendly-bits commented 8 months ago

After some google-fu, I found this website which goes into (very much) detail about keyboard escape seqeunces. To say that I understand less than 5% of it is an understatement. However, this passage seems quite clear. image This is less clear [to me] but probably useful to someone here: image

Here's the page, in case someone smarter than myself can decode what's written in it. https://invisible-island.net/xterm/terminfo-contents.html#tic-xterm_with_cursor_keys_in_normal_mode

JoeKar commented 8 months ago

On Putty, it's exact same behavior as in MobaXterm, and exact same raw escape sequences.

on Kitty, CtrlShiftRight & Left work in micro correctly. Raw output:

It seems you already gave the answer by yourself between the lines:

PuTTY isn't able to transmit the correct key code not matter if application mode is set or not and how TERM is set...tcell can't fix this, as long as it receives the same key code for e.g. left and c-s-left. Now it seems that MobaXterm behaves like PuTTY or even uses the same code base:

Mobaxterm is based on PuTTY (starting with its source-code), and adds some key-bindings to imitate xterm, but the set of key-bindings is not complete.

What you can try to do is if cat reports exactly the same as the raw mode of micro or not, because this bypasses tcell: https://superuser.com/questions/1536450/why-is-tmux-not-recognizing-ctrlshiftarrow-through-mobaxterm

friendly-bits commented 8 months ago

tcell can't fix this, as long as it receives the same key code for e.g. left and c-s-left.

As I mentioned above, the actual escape sequences are not the same for c-s-left and for left. The escape sequence MobaXterm transmits for c-s-left (as reported by cat) is ^[OD and for left it's ^[[D. It so happens that (as far as I understand) ^[OD is the same as some old VT100 terminals were using for 'left'. I couldn't really figure out whether any modern computer actually transmits this for 'left' but my impression is that it's a thing of the past that somehow made its way into xterm and subsequently into tcell. That's (in my understanding) what this aforementioned passage says. The original assignments from patch #94 for cursor-keys had some technical issues .... For these reasons, the original assignments were deprecated. For compatibility reasons, they are still available as a setting of xterm's modifyCursorKeys resource.

So what I'm saying is, perhaps there is no need to force everyone who uses Putty and Putty-based terminals to "live with it" or to modify the config file, if there is a simple solution of programmatically interpreting the (likely no-longer used) VT100 escape sequence for 'left' as (actually actively used by many people) Putty escape sequence for c-s-left.

JoeKar commented 8 months ago

As I mentioned above, the actual escape sequences are not the same for c-s-left and for left.

Ok, then we've at least the chance to handle this. But before we go further into tcell please test some other stuff:

With the last hint tcell will fallback to a dynamical terminfo (best guess) setting and maybe this changes something. In case not it's a good point start hacking. But most probably a dedicated setting for PuTTY/MobaXterm in tcell is better approach.

I know that there is an option to launch micro with --xterm [...]

The option xterm set to true will only overwrite a possible set TERM with xterm, which is...as far as I read...the setting of PuTTY(-likes) too.

friendly-bits commented 8 months ago
  • change switch between application and normale mode in PuTTY and/or MobaXterm and see if the respective code for your sequences changes
  • force with PuTTY and/or MobaXterm to indicate the TERM other than xterm (at least PuTTY should be able to do so) to something unknown to tcell (e.g. TERM=putty).

Tested in putty (because I don't know if it's even possible to switch between application mode and normal mode in mobaxterm - I couldn't find this setting).

Both settings don't affect transmitted/received escape sequences for c-s-left and c-s-right. That said, reading the putty user manual, I became aware that the 2 modes (normal and application) seem to be still a thing, I'm not sure for what purpose. https://documentation.help/PuTTY/config-appcursor.html

Now according to that manual, this is supposed to be set by the server, although the client (putty) can disable the application mode. I can't tell whether anybody or anything is still using the application mode, but it sounds like it's safe for a server application (such as micro) to disable application mode while it's running and then (just in case) to restore the previous mode when exiting. I assume that there is an API to do that (and possibly micro or tcell is already doing that?)

The idea for disabling the application mode while micro is running is to avoid a (possible?) regression where someone's cursor keys stop working because of c-s-left and c-s-right escape sequences override.

friendly-bits commented 8 months ago

Forgot to mention that also, changing both settings doesn't affect how micro reacts to c+s+left and c+s+right when editing (i.e. the result is still the same as pressing 'left' and 'right').

JoeKar commented 8 months ago

Ok, tried it myself and realized that PuTTY simply doesn't forward the Shift as a modifier key and therefore Left = Shift+Left and Ctrl+Left = Ctrl+Shift+Left. From my point of view this is a limitation/bug of PuTTY and can't be (simply) solved, without changing PuTTY itself.

friendly-bits commented 8 months ago

Ok, tried it myself and realized that PuTTY simply doesn't forward the Shift as a modifier key and therefore Left = Shift+Left and Ctrl+Left = Ctrl+Shift+Left. From my point of view this is a limitation/bug of PuTTY and can't be (simply) solved, without changing PuTTY itself.

Interesting - I confirm that this is the behavior in putty. However, this is not the behavior in mobaxterm.

raw in micro: pressing Left: EventKey: Left: "\x1b[D" pressing Shift-Left: EventKey: Shift-Left: "\x1b[1;2D" pressing Ctrl+Left: EventKey: Ctrl-Left: "\x1b[1;5D" pressing Ctrl+Shift+Left: EventKey: Left: "\x1bOD"

friendly-bits commented 8 months ago

That said, I don't want to waste your time on mobaxterm-specific solution, specifically for me. So unless other people need this, probably this issue can be closed. (especially if I can assign the raw escape sequence in the config file after your fix gets merged)

grinapo commented 5 months ago

I have to confess that I was not patient enough to read all the 100+ comments here. Still I hope I can be relevant.

I am new to micro, so I read the documentation. When I realise that micro does not work through ssh/tmux/... because it gets the escape sequences and does not interpret them, I started looking for why and failed (it seems either documentation is oblivious about escapes not supported [yes I see the mention, but not the resolution], or maybe it's supported but my version isn't latest, v2.11).

Anyway. The documentation talks about \x1b, and 0x1b and it seems neither works (as they are) in the config. \uXXXX seems to be vaguely mentioned but it's not obvious for the causal reader that despite how it's referred to and how raw mode displays the \u001b form is what works, others aren't. This is mainly the problem of the documentation (it should be spelled out that \x1b shall be written as \u001b), but inconsistent with raw output as well.

Summarizing:

  1. the page keybindings.md >> Binding raw escape sequences shall be consistent about ESC format and be unabiguous about how the config file expects it.
  2. either \x1b should work in config file or raw should be fixed?
  3. a few words about what's expected through xterm->ssh->tmux (I get \x1b[1;5C for CtrlRight which shall be in fact WordRight in config).
  4. doc as well: may mention that how the current bindings could be printed. If possible, I mean. :-)
JoeKar commented 5 months ago
  1. either \x1b should work in config file or raw should be fixed?

It would work with the config files as well, when zyedidia/json5#1 would be merged. Unfortunately I can't speed this up, since I've no merge rights. Afterwards the dependency within micro needs to be bumped.

  1. a few words about what's expected through xterm->ssh->tmux (I get \x1b[1;5C for CtrlRight which shall be in fact WordRight in config).

From my perspective this highly depends on the users local scenario and mentioned every possibility is too much for the documentation of micro itself. But I confirm, that the documentation should be consistent.

  1. doc as well: [...]

Currently it makes no sense, at least in my opinion, because that part need to be changed again after the above mentioned PR is merged (if and when ever). But everyone is encouraged to support with his own PR. :wink:

grinapo commented 5 months ago

From my perspective this highly depends on the users local scenario and mentioned every possibility is too much for the documentation of micro itself. But I confirm, that the documentation should be consistent.

I meant a brief list what's included and what's not, like:

I have started writing bindings for ESC[ sequences but I was wondering whether it's already have been created somewhere by someone, since I do not like to reinvent warm water.

4.

I am too fresh meat here to start rewriting the docs, so I ask around first! :-) But that one contained an implicite question as "can I print the current bindings please?" ;-)