CSI u

[pickfire]

I know this is unrelated to colors but I wish there is a similar comparison for CSI u.

Only iterm2 that I know supports it https://iterm2.com/documentation-csiu.html, maybe other terminals do as well. IIRC neovim supports it out of the box.

Not sure where to create an issue for this so putting it in this issue.

[XVilka]

@pickfire I think, it's better to create a separate repository for this purpose. If you make a draft with README.md as we have for colors, I could create a new repository in this organization and we will start our new crusade 😆

[pickfire]

Maybe we can get started with something simple first?

CSI u
=====

Olden days terminals encode certain keys as the same keypress. For example,

- Tab is Ctrl-I

CSI u extends the terminal encoding specification to allow encoding any
possible keypress uniquely.

<http://www.leonerd.org.uk/hacks/fixterms/>

Maybe explain how CSI u works?

CSI u Support in Output Devices
===============================

Supported
---------

### Terminal Emulators

iterm - https://iterm2.com/documentation-csiu.html

[kurahaupo]

i support the idea of a new repo under the standardterm organisation.

Or at least, to create a new file.

[pickfire]

Then we put in some minimal stuff and throw it to lobste.rs reddit or something, to let people fill it in.

[kurahaupo]

@pickfire any suggestion(s) for a project or repo name?

[pickfire]

csiu? Not sure if there is a better name.

[kurahaupo]

I was thinking more of a name like termstandard/keychord that focuses on the purpose rather than the method, and has a similar structure to the name of this repo.

[kurahaupo]

BTW...

I note the last comment in LeoNerd's page suggests sending CSI as a bare 0x9b byte rather sending than esc+[.

I think that suggestion is seriously flawed.

Firstly, it makes dealing with UTF more awkward for the programs inside the terminal. As things stand the control sequence parser and the UTF-8→UCS-32 conversion can happen as separate steps, and in either order.

This proposal would force them to become intertwined. The control sequence parser would have to:

1. know whether or not UTF-8 is in use, and
2. if so, look for UTF-8 lead bytes, and not treat byte 0x9b as CSI if it occurs within a valid (or degenerate) UTF-8 sequence, and
3. decide on a synchronisation policy, to cope with a truncated UTF-8 sequence (where a UTF-8 continuation byte is expected but some other byte is received)

Secondly, although Xterm did initially recognize 0x9b as CSI - essentially alt-esc - that made it incompatible with:

1. the CP-125x encoding family (where it usually encodes the › character);
2. mouse-position reporting beyond column (or line) 123;

which meant it had to be turned off most of the time anyway.

It's basically on the wrong end of history; it's going, not coming back, and for many good reasons.

[leonerd]

I note the last comment in LeoNerd's page suggests sending CSI as a bare 0x9b byte rather sending than esc+[.

I think that suggestion is seriously flawed.

Yeah turns out it's not great. Though there aren't really any other suggested alternatives to fix the "Escape key needs timing" problem. If you don't use S8C1T mode you're basically stuck in the realm of encoding an "" key using some longer sequence of bytes, perhaps "CSI 27; 1 u" or somesuch. Doing that means existing programs looking for a literal ESC byte no longer see it.

It's not great but overall S8C1T seems to be a simpler solution than anything else.

[leonerd]

I was thinking more of a name like termstandard/keychord that focuses on the purpose rather than the method, and has a similar structure to the name of this repo.

"keychord" would be a bit of a weird name. I'd probably go with something like "key modifiers".

[leonerd]

Also if anyone's interested I had a go at starting to write some better docs/spec at

https://docs.google.com/document/d/1Ywq9eVHlGhpGodG7sSd1EbxsTeqJptlhCrYmMSyOn8E/edit#

I kinda got stuck because I realised I need more people to ask me more questions about it, which I can then answer in the form of writing more docs. Can anyone help?

[o-sdn-o]

I have some thoughts on the CSIu protocol. Perhaps someone has already stated that a different (more general) keyboard protocol is needed, since 'CSIu' is not well suited for terminals/multiplexers/applications for the following reasons:

• Not all possible keyboard modifiers are tracked. No division into left and right modifiers (actually, keyboard modifiers tracking by the protocol is redundant, see below).
• No support for virtual keys to fully support virtual/on-screen keyboards and touchbars.
• No support for keyboard scancodes to track pressing combinations, regardless of the selected keyboard layout. This is critical for remote use by systems with distinct locales.
• The Shift key is not transmitted in most cases. I would like the application to decide how to react to the pressed buttons, not the protocol.
• I would like that if an application requests the use of the keyboard protocol, then it should ignore all input outside of this protocol. The text generated by the terminal outside the keyboard protocol is only for the exceptional case where the application crashes without disabling the keyboard protocol.
• I don't need to support key combos from the protocol side. My application only needs to track key presses and releases. The application will keep track of the key combinations on its own.
• My application requesting the keyboard protocol is able to keep track of the current state of the keyboard modifiers and there is no need to duplicate this work in the protocol.

[kurahaupo]

"keychord" would be a bit of a weird name. I'd probably go with something like "key modifiers".

I (used to) play the piano, so "chord" seemed to me like the most obvious word for "hitting several keys at once", but yeah, it's probably not obvious to non-musicians.

A repo name has to be a single "word", though it can be hyphenated.

[kurahaupo]

@o-sdn-o This does seem to be veering off towards something that would be better handled through VNC, RDP or X-over-ssh, rather than a layer over the tty subsystem.

However if you still want to pursue it then I think it's worth implementing it as an entirely separate protocol that can be enabled as an alternative to CSI-u.

[kurahaupo]

It turns out that the "timing problem" is a phantom.

Because all the bytes of a control sequence are sent to the master pty with a single write() call, this guarantees that they're all available to a single read() call on the slave pty.

Conversely, separate key events from the display server will result in separate calls to write() them to the master pty, and the client will receive them in separate read() calls.

The magic is simply to check the length of the read:

#include <termios.h>
...
struct termios t;
tcgetattr(0, &t);
t.c_lflag &= ICANON;
t.cc[VMIN] = 1;
t.cc[VTIME] = 0; /* <<< key part of the magic */
tcgetattr(0, &t);
...
char b[MAX_CS_LEN];
ssize_t nb = read(0, b, sizeof b);
if (b[0] == 27)
  if (nb == 1)
    handle_esc_key();
  else
    handle_csi(b,nb);

Ssh uses a form of Nagle's Algorithm, so that it will keep this packetization intact. There's a small chance it could combine adjacent reads, but that can be dealt with by adding support to ssh for "zero timeout" when the input is a tty.

[kurahaupo]

Ideally I'd like to see this controlled by tcsetattr(), so that it's automatically turned off when the terminal goes back to "cooked" mode at the command line.

[o-sdn-o]

Because all the bytes of a control sequence are sent to the master pty with a single write() call, this guarantees that they're all available to a single read() call on the slave pty.

Control sequences will be combined into one read() when there is a heavy flow, for example, when mouse reporting is enabled.

[kurahaupo]

There are two considerations:

1. whether the master writes each event as a separate write() call; and
2. the client isn't fast enough, so that the tty buffer contains bytes from multiple sequences, which means that multiple sequences can be received with a single read() call

I suggest that we fix the first simply by making non-aggregation a requirement of this protocol.

The latter can be remedied by enabling pty remote mode, ioctl(0,TIOCREMOTE,&(int){1});, which causes a one-to-one mapping between master writes and client reads. (There are two catches: (1) that the master then becomes responsible for emulating the effects of the termios options, implementing input modes including ICANON and output translations including OCRNL, and (2) that Linux may not have implemented this ioctl yet, so we might need to submit a kernel patch. However if we're going that route then I'd suggest implementing a new line discipline instead.)

[kurahaupo]

Even without the benefit of non-aggregation or packetization, it takes a particularly perverse set of circumstances to create a pty input queue that contains CSI from multiple writes by the master.

• either

  • the user would have to type ESC and [ as separate keystrokes in quick succession; or
  • the user would have to type ESC while also moving the mouse, and the parser would have to be very naïve, failing to interpretESC ESC [ as ESC then CSI;

  and

• the reader process has to be stalled for long enough to allow a complete fake control sequence to accumulate in the pty buffer, and
• without the tpgrp changing (so not simply "stopped" inside a shell with job control), and
• without the pty buffer being flushed (so not dropping back to ICANON mode).

[SlySven]

@kurahaupo wrote:

I note the last comment in LeoNerd's page suggests sending CSI as a bare 0x9b byte rather sending than esc+[.

Well, Telnet users might have something to say on the subject of C1 control characters - RFC5198 explicitly prohibits them:

==== 2. Net-Unicode Definition

The Network Unicode format (Net-Unicode) is defined as follows. Parts of this definition are deliberately informal, providing guidance for specific profiles or rules in the protocols that reference this one rather than firm rules that apply globally.

...

3. The control characters in the ASCII range (U+0000 to U+001F and U+007F to U+009F) SHOULD generally be avoided. Space (SP, U+0020), CR, LF, and Form Feed (FF, U+000C) are exceptions to this principle, but use of all but the first requires care as discussed elsewhere in this document. The so-called "C1 Controls" (U+0080 through U+009F), which did not appear in ASCII, MUST NOT appear.

[kurahaupo]

@SlySven I completely agree with you on this; sending a bare 0x9b when in UTF-8 mode is a terrible idea.

Even in the vanishingly rare case of not being in UTF-8 mode, standards bodies and implementers alike deprecated C1 control chars decades ago.

(For anyone else reading this, UTF-8 overtook ISO-8859-1 as the most-used encoding back in 1999. It's been presumed as the encoding for all programming standards written in the last 15 years.)