Protocol extension: multiple cursors

[maximbaz]

More and more text editors are adopting the concept of multiple cursors, I'm playing with one of them called kakoune.

In order to draw multiple cursors consistently, kakoune has to hide the terminal cursor completely and "draw" its own cursors by manipulating background color of a cell. This is ugly hack that prevents people from using the much superior native cursor with all of its features (different shape, blinking, proper color).

In vim you only see one cursor, even if you are editing multiple lines, but there you are not editing multiple lines simultaneously anyway (vim waits for ESC to be pressed, then applies the same edit on the rest of the lines).

In kakoune, multiple lines are being edited at the same time, potentially in different positions, so it's really beneficial to show multiple cursors.

---

What I was thinking about is to propose you to think about multiple cursors as an extension over xterm protocol.

I asked for how this works in layman terms, I was told applications like kakoune or vim take care of handling keypresses and rendering the text on the screen, and then they tell the terminal to put a cursor in a certain place, kind of like move (x,y). What is missing is the ability to draw more cursors, e.g. like draw (x1,y2). These "secondary" cursors should basically be a copy of the main cursor in terms of color, shape and blinking. And then there should be a way to remove all "secondary" cursors from the screen.

I brought this to you because you understand a lot more about the terminals, and I want to hear your thoughts. I hope I managed to at least explain the need for this feature: multiple cursors is a thing that is becoming more and more popular and demanded by users (here's just one example in neovim), unless terminal allows drawing them, software has to completely take over drawing cursors, and of course software-rendered cursors cannot ever have the features like shapes and colors and blinking that native cursors easily provide.

[kovidgoyal]

I have added a "cursors" terminfo property to the spec that applications can use to detect support for this feature.

[mawww]

To give a bit of input, its likely on Kakoune side we would just clear and resend cursors on every redraw. This is because the display backend (which uses ncurses at the moment) does not know anything about cursors (so far) except for the main cursor location (which I added only to let the terminal know a sane place to locate their insert related widgets). It would not know how to relate one cursor from one draw to another cursor from the next draw.

I wonder if other editors would have other needs, but for Kakoune, a single sequence defining all cursors (and implicitely removing any previous ones) would be good enough, which means we would never need to identify a specific cursor.

[kovidgoyal]

On Thu, Jul 19, 2018 at 11:29:36PM -0700, Maxime Coste wrote:

To give a bit of input, its likely on Kakoune side we would just clear and resend cursors on every redraw. This is because the display backend (which uses ncurses at the moment) does not know anything about cursors (so far) except for the main cursor location (which I added only to let the terminal know a sane place to locate their insert related widgets). It would not know how to relate one cursor from one draw to another cursor from the next draw.

Thanks for the input. So the current proposal meets your needs? In pseudo-code, I image you would do something like this, to display the cursors:

for cursor in cursors: write(_Ci=counter++,r=cursor.x,c=cursor.y)

And to delete: write(_Ca=d)

You simply output the delete sequence at the start of your display loop and the creation sequences at the end.

[egmontkob]

APC is not ignored by Terminal.app. How does VTE treat it?

As of the current development series (forthcoming 0.54), thanks to Christian's parser rewrite, it ignores them.

Current stable 0.52 prints garbage. Given that it'll take a few years for this feature to get widely adapted, I don't mind this.

[egmontkob]

If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden.

If the application is maintaining a list of extra cursors anyway. What if not? What if it uses just another bit per cell in its already existing internal WxH matrix denoting what's already on the screen (i.e. treats them as if they were text attributes, as you told mawww it'd be fine)?

ids are more robust.

I don't see it that way, I see them as an additional burden for no real use.

They support multiple cursors at the same location.

A feature you would use for... for what? (I don't mind it as a by-product of any design, but I don't see any use for it that could make it a pro argument.)

Not to mention that knowing the location is not always easy (think of wide characters).

I'm not sure I get it. I can't imagine a text editor that doesn't have a consistent internal representation (including wide characters) of what it believes is on the screen. If the code places a cursor at (x,y) then removing a cursor specifying the same (x,y) coordinates will remove that. If an app places an extra cursor at the real cursor's current location, then later repositioning the real cursor there and removing the extra cursor should (see below) also remove it.

As it is I see a lot of bug reports caused by the fact that most editors use out of date wcwidth() implementations that dont agree with kitty which uses one derived from the unicode 11 standard. This is the main reason I chose to use ids.

If there's a misagreement (which unfortunately does happen every once in a while, just as well as another app (e.g. write(1)) can pollute the display) then contents fall apart anyway. Hence most editors offer a shortcut to redisplay their contents, which, depending on the nature of the breakage, may or may not fix it. I understand that using IDs could make the display of extra cursors more robust against these kinds of scenarios, but I honestly don't see any point in having this as a goal, if it's broken then it's broken anyway in ways that make it hard or practically impossible to use the editor at those lines of the file. A few leftover cursors IMHO doesn't make it worse then, it's just as bad.

Thanks for your explanation, though, I can now see what's your rationale behind this design. I just disagree with the priority order of the factors behind this decision :-) I think ease of use, even from apps that don't maintain the cursors as a list, or from terminal multiplexers that would have to maintain a mapping with your proposal, is a more important factor than damage control.

[vertical scrolling] Still it's not an absolute no-no from me.

Another use case that occurred to me is if the editor crashes. This is not something that has never happened to me. The prompt might be displayed amidst the leftovers from the file's contents, it might be colored (but my prompt does reset that), there's no scrollback buffer since we're still on the alternate screen, but I just press a couple of Enters to get to the bottom of the screen and it's usable. Repeating myself: I wouldn't want cursors to remain there cluttering my command line.

[one escape sequence for multiple operations]

We've heard about at least one app that would always clear all existing extra cursors and place them again (rather than keeping track of IDs, moving them etc.). Being able to specify multiple actions in a single sequence has the small advantage that the terminal emulator can handle all of them as an atomic step. That is, even if the connection stalls while receiving them, it could avoid the visual flickering of removing and later at the same place reinstalling some extra cursors. My bad for not mentioning this the last time. It's by far not the most important issue here, but one that we might consider nevertheless.

[kovidgoyal]

On Fri, Jul 20, 2018 at 09:16:58AM +0000, egmontkob wrote:

If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden.

If the application is maintaining a list of extra cursors anyway. What if not? What if it uses just another bit per cell in its already existing internal WxH matrix denoting what's already on the screen (i.e. treats them as if they were text attributes, as you told mawww it'd be fine)?

It doesn't matter how it stores them. In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.

Not to mention that knowing the location is not always easy (think of wide characters).

I'm not sure I get it. I can't imagine a text editor that doesn't have a consistent internal representation (including wide characters) of what it believes is on the screen. If the code places a cursor at (x,y) then removing a cursor specifying the same (x,y) coordinates will remove that. If an app places an extra cursor at the real cursor's current location, then later repositioning the real cursor there and removing the extra cursor should (see below) also remove it.

Remember that the escape code places the extra cursor at the current cursor position, bu default, which apps often dont know, and cannot in general know unless they either query the emulator or have a wcwidth() implementation that exactly matches that of the emulator. Obviously when the app specifies the location (x, y) in the escape code, it knows the location. When it does not, it may or may not. ids on the another hand it always knows.

And you forgot about scrolling and resizing of the screen. If we make the extra cursors move with scrolling then the app has to update their locations just to address them. Similarly after a resize it cannot know the new locations. And given that screen scroll all the time, for example when wrappiing is enabled, this is not a small issue.

ids are just better :) And sorry, but needing to add two extra lines of code in applications that dont care about ids is not an undue burden, to make the implementation robust for applications that do care about addressing individual cursors.

[vertical scrolling] Still it's not an absolute no-no from me.

Another use case that occurred to me is if the editor crashes. This is not something that has never happened to me. The prompt might be displayed amidst the leftovers from the file's contents, it might be colored (but my prompt does reset that), there's no scrollback buffer since we're still on the alternate screen, but I just press a couple of Enters to get to the bottom of the screen and it's usable. Repeating myself: I wouldn't want cursors to remain there cluttering my command line.

If the editor crashes, I dont see how you can possibly hope to recover other than by doing a reset. Everything from colors, line discipline, echo mode, all the dozens of bbits of state in termios and the emulator, the keyboard mode, etc, etc, etc could be wrong. And reset or clear removes extra cursors. Terminals are not robust against program crashes and never will be. It's pointless worrying about it.

[one escape sequence for multiple operations]

We've heard about at least one app that would always clear all existing extra cursors and place them again (rather than keeping track of IDs, moving them etc.). Being able to specify multiple actions in a single sequence has the small advantage that the terminal emulator can handle all of them as an atomic step. That is, even if the connection stalls while receiving them, it could avoid the visual flickering of removing and later at the same place reinstalling some extra cursors. My bad for not mentioning this the last time. It's by far not the most important issue here, but one that we might consider nevertheless.

If the connection stalls in the middle there is no visual flickering, you get some cursors that appear before and some cursors that appear after. Just as you get some text that appears before and some text that appears after. Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will. But if they ever do, whatever mechanism they use for it, can just as well apply to this protocol just as it would have to apply to all the rest.

[kovidgoyal]

And just because I'm bored, let's do a bit of simple maths to calculate the probability of their being one extra cursor on the screen after catting a 10MB binary file.

You need a minimum of an 8-byte sequence to create an extra cursor. The probability of such a sequence is ((1/255)*7) (1/245) which is 1e-20 (technically its a bit higher since you can have longer than 8 byte sequences as well that create extra cursors, but I doubt the contribution from them will be significant compared to the shortest sequence). There are approx 1e-6 8 byte sequences in a 10 MB file. Which leaves you with a first order probability of 1e-14 of seeing a one single extra cursor on your screen after catting a 10MB file. I can live with that.

[egmontkob]

In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.

If the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs.

If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs is an additional, nontrivial burden.

Similarly after a resize it cannot know the new locations.

Why can't? Won't it be part of the specification?

Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will.

Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something shouldn't change, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible.

[kovidgoyal]

On Fri, Jul 20, 2018 at 03:06:00AM -0700, egmontkob wrote:

In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.

If the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs.

If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs is an additional, nontrivial burden.

Huh? It is precisely in that case that having ids is superior to not having them. If the application wants to address individual cursors in the terminal, then it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor). Which means there is no extra burden giving them an id in the escape code, simply add an id field to the internal structure which uses a globally incrementing counter, again two lines of code.

Similarly after a resize it cannot know the new locations.

Why can't? Won't it be part of the specification?

No, at least I dont have any interest in adding it to the specification, but if you do, feel free to send a PR.

Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will.

Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something shouldn't change, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible.

What shouldn't change? You are sending n cursor creation escape codes one after the other. The only thing that can happen is that they might not appear on the screen at the exact same moment. Nothing can appear that later should not appear since you are transmitting the escape codes immediately after each other, regardless of how many pauses there are in transmission.

[egmontkob]

it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor)

This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come up with something new as well.

What shouldn't change? You are sending n cursor creation escape codes one after the other. [...]

Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker.

[kovidgoyal]

On Fri, Jul 20, 2018 at 03:24:14AM -0700, egmontkob wrote:

it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor)

This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come with something new as well.

Sigh, this gain? If it is 1-bit per cell, then individual cursors are not addressable by the application. The application does not care about ids and must delete all cursors on each round anyway and so can use a simple global counter for the ids.

Let me spell it out for you in pseudo code.

struct {
   bool has_extra_cursor;
   SGRAttributes text_formatting_attributes;
   char *ch;
} Cell;

struct {
    Cell *cells;
} Line;

counter = 1
write(<delete all cursors>);
for line in lines:
   for cell in cells:
      write(cell.text_formatting_attributes)
      if cell.has_extra_cursor:
        counter += 1;
        write(<esc>_Cid=counter<esc>\)
      write(cell.ch);

As I said, literally two extra lines of code. Initialize the counter and increment it.

What shouldn't change? You are sending n cursor creation escape codes one after the other. [...]

Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker.

You mean between clearing the cursors and re-creating them? That would be present regardless, even if the protocol allowed you to create multiple cursors in one escape code.

If you mean you want to be able to both delete and create cursors in the same escape code, I have no idea how that would work while still remaining small and relatively easily parseable. I supose one culd do something like this

_C;;...\_ Doesn't seem important enough to me to bother.

[egmontkob]

As I said, literally two extra lines of code.

Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).

You mean between clearing the cursors and re-creating them? That would be present regardless, even if the protocol allowed you to create multiple cursors in one escape code.

The terminal emulator could decide whether to process individual requests as it encounters them, or batch them up until the terminating <ESC>\ sequence and then process them all, atomically. In case of VTE, due to the way its parser used to work (and I presume still works as of the rewrite, although I haven't checked), we'd do the second approach anyway, it'd be atomic.

[egmontkob]

Two extra lines of code

Sorry, I have to correct myself, I responded too quickly.

Screen drawing is probably more sophisticated than the pseudo-code you outlined. I mean they probably do some minimal diff-ing between the old and new buffers, e.g. update only the lines that changed, and maybe even within the line they omit the beginning and trailing portion that remained the same.

With your approach, they'd need a bit more than those 2 lines to re-process the extra cursors in lines that didn't change.

I'm not saying it's hard to do, but I'm still saying it's an unnecessary additional complexity.

[kovidgoyal]

On Fri, Jul 20, 2018 at 03:47:16AM -0700, egmontkob wrote:

As I said, literally two extra lines of code.

Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).

What on earth have ids to do with flickering? At this point you are just arguing for the sake of arguing.

And I'll just note in passing that cursors blink, which means that flickering for them is meaningless.

Hmm, I think I have had it with this discussion. I'm afraid I no longer have the motivation to implement it if it involves this level of argumentation.

@everyone: sorry for wasting your time. If somebody else wishes to implement this in their terminal emulator feel free to use my work as a baseline, or not, as you wish.

[lenormf]

The maintainer of tmux had a few comments about the design, hope this helps:

Their choice of escape sequence for this seems unnecessarily different from everything else. Why make it delimited rather than using something with arguments? Why make it comma-separated when ; or : are the usual separators? If you are going to use key-value pairs, why not make them comprehensible rather than single letters? If you want my opinion on the design then I would personally forget all this i=x,c=y,r=z stuff and just have a cursor set and cursor clear sequence. There seems no harm in requiring the existing cursor to move first like for tab stops. I'd just do something like add some extended WINOPS:

\e[99999;0;10t <-- add cursor ID 10 at current main cursor position
\e[99999;1;10t <-- delete cursor ID 10
\e[99999;1t <-- delete all extra cursors
\e[99999;2;10;x;y;zt <-- and so on, whatever else you want, cursor colour, etc

You are much more likely to get support if the escape sequence is similar to or extends an existing sequence rather than making everyone add a parser for yet another form.

[egmontkob]

What on earth have ids to do with flickering?

With the code snippet you showed, the removal of the cursor is followed by potentially up to a few kilobytes of text data (and attribute changing escape sequences), then re-installing the cursor at the same position. The amount of data inserted in between could easily make them no longer fit in the same ethernet or similar packet, eventually noticeably increasing the chance of a network lag kicking in.

And I'll just note in passing that cursors blink

or not (oh, wait, it seems to me that in Kitty it always blinks)

which means that flickering for them is meaningless.

A flickering during the "on" state of a blinking cursor is IMHO still a bit annoying.

At this point you are just arguing for the sake of arguing.

No. I'm trying to argue (although I'm really lost) because I have a bad feeling that no matter how I phrase my thoughts, I cannot make them come across to you. I mean, if I felt you understood what I'm saying and still disagreed, that'd be fine. But unfortunately I feel like you don't understand what I'm saying (maybe don't take the time or don't have the motivation to try to understand what I'm saying?? I'm really unsure) then it's very hard. I sincerely hope that my feelings are incorrect. I really don't know what I am doing wrong if my thoughts don't come across clearly.

I was just about to quit this conversation and let you continue in your preferred way. I'm sorry you did it first and not me, and sorry from others too. This enhancmenet request was filed against Kitty, it's your project, do as you wish. If you reopen and continue the work, I promise I'll stay out of the way.

[kovidgoyal]

Sorry, but this discussion has made it clear to me that I lack the patience to steward something like this. As I said, feel free to implement it in your own terminals using my existing proposal as a base or not. If and when some proposal becomes widely adopted/used by editors, I will be happy to implement it in kitty.

But I am done spending my valuable time creating proposals just to engage in endless meaningless arguments about them.

[leonerd]

@justinmk wrote:

Actually, I'd be more interested in new text attributes as @mawww hinted. E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

This would be nice. Totally independently of extra cursors, I could see this being useful for other cases besides. They'd be very easy to add as new numbered SGR attributes

[leonerd]

@egmontkob wrote:

SGR 51, 60, 62 might be relevant here.

I hadn't heard of those before. Do you have some relevant documentation? I'd like to add them to my spreadsheet, where I'm trying to keep things like this organised:

https://docs.google.com/spreadsheets/d/19W-lXWS9jYwqCK-LwgYo31GucPPxYVld_hVEcfpNpXg/edit#gid=836709407

(I'm happy to add/edit things in there for other terminal emulators, also)

[kovidgoyal]

@leonerd it's in wikipedia https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters

[justinmk]

So the "hardly ever supported" (51-65) SGR parameters most likely will continue to lack support in kitty, iterm2, VTE, because of struct space?

[kovidgoyal]

If you want them supported, I am not adamantly opposed, but you will need to make a case for it to convince us :) kitty actually has space in its struct at the moment. The downside of supporting this in kitty will be extra per pixel operations in the shader. There are also some subtleties to be aware of, namely those are meant for ideographic characters, I dont know if it is appropriate to simply use them for ordinary characters.And one has to consider what happens with them on wide characters.

[kovidgoyal]

And if you do want to make a case, I suggest opening a new enhancement report for it, this one is already quite long :)

[egmontkob]

Do you have some relevant documentation?

No, I don't. All I know is what's written in ECMA-48 and the Wiki page linked by Kovid. I don't even know what "ideogram" means, nor have an idea what to do with "underline or right side like", I mean which one?

The feature could be useful for nice and compact representation of tables. Double width lines could be achieved by placing e.g. a character with a right side line and a next character with a left side line next to each other. Not sure if it's worth 4 new bits per cell, though. Not sure how we are with free bits in VTE, the main developer recently reorganized the bits and I haven't checked the code since. Just as Kovid, I'd also prefer to see actual (not hypothetical) use cases, utilities willing to add support.

If we'd like to move forward, I'd like to propose 4 new attributes for the 4 sides. SGR 51 ("boxed" or "framed") might be a shortcut for enabling all 4 at a time. The wording of SGR 60 and 62 are so unclear to me that I wouldn't want to use them, unless someone has a clarification on them. SGR 53 (overlined, hardly ever used) may be used for the top one, or may be obsoleted by the top one.

I'd like the bottom one to be separate from the underline. The underline goes slightly below the baseline of the font, and might be subject to "text-decoration-skip-ink" rendering, curliness, different color (or would we want that for the frame too?) etc. The border bits would I imagine go to the very edge of the character cell, so that adjacent ones are joined even in case of increased line spacing or character spacing.

[egmontkob]

As for the very concrete escape sequences, we might be thinking along the lines of 51:n (a subparameter just as with the underline style), the second parameter going from 0 to 15, being a 4-bit bitmap for the four sides. That way we don't pollute the "global" SGR space with 4-5 new entries.

[kovidgoyal]

I agree with Egmont about 51:n and these being independent of underlines. Those were the choices I would have made myself. The only question is whether to use 51 or some other number. After all some terminal emulators out there may already use 51.

I am curious as to what the use case for this is though. Since the border lines could potentially overlap with the rendered character, I dont know how nice it would look (think of wide characters like W that touch the edges off the cell in most fonts. But, as I said, feel free to make your case, preferably in a separate bug report.

[egmontkob]

51 could remain an alias for 51:15, that is, turning on all four sides. Existing implementations, however, may not put the frame at the very border of the cell, which – as far as I understand – would be a requirement for the new one. So I don't mind introducing a brand new number. We've recently assigned 58 and 59 to colored underline, but 56 and 57 are still free, aren't they? And we'd only need one.

I agree that this whole discussion should continue in a separate issue.

[gnachman]

As long as we're bikeshedding, I'd like to add an extension to enable atomic drawing so that when the entire screen gets redrawn it doesn't flash a half-drawn screen for a fraction of a second before the rest of it appears. I have a hack in iTerm2 that tries to use whether the cursor is visible to provide synchronization but it is far from perfect. Since that came up in this discussion, I wonder if anyone would be open to the idea? If so I'll open a new issue and write up a proposal.

[kovidgoyal]

Sure, I know this bother's Egmont a lot, so it's worth discussing. It should be trivial to have an escape code that turn on and off "pending mode". In pending mode all received bytes would be buffered. You'd probably also need a max pending timeout and a max buffer size.

[egmontkob]

Sounds good.

[gnachman]

Link to spec and issue to host the discussion are here: https://gitlab.com/gnachman/iterm2/issues/6873