Protocol extension: multiple cursors

[maximbaz]

More and more text editors are adopting the concept of multiple cursors, I'm playing with one of them called kakoune.

In order to draw multiple cursors consistently, kakoune has to hide the terminal cursor completely and "draw" its own cursors by manipulating background color of a cell. This is ugly hack that prevents people from using the much superior native cursor with all of its features (different shape, blinking, proper color).

In vim you only see one cursor, even if you are editing multiple lines, but there you are not editing multiple lines simultaneously anyway (vim waits for ESC to be pressed, then applies the same edit on the rest of the lines).

In kakoune, multiple lines are being edited at the same time, potentially in different positions, so it's really beneficial to show multiple cursors.

---

What I was thinking about is to propose you to think about multiple cursors as an extension over xterm protocol.

I asked for how this works in layman terms, I was told applications like kakoune or vim take care of handling keypresses and rendering the text on the screen, and then they tell the terminal to put a cursor in a certain place, kind of like move (x,y). What is missing is the ability to draw more cursors, e.g. like draw (x1,y2). These "secondary" cursors should basically be a copy of the main cursor in terms of color, shape and blinking. And then there should be a way to remove all "secondary" cursors from the screen.

I brought this to you because you understand a lot more about the terminals, and I want to hear your thoughts. I hope I managed to at least explain the need for this feature: multiple cursors is a thing that is becoming more and more popular and demanded by users (here's just one example in neovim), unless terminal allows drawing them, software has to completely take over drawing cursors, and of course software-rendered cursors cannot ever have the features like shapes and colors and blinking that native cursors easily provide.

[kovidgoyal]

Personally, I have never needed multiple cursors, but sure, I can see how they might be useful. It isn't particularly hard to design a protocol to do this in kitty, but it should ideally have involvement from somebody who is implementing this feature in some terminal editor, so that the protocol can be designed to meet their needs.

If you can get somebody like that to join the discussion, I will be happy to work with them on designing and implementing such a protocol in kitty.

I'll leave this issue open for a while, lets see if there is buy-in from editor developers.

[maximbaz]

Good news! Let me ping a few kakoune's contributors directly, to see if anyone is interested in getting the native support for multiple cursors from terminal emulators: @mawww, @lenormf, @Delapouite, @alexherbo2, @casimir, @danr, @ekie, @occivink.

[erf]

Would this be relevant for the vis editor? @martanne

[mawww]

Hello,

For Kakoune, I believe an easy approach for that would be an attribute, similar to underline/bold/itallic... This attribute would just mean "highlight each character as-if they had the cursor on top of them". This would stay separate from the "real" hardware cursor, which would remain at the hardware cursor position and would be displayed according to the current cursor display mode. The cursor attributes would not respect the current cursor display mode, as it could make sense to hide the real cursor while still display various characters with the cursor attribute, we might even want multiple cursor style attributes (block/vertical line), combined with existing blinking attribute.

The tricky part for Kakoune is that we use ncurses for output, which unfortunately does not give us a way to use arbitrary attributes, so before being able to support this, we would need to rewrite the terminal UI code to replace ncurses (at least for the display parts).

Those are my quick thought on that, but to be clear, I dont consider this a priority in Kakoune at the moment, and it depends on replacing (parts of) ncurses, which is a significant amount of work.

[kovidgoyal]

That has a few concerns:

1) What happens when you erase the character at a location with a cursor attribute set? Does the cursor remain? Does it get deleted? Similarly what happens when you write a new character into that location? What happens when you insert a character somewhere before in the line? Does the cursor move?

2) How does it work? Character attributes in terminals typically work by setting the attributes on the cursor, and those attributes are copied into the cell when you output new characters. So to render your cursor you would need to move the hardware cursor to each location and output the character at that location again.

3) From an implementation perspective, it is not ideal, because while it works well for block cursors which are basically rendered by changing the fg/bg colors during normal character rendering, for other types of cursors, it would mean an whole extra scan of every cell to check if it contains a special cursor, in every rendering pass.

I think a better design is to have a dedicated escape code to create/style/position/delete extra cursors. This has the additional advantage that you dont need to change our ncurses based code at all. Simply write the extra escape codes to stdout in between calls to ncurses' API.

Offtopic and a bit ranty: ncurses is a horrible library, I strongly urge you to get rid of it anyway for your own future sanity.

[mawww]

Regarding your concerns, I was thinking just do exactly as the terminal does for any other attributes, my idea was that displaying characters as-if a cursor was there would work exactly like displaying say an underline attribute on a single character.

In Kakoune display code, we only know of the cursor position in order to tell the terminal at the end of redraw where to place the hardware cursor so that it can use that as a hint for its eventual widgets (such as displaying an emoji picker, or placing the input method window OnTheSpot...). The rendering code itself just paints the full window with ncurses, and cursors have no special semantics, they are drawn exactly as the rest (Kakoune display structure is basically a list of display lines, themselves being a list of display atoms themselves being a tuple of a string, fg/bg colors and attributes).

For the implementation side, I am not sure how kitty paints its cursor, but couldn't it reuse the cursor painting logic and apply it if the cursor attribute is on when it paints a cell ? No need for a second pass.

[kovidgoyal]

The problem is that different attributes are treated differently in various situations. For example, reverse video has no effect, background may or may not change depending on the bce terminfo property, underline may or may not take effect depending on the character drawn, etc. So the semantics of this "cursor attribute" would need to be defined carefully keeping in mind interactions with all other terminal escape codes.

As for implementation, kitty does not draw individual cells, it is GPU based, which means it draws all cells at once, in parallel on the GPU. So while this is fine for block cursors, which as I said are drawn as normal cells, just with colors changed, for non block cursor there is a separate draw call needed. This separate draw call currently draws only a single cell where the hardware cursor is. With your proposal it would need to draw all cells, since it cannot know which cells have special cursors in them. And worse, it would have to do that all the time, even if no special cursors are being used, which will be the case the vast majority of the time.

With my proposal, just as you tell the terminal where to put the hardware cursor after drawing the lines, you tell it where to put the extra cursors. This is low overhead for you (you dont have to maintain an extra attribute per character which is used only rarely) and low overhead for the terminal, for the same reason. Of course, it does mean that it works differently from how you do special cursors now, which I guess, is why you dont like it.

[kovidgoyal]

And I would like to add, semantically, a cursor is not a text attribute. A cursor is instead an element of the UI that is independent of the text under it. Treating it as a text attribute seems wrong to me.

[mawww]

I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ?

Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...)

Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though.

And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is).

All that said, I do not care that much, as I already said this is not a priority for Kakoune at the moment.

Cheers.

[kovidgoyal]

On Mon, Jul 16, 2018 at 11:22:00PM -0700, Maxime Coste wrote:

I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ?

Underlines are drawn as special characters. I suppose I could draw cursors as special characters as well, but that comes with a new set of overheads, essentially on every render call a byte of extra data would need to be sent to the GPU, and the GPU would need to perform several extra calculations per pixel on every render. The point is not so much that this cannot be done, but that it adds extra overhead in the case of no special cursors, which is by far the common case. I would like a solution that means the CPU/GPU are doing no significant extra work when there are no extra cursors.

Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...)

Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.

Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though.

See the discussion of background color erase in the ncurses FAQ for some of the complexity of this. http://invisible-island.net/ncurses/ncurses.faq.html#bce_mismatches

And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is).

That's true, however, I dont see what it has to do with the mechanism of transmission of the location of the extra cursors. Using either my proposal or yours the terminal still knows where the extra cursors are and still has to draw them. The only difference is, in one case that data is tied to text state which is semantically orthogonal to UI state and cursors are UI not text.

[mawww]

I would like a solution that means the CPU/GPU are doing no significant extra work when there are no extra cursors.

Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).

Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.

Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.

Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursors by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them").

[kovidgoyal]

On Tue, Jul 17, 2018 at 12:15:53AM -0700, Maxime Coste wrote:

I would like a solution that means the CPU/GPU are doing no significant extra work when there are no extra cursors.

Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).

The thing about GPUs is that they dont like branches (it breaks parallelism). So underlines are implemented as a blend of colors with an alpha which is set to zero when no underline is present. In order to do cursors that way, would imply an extra blend operation per pixel (EDIT: and an extra texture sample per pixel)

Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.

Exactly the same thing would happen if the application crashes while using a cursor text attribute. The screen would still have lots of extra blinking cursors. In general, terminals are not robust against application crashes, you pretty much always have to reset the terminal after a crash.

Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.

Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursor by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them").

No, extra cursors are entirely unaffected by text operations in any way. Repainting a cell, outputting a character, erasing the screen, erasing lines, words, characters, etc, all have no effect on the cursor. The one exception is, perhaps, scrolling the screen, though I can see arguments for either way in that case.

[egmontkob]

Hi guys,

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

On one hand this sounds like a bit of an overkill to me, on the other hand I can also imagine other useful situations. E.g. even a relatively simple text editor such as "joe" can show multiple viewports to the same file at once, it would be nice if the cursor was shown in all of them (or would it be distracting, since the user couldn't be sure which one was the main one, i.e. the one which will start scrolling to keep the cursor inside the viewport, and where special operation (such as Find) will work?). Tmux-like apps could have synchronized (multiplexed) input sent to all panes, and show the cursor in all.

Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature.

I don't really care about ncurses either, and I don't care about hacks where an app uses a mixture of ncurses and off-ncurses output. Ncurses already has quite a few limitations that make it unsuitable for most "big" projects. E.g. due to its palette approach, it's impossible to display arbitrary (not predefined) colors as they arise (e.g. "mcview" cannot display ANSI colors like "less -R" does, "watch" cannot support 256 colors etc.), and then we haven't even talked about true colors. Then there's the colored and wavy underline, the explicit hyperlinks (VTE and iTerm2; not yet in Kitty), and presumably more. Any application that wishes to use multiple cursors probably sooner or later will also wish to use some of these as well, so will have to switch to its own screen drawing (or some other library) anyway.

I think you guys have already agreed that there would still be only one "main" cursor (where new incoming data is printed etc.), the rest are "extra" cursors. I concur. Usually the user probably wouldn't be able to tell which is the main one, but there might be exceptions. E.g. when you press Ctrl+Shift+U to enter a Unicode character, Kitty brings up its own stuff at the top of the window, so this would be unaffected; however, VTE presents an inline box at the cursor, I think it would still go to the "main" one. Other emulators have visual aids helping you locate the cursor (all the time, or upon a keypress), they might decide to highlight the main one only, not all of them. In VTE we experimented with hiding the cursor on focus out as pretty much every graphical app does; even though we dropped this idea, it might make sense (I'm not sure, we'd need to try it out) to hide the "extra" cursors on focus out.

With this hidden concept of "main" vs. "extra" cursors, the extra cursors carry some semantics in the applications (text editors) towards the users, but they don't have any semantics for the terminal emulator. For the terminal emulator it's just a visual attribute.

As for the overall design (just another attribute, or a totally different kind of escape sequence updating and maintaining the extra cursors' location) I'm firmly in favor of the "just another attribute", that is, mawww's approach.

[kovidgoyal] 1. What happens when you erase the character at a location with a cursor attribute set?

With this approach, this question is already answered. The same should happen that to any other attribute, there's absolutely no point in treating this attribute differently. The background color wrt. bce is a bit special so it deserves a sentence: I don't see any point in applying the new "make it look like as if it was a cursor" attribute to the rest of the line in case of a bce, so still the background color should be the only special one here, the new attribute should behave just like the existing ones (e.g. underline, strikethrough) do and not get applied here.

[kovidgoyal] 2. How does it work? [...] So to render your cursor you would need to move the hardware cursor to each location and output the character at that location again.

That's exactly what happens now with any attribute that an app needs to modify, so I don't see any problem here.

[kovidgoyal] The problem is that different attributes are treated differently in various situations. For example, reverse video has no effect [...], underline may or may not take effect depending on the character drawn, etc. [...]

In VTE there's no special handling of reverse or underline anywhere. Why is there in Kitty? Do we miss anything? I believe all our non-background attributes have the very same handling.

[kovidgoyal] With my proposal, just as you tell the terminal where to put the hardware cursor after drawing the lines, you tell it where to put the extra cursors. This is low overhead for you (you dont have to maintain an extra attribute per character which is used only rarely) and low overhead for the terminal, for the same reason. Of course, it does mean that it works differently from how you do special cursors now, which I guess, is why you dont like it.

It seems to me that for you, "low overhead" means saving a bit, or some CPU cycles. For me, "low overhead" primarly means cleaner code, saving developer time, and hence reusing existing pieces wherever appropriate rather than coming up with something brand new.

[kovidgoyal] The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

So we'd need two brand new escape sequences: Placing an extra cursor at a specific position, and deleting. We'd even need a "delete all", for when the application repaints its screen, e.g. to repair a potentially damaged one. Then maybe we don't even need a "delete one". Plus, figure out how these positions are affected by text being output or deleted, by scrolling (and scroll regions), by clear-to-eol, by window resize and so on and so forth. Sounds magnitudes more complex and error-prone (e.g. there's not even a "standard" terminal emulator to follow; what if two emulators implement some corner case differently and it causes troubles later on?) to me than assigning let's say ANSI 66 and 67 to enable/disable "extra cursor" mode and just doing everything the same way we do for other attributes.

A few more points:

What would happen on the normal screen? Would extra cursor be supported at all? Could it scroll out to the scrollback buffer? What would happen to it on rewrap-on-resize?

Repainting the screen after a text editor operation (e.g. page down) can take a few kilobytes of traffic, which might get stuck over ssh. With off-channel communication for the cursor positions, this might easily result in intermittent visual glitches, the extra cursor being shown at the wrong place. With in-channel communication I believe the visual experience would also be better.

[mawww] For Kakoune, I believe an easy approach for that would be an attribute, similar to underline/bold/itallic...

I think it's not just for Kakoune, but for most apps out there. If they have an existing infrastructure of tracking where underline/bold/italic are present, it's way easier to just add yet another attribute to this infrastructure, rather than coming up with a new one for the sake of the cursors. It feels to me that this approach is not only cleaner, but is also easier to implement (replacing ncurses put aside) in most applications, and most terminal emulators (sorry to hear if Kitty is not one of these).

[maximbaz]

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

My bad, I simply didn't know where to open such discussion, and since I'm using kitty and I saw it already implement a few protocol extensions, I decided to propose here yet another one.

Of course this is intended to be adopted by other terminals as well, so your input is very valuable 👍

[egmontkob]

I simply didn't know where to open such discussion, and since I'm using kitty and I saw it already implement a few protocol extensions, I decided to propose here yet another one.

Fair enough, actually there's no right place (other than cross-posting to a few). Kitty is probably the "bravest" emulator now, experimenting with new features, so it's a great place to start. I've notified VTE's main developer, as well as iTerm2's developer. Thanks for posting this idea!

[kovidgoyal]

On Tue, Jul 17, 2018 at 04:59:07AM -0700, egmontkob wrote:

Hi guys,

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

You are welcome to contribute to the discussion.

Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature.

It's fine that it does not matter to you, it does however matter to me. So let's not just dismiss it out of hand. Not everyone is happy with just waving their hands and saying its a few CPU cycles, who cares. Those few CPU cycles you so casually dismiss for every feature (I've heard this argument form you before) add up over time. For proof of that, just compare the CPU usage of kitty+X when scrolling a file in less: https://sw.kovidgoyal.net/kitty/performance.html

Now to the rest. Rather than responding point-by-point, let me summarize how the two proposals will work in practice, from the perspective of the bytes to send to the terminal.

Let's take the use case illustrated by @maww namely having the data stored in lines which contain runs of formatted text. Suppose we want an extra cursor at column 2 in the line ABCD

1) Using a text attribute Bytstream

A B CD 2) Using an escape code A BCD So, in this case my proposal is actually simpler to implement for an application developer. Now suppose that the editor wants to exit multiple cursor mode: 1) using a text attribute Two possible approaches: a) Redraw entire screen b) manually move the cursor to every extra cursor location and output the character there again without the cursor text attribute. 2) Using a separate escape code Simply output the escape code to delete all extra cursors Again, my proposal is much simpler. Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen 2) Using separate escape code Hide all cursors and then either redraw entire screen or just redraw the cursors In this case it is pretty much a wash. I think this conclusively demonstrates that my proposal is actually *easier* to implement than using text attributes for application developers (putting aside issues with ncurses, which I agree with @egmontkob are irrelevant). Now to address a few points raised by @egmontkob in detail: P1) there needs to be only a single escape code in my proposal. It can take parameters to decide what effect it has, i.e. creating a new cursor, deleting a cursor, deleting all cursors, whatever. P2) extra cursors in my proposal have no interactions with erasing/deleting/printing characters lines etc. So there are no complications there that emulator developers can get wrong. P3) Regarding normal screen and scrollback In my proposal you can have a separate set of cursor for the normal and alternate screens and they obviously dont move into the scrollback buffer, since they are not text attributes. If you do want to make them text attributes, you would need to waste even more CPU cycles sanitizing lines before copying them into the scrollback. And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are **not text attributes**. They are part of the UI. In no other display system is cursor information carried by text attributes. Nowhere, ever. The very idea of putting cursor information into text is a horrible hack and requires extra-ordinary justification.

[kovidgoyal]

And let me add, that if there is some use case I missed in my bytestream demonstration above, feel free to point it out, and I will attempt to show how it is easier to do using dedicated escape codes rather than overloading text attributes.

And just to reinforce the point, @mawww my proposal allows you to continue storing extra cursor information as text attributes in you application, if that is what you want to do. The terminal emulator does not care what internal data strucutre you use.

This discussion should be about the protocol between the application and the terminal emulator and the interactions of the extra cursors with other terminal commands such as erase, scroll, print characters, etc.

[egmontkob]

Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:

1) Using a text attribute [...] So, in this case my proposal is actually simpler to implement for an application developer.

No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.

[...] Again, my proposal is much simpler.

Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.

Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen [...]

For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.

---

And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are not text attributes.

I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach.

Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar between characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless).

While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it.

So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:

• Leave a "trail", place an extra cursor where the real cursor currently is.

• Place an extra cursor at an arbitrary (x,y) position.

• Would printing data over an extra cursor (or if thinking in I-beam shape: after the extra cursor) leave it there, or would it wipe it out?

• Remove the extra cursor from the real cursor's current location.

• Remove the extra cursor from an arbitrary (x,y) position.

• Remove all extra cursors.

• How would an extra cursor be affected if the content underneath moves in any of the four directions? (Vertical scroll, or a previous character in the line inserted/deleted.) Would the cursor follow them, or stay at the same absolute place, or get wiped out?

• Would the (x,y) positions affected (and bound) by the scrolling region?

• What to do on resize? Is it okay to wipe them out all (as e.g. the scroll region is reset), expecting the app to do a full repaint anyway?

• Nit: How would a cursor positioned to the middle of a wide character be handled?

• Nit: With I-beam shape in mind, would it be possible to place a cursor after the last column?

• Nit: Would all extra cursors automatically share the main one's properties (shape, color, blinking)? Or could they each have their own ones?

• Anything that I'm missing or overcomplicating? :)

[kovidgoyal]

On Tue, Jul 17, 2018 at 06:55:10AM -0700, egmontkob wrote:

Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:

1) Using a text attribute [...] So, in this case my proposal is actually simpler to implement for an application developer.

No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.

Already implemented by what? Nothing I know of. To take the concrete example in this thread of kakoune, it uses ncurses, so to implement this it would either need to ditch ncurses or get ncurses to implement it. In either case whoever does the implementation, it is easier to output fewer escape codes than more escape codes.

As I mentioned to @mawww in my follow up post, he can keep using extra cursors as text attributes in his internal implementation if he likes to do that, the only difference between the two approaches is in one case you output more new escape codes than in the other. So when he is translating his internal text attributes into escape codes for the terminal he would do less work with a dedicated escape code.

The point is that his total draw cycle appears to be to redraw the entire line on changes. In this use case, as I demonstrated, using a dedicated escape code is better or at least no worse than using a new text attribute.

BTW @mawww apologies if you are female, please read the above he as she.

[...] Again, my proposal is much simpler.

Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.

See above.

Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen [...]

For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.

That is exactly the same as redrawing only the affected cursors with the escape codes and is why I concluded this use case is wash. Apologies for neglecting to spell it out.

---

And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are not text attributes.

I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach.

Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar between characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless).

While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it.

Sure take your time, questions like this deserve due consideration.

So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:

• Leave a "trail", place an extra cursor where the real cursor currently is.

Not sure what you mean by this? The real cursor will remain where it is. Are you saying, you would like to distinguish it from other cursors? If so isn't the point of multiple cursors that they all behave identically, so that if you insert text, it is inserted at all cursor positions?

• Place an extra cursor at an arbitrary (x,y) position.

Easy to do just have the escape code take an extra optional parameter or require the application to move the real cursor to place the extra cursor.

• Would printing data over an extra cursor (of if thinking in I-beam shape: after the extra cursor) leave it there, or would it wipe it out?

I strongly vote for leave it there, otherwise every time your print a character you have to check if there is an extra cursor there. And then if you print characters in "insert mode" you have to check for extra characters all down the line and potentially cascading all down the screen thanks to line wrap.

• Remove the extra cursor from the real cursor's current location.

Not sure what you mean here? If there is both a real cursor and an extra cursor at some location the terminal will draw only the real cursor.

• Remove the extra cursor from an arbitrary (x,y) position.

Again no problem, simply add a parameter to the escape code. By the way for how I think this escape code should be designed see the graphics escape code that kitty uses, it also takes lots of parameters. https://sw.kovidgoyal.net/kitty/graphics-protocol.html

• Remove all extra cursors.

Same, parameter to escape code.

• How would an extra cursor be affected if the content underneath moves in any of the four directions? (Vertical scroll, or a previous character in the line inserted/deleted.) Would the cursor follow them, or stay at the same absolute place, or get wiped out?

IMO the extra cursor should not be affected at all. This is the simplest way and the one least likely to be implemented incorrectly. The one exception is perhaps vertical scroll, but I dont feel strongly about it either way.

• Would the (x,y) positions affected (and bound) by the scrolling region?

IIRC (and I could be wrong) scrolling regions only affect cursor movement commands. There are no movement commands as such for extra cursors, so the question does not arise. (Instead to move a cursor, you remove and re-create it).

• What to do on resize? Is it okay to wipe them out all (as e.g. the scroll region is reset), expecting the app to do a full repaint anyway?

I am OK with either wiping them all or just bounding them in the new size as you do with the real cursor.

• Nit: How would a cursor positioned to the middle of a wide character be handled?

The same way the real cursor is handled in this case, i.e. it would stretch over two cells if block or replace and one cell if beam (it is an error to place a cursor in the middle of a wide character).

• Nit: With I-beam shape in mind, would it be possible to place a cursor after the last column?

I dont see the point, I think this should be the same as how the real cursor is handled.

• Nit: Would all extra cursors automatically share the main one's properties (shape, color, blinking)? Or could they each have their own ones?

I am in favor of them all being the same. That is the use case needed for editors and allowing them to be different is a fair bit of overhead for no actual use case.

• Anything that I'm missing or overcomplicating? :)

There's the question of clear screen and reset. However, let's leave the fine details to the second stage after we have consensus on the basic question of whether to implement this or not and if so, whether to use text attributes or extra escape codes.

See my next post.

[kovidgoyal]

So for the moment I propose we focus on Stage 1, which is answers to the following questions:

1) Editor developers: Is this something you care enough that you are likely to implement this in the reasonably near future if some terminal emulators add support for it?

2) Terminal emulator developers: Is this something you are willing to implement in principle? I say yes for kitty.

3) Everybody: Do we want to use a new text attribute or a new escape code to implement

[justinmk]

Though multicursor support is in the works for Nvim, I don't yet see a need for special terminal features.

kakoune has to hide the terminal cursor completely and "draw" its own cursors by manipulating background color of a cell. This is ugly hack that prevents people from using the much superior native cursor with all of its features (different shape, blinking, proper color).

The way I envision multicursor is a "primary" cursor plus other "marker" (secondary) cursors. The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may increase confusion.

Actually, I'd be more interested in new text attributes as @mawww hinted. E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

[gnachman]

iTerm2 author here: I'm fine with adding this, but I don't want it to be an attribute. I only have a few bits of attributes left before my struct needs to grow. That will increase memory usage and decrease cache efficiency for an attribute that would almost never be set. That's an implementation detail, but it's an important one.

More generally, it's easy to draw an analogy between the "real" cursor and the list of "virtual" cursors. It's a thing that has a coordinate that gets drawn on top of the text and there are some rules for how it's drawn that are configurable both by the user's preferences (e.g., color, shape) and by escape sequences (shape).

Provided we go with a list of cursors, I would be happy to implement it. I think we'd need escape codes to:

1. Set the number of virtual cursors
2. Position virtual cursor i at x,y
3. Set the style (box, line, bar) of virtual cursor i

I would also propose that virtual cursors change position only when told to and have no automatic behavior. This pushes the complexity into the app that wants them, but makes it much more predictable in the face of scrolling regions and double-width characters. However, I'm not quite sure how this would work with an ncurses-based app. Someone who has more experience can perhaps comment here.

[maximbaz]

The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may increase confusion.

I'm not so sure, I would say it is quite rare when I actually need to know which of the cursors is the primary one. Look in the gif in the first post, I'm manipulating a few lines and I want the same manipulation to happen in all lines, regardless of where is the primary cursor position. I would much prefer to see all cursors turn into identical vertical bar when I'm about to prepend the word "private" on every line, because this would indicate to me that I'm about to insert some text on all of these lines.

Sometimes it is helpful to know where the primary selection is, not the cursor (at least in kakoune), because there are some extra actions you can do with the primary selection (search for the word "main" here), but no actions to do with the primary cursor — and the primary selection is highlighted by using the white text color.

However the above is based on my experience with kakoune, I realize that the situation might be different in other editors.

[egmontkob]

[justinmk] E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

SGR 51, 60, 62 might be relevant here.

[justinmk] The secondary cursors don't need to look identical to the primary cursor, they are just visual clues.

[maximbaz] I'm not so sure, I would say it is quite rare when I actually need to know which of the cursors is the primary one.

I can easily think of use cases for either approaches. Maybe the best is if we consider supporting both kinds straight from the beginning.

[gnachman] I would also propose that virtual cursors change position only when told to and have no automatic behavior.

I guess I'd rather they scroll along with the content. Otherwise e.g. cat'ing a binary file may result in visual glitches (additional cursors) that don't even scroll out as you tap Enter.

[egmontkob]

[kovidgoyal] Already implemented by what? [...] In either case whoever does the implementation, it is easier to output fewer escape codes than more escape codes.

You still don't understand what I'm speaking about. This will probably be my last attempt in explaining.

Let's assume a non-ncurses based editor (there are plenty of them). Presumably it already handles bold, italic, underline etc. By some internal means it figures out which cells it wants to render with such attributes, and then implements telling this to the terminal emulator. Given this current state, it's much easier to add another attribute that follows this pattern, than an attribute that uses a totally different approach. Easiness doesn't mean fewer bytes to be emitted, easiness doesn't mean something that would be easier to implement from scratch, easiness means easier to implement and maintain given the current codebase, that is, less deviations and less additional code to introduce, reusing whatever's already available (i.e. the way bold/italic/underline are handled by this app).

[gnachman]

I guess I'd rather they scroll along with the content. Otherwise e.g. cat'ing a binary file may result in visual glitches (additional cursors) that don't even scroll out as you tap Enter.

To be sure they should not be allowed to leave the mutable section. I don't think that additional cursors make sense outside an interactive application (i.e., one that would use the alternate screen buffer), so their behavior in the face of cat is not that interesting—no matter what you do it's hopelessly broken. What I want to avoid is having to define their behavior within and without horizontal and vertical scrolling regions. It's a big mess that's hard to test, and we'd likely end up with diverging interpretations of the spec. Interactive apps would rarely scroll the whole screen anyway.

I'm not persuaded that attribute-vs-not-an-attribute makes a meaningful difference to applications, except for backward compatibility with terminals that don't support multiple cursors. The cost to adding an attribute to terminal emulators is significant (assuming others have a similar design to the two I'm familiar with, iTerm2 and tmux).

[egmontkob]

[gnachman] To be sure they should not be allowed to leave the mutable section.

That's what I had in my mind, too. I guess we agree here.

Indeed my wording wasn't specific enough. I meant I'd prefer these additional cursors to scroll, so that they disappear when they reach the top of the mutable section; rather than stay at the same absolute position, cluttering my work, until a more brutal reset-like operation is issued.

[egmontkob]

[kovidgoyal] Are you saying, you would like to distinguish [the real cursor] from other cursors? If so isn't the point of multiple cursors that they all behave identically, so that if you insert text, it is inserted at all cursor positions?

No way!

I mean, that's the behavior that apps are likely to implement, by emitting the proper output that makes the new text appear at all such position. But the terminal emulator shouldn't automatically do it.

I must have misunderstood you when I said:

I think you guys have already agreed that there would still be only one "main" cursor (where new incoming data is printed etc.), the rest are "extra" cursors. I concur.

There's no way multiple equivalent cursors, all processing the incoming data, could work reasonably. There has to be exactly 1 main cursor (as it is now), and extra ones that are display-only, without any cursor functionality.

Processing input at all cursors would lead to at least the following bunch of problems:

• In probably most cases, it's not the very same action (and the only action) that needs to be performed by applications when multiple cursors are present and the user hits a key. E.g. take a look at the opening request's video, when the word "private" is being typed. At each character of this word, all three cursor lines can probably be updated using the same sequence (e.g. ICH followed by the new letter), however, notice that there's also a bottom line showing some data that also changes. In order to update the bottom line, you'd need to remove two cursors so that you're left with only one, then update at the bottom, then recreate the two additional ones. Anything similar, e.g. a coloring change due to syntax highlight in only some of the lines being modified, or if your app is tmux then input arriving in the other pane would require the same steps. Also what if the app's drawing routines are not clever enough to use ICH but rather print out the whole lines again, then also the cursors would need to be unified and then split again.

• Even if the very same operation at each cursor would be sufficient (e.g. ICH followed by the new letter, if the example video didn't have a bottom status line), how would the applilcation know to rely on this feature of the terminal? A change to the internal buffer (the text file being edited) probably doesn't directly result in escape sequences being emitted. Rather, most likely it causes changes to its in-memory representation of what should be on the screen (including updates to the syntax highlighting and such), and then a subsequent step compares this buffer with what's on the screen and does the update. Even with a single cursor, finding optimal steps to update the screen is a hard problem. This is where ncurses optimizes heavily, while other implementations are typically more lazy, and simply update everything that's changed without too much optimization. With multiple cursors, an already hard problem becomes even harder. How would the screen drawing code of an app figure out that it can reach the desired result by applying some clever steps at all the cursors simultaneously? Needs way too much engineering work that probably hardly any (if at all) application developer is willing to put in, plus the runtime cost is high too.

• In what order would the cursors handle incoming data? It's not hard to come up with examples where the order does matter (e.g. ICH at two cursors within the same line, if the cursor stays at the same physical position). Would it be the display order of cursors (and then isn't it an expensive operation to keep the cursors sorted all the time)? Or would it be the creation order? Or some random unspecified, resulting in undeterministic behavior?

• Would it be allowed to have two or more cursors at the same offset? If so then e.g. an ICH would insert two or more spaces there, correct? If not then at every step the cursors need to be deduped, again a potentially expensive operation.

• What if there's a cursor at the bottom right, and one in the top row? New input at the bottom right causes a line wrap and hence the contents scroll, but then the cursor at the top will be scrolled out. Is it lost then, or what?

• How to clean up additional cursors so that only one remains? I mean, which one would stay?

• Compatibility: If an application incorrectly believes that the terminal emulator supports multiple cursors, while in fact it doesn't, ideally the worst that should happen is that these escape sequences are silently ignored and hence the additional cursors don't show up. Instead, the entire screen could fall apart big time, since things that should be updated at multiple places will be updated at only one.

• Protection against cat'ing a random binary file with random escape sequences: I'm okay with seeing garbage as a result, but I heck don't want to end up with multiple "active" shell prompts, typing into all of them at once.

[kovidgoyal]

On Tue, Jul 17, 2018 at 01:52:13PM -0700, egmontkob wrote:

[kovidgoyal] Already implemented by what? [...] In either case whoever does the implementation, it is easier to output fewer escape codes than more escape codes.

You still don't understand what I'm speaking about. This will probably be my last attempt in explaining.

Let's assume a non-ncurses based editor (there are plenty of them). Presumably it already handles bold, italic, underline etc. By some internal means it figures out which cells it wants to render with such attributes, and then implements telling this to the terminal emulator. Given this current state, it's much easier to add another attribute that follows this pattern, than an attribute that uses a totally different approach. Easiness doesn't mean fewer bytes to be emitted, easiness doesn't mean something that would be easier to implement from scratch, easiness means easier to implement and maintain given the current codebase, that is, less deviations and less additional code to introduce, reusing whatever's already available (i.e. the way bold/italic/underline are handled by this app).

And you still dont understand what I am saying. The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator. In my proposal, those escape codes are simpler, more flexible, semantically correct and therefore easier to use.

[kovidgoyal]

On Tue, Jul 17, 2018 at 02:47:10PM -0700, egmontkob wrote:

[kovidgoyal] Are you saying, you would like to distinguish [the real cursor] from other cursors? If so isn't the point of multiple cursors that they all behave identically, so that if you insert text, it is inserted at all cursor positions?

No way!

I mean, that's the behavior that apps are likely to implement, by emitting the proper output that makes the new text appear at all such position. But the terminal emulator shouldn't automatically do it.

Dont worry, I'm not suggesting the terminal emulator do it. I'm just saying that is the way multiple cursors are usually used so in the absence of any other use case I dont see a compelling need for distinguishing one cursor from the others.

[kovidgoyal]

On Tue, Jul 17, 2018 at 09:06:20AM -0700, George Nachman wrote:

iTerm2 author here: I'm fine with adding this, but I don't want it to be an attribute. I only have a few bits of attributes left before my struct needs to grow. That will increase memory usage and decrease cache efficiency for an attribute that would almost never be set. That's an implementation detail, but it's an important one.

Good to hear. I agree with you that attributes are a bad implementation strategy performance wide. kitty currently does have enough space in its struct, but it would add overhead for other reasons, I have detailed above.

1. Set the number of virtual cursors
2. Position virtual cursor i at x,y
3. Set the style (box, line, bar) of virtual cursor i

We'd also need escape codes to delete cursors either specific ones or all of them. But, lets hold off on designing the actual protocol for a bit longer until we have buy in from editor developers. I dont want to spend time on designing/implementing this only to have it not be used by anybody.

I would also propose that virtual cursors change position only when told to and have no automatic behavior. This pushes the complexity into the app that wants them, but makes it much more predictable in the face of scrolling regions and double-width characters.

Agreed.

However, I'm not quite sure how this would work with an ncurses-based app. Someone who has more experience can perhaps comment here.

It wouldn't. But neither would any other approach. ncurses needs new API for this either way.

[kovidgoyal]

On Tue, Jul 17, 2018 at 09:06:07AM -0700, Justin M. Keyes wrote:

The way I envision multicursor is a "primary" cursor plus other "marker" (secondary) cursors. The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may increase confusion.

Are you saying that if terminals were to implement it, you would not be interested in using it -- i.e. you think whatever multi-cursor features you intend to implement in nvim would be better off not using terminal rendered cursors?

[mawww]

As an additional data point, its likely we would need to display different cursors with different "style", at least in the case where there are multiple cursors on the buffer, and we are in prompt mode (having the buffer cursor change shape due to the interactions with the prompt would be pretty strange).

And you still dont understand what I am saying. The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator. In my proposal, those escape codes are simpler, more flexible, semantically correct and therefore easier to use.

I still think an attribute makes the most sense, and is the semantically correct thing, it seems established that the only cursor that has semantic meaning in the terminal is the main one, the others do not have any semantic meaning for the terminal, they are literally a display attribute of the character they lie on. I believe most full screen terminal applications will find this model much simpler than having to clear and replace all cursors on each redraw (or track where they are to move them around).

Representing the cursor as text attributes inside the application only to extract those attributes to generate a list of cursors is not really practical, in Kakoune it will be far simpler to pass the cursors position as a separate list to the display backend if that was the way the terminal wanted it.

In my view, the only argument in favor of non-attribute cursors is that it is easier/cleaner for terminals to implement (which is a good reason, dont get me wrong).

[kovidgoyal]

@gnachman Just FYI, there are escape sequences to set both the cursor color and the shape, not just the shape. See http://invisible-island.net/xterm/ctlseqs/ctlseqs.html (search for cursor color) There are fairly widely implemented (IIRC kitty, VTE based terminals, xterm, and konosle, though as usual konsole will be using escape codes different from everybody else).

[kovidgoyal]

Let me summarize where I think we are, please correct me if I misrepresent any of your positions. Regarding the three questions of stage 1:

1) Editor developers: Are you willing to write code to use this feature if terminals implement it. kakoune says maybe in the distant future. nvim at least for the moment seems to say no.

2) Terminal developers: Are you willing to implement this feature. kitty and iterm2 say yes. VTE hasn't said yes or no yet.

3) Text attribute or new escape code: kitty - new escape code iterm2 - new escape code VTE - probably new escape code kakoune - relunctantly OK with new escape code? nvim - no opinion?

[egmontkob]

The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator.

Which, if the exitsing pattern is followed, might be as easy as adding a new entry into an array (let's say an array mapping positions of a bitmap into ANSI codes), or as easy as copy-pasting an existing few lines and just altering the internal attribute ID and the ANSI numbers. Whereas, with a different approach, brand new code has to be written (even it's just a few lines) and thoroughly tested.

In my proposal, those escape codes are simpler, more flexible, semantically correct and therefore easier to use.

Simpler, easier to use: no. Semantically correct: yes; let's remember that all this debate occurred assuming we take the "just another attribute" approach, which no longer seems to be the case, and the new approach makes this debate pointless.

Dont worry, I'm not suggesting the terminal emulator do it. [...]

In that case it's unclear to me why you had all those question to the list of suggested actions I posted. E.g.

• Leave a "trail", place an extra cursor where the real cursor currently is.

We may or may not need such an escape sequence, but it's unclear to me what is unclear in this proposal :) There's always exactly 1 "real cursor"; do we need an escape sequence that sets an "extra cursor" at the very place where the "real cursor" is right now? That is, if the real cursor is subsequently moved away, that position still visually shows up as a cursor.

• Remove the extra cursor from the real cursor's current location.

I meant to make that position no longer an extra cursor position, that is, if the real cursor is moved away, it no longer shows up as a cursor.

[egmontkob]

I'm just saying that is the way multiple cursors are usually used so in the absence of any other use case I dont see a compelling need for distinguishing one cursor from the others.

I do, and have vaguely mentioned one, so let's look at a concrete example in depths.

Take the joe text editor as example, open a writable file with joe filename. Split the view with Ctrl+K O. Now you have two viewports showing the same file. If you edit one, the changes are reflected in the other one as well. You may jump to the other viewport at any time using Ctrl+K N.

Since characters typed are being inserted in both viewports, it makes sense to have an additional cursor shown in the other viewport as well (assuming the viewport does contain the cursor position, of course), using our brand new feature, correct?

Now let's imagine we've implemented this, we use joe with this new feature, and stand up for a coffee break and then return. There's no visual distinction telling which of the two viewports is the "active" one, and we forgot it during the break. Although, this is an important piece of information for using the software.

Just move the cursor Down, expecting to walk downwards in the file. In one of the viewports, the file contents will scroll to keep the cursor visible. In the other one, the cursor will disappear from the viewport. You want to know it unconsciously in advance, rather than being a surprise when the first viewport's cursor reaches the bottom.

Operations like Find (Ctrl+K F) work in the "active" viewport. (Cancel with Ctrl+C.) Closing the window (Ctrl+C) works on the "active" viewport etc.

If this feature was implemented in terminal emulators and in turn the joe text editor, I'd definitely want to see "extra cursors" marked differently, somewhat weaker than the normal one.

I'd like to design this feature in a way that "extra cursors" can either look the same as the normal one, or can look weaker, according to the application's request.

[kovidgoyal]

On Wed, Jul 18, 2018 at 02:04:59AM -0700, egmontkob wrote:

The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator.

Which, if the exitsing pattern is followed, might be as easy as adding a new entry into an array (let's say an array mapping positions of a bitmap into ANSI codes), or as easy as copy-pasting an existing few lines and just altering the internal attribute ID and the ANSI numbers. Whereas, with a different approach, brand new code has to be written (even it's just a few lines) and thoroughly tested.

In either case you will need to write brand new code to convert the extra attribute into something a terminal emulator will understand. In one case that brand new code is simpler than the other and will require less testing and less validation.

Anyway, I am tired of this argument. It's pretty clear that the consensus is to not use an attribute. Further verbiage on this issue is a waste of everyone's time.

[kovidgoyal]

On Wed, Jul 18, 2018 at 09:15:52AM +0000, egmontkob wrote:

I'd like to design this feature in a way that "extra cursors" can either look the same as the normal one, or can look weaker, according to the application's request.

That's fine, if you have a use case for it, I have no objection. Although ideally, I'd like to see that use case backed up by some editor developer that actually intends to use it.

In any case it can be discussed in detail when we nail down the protcol for multiple features.

[egmontkob]

I'm fine if we just keep it in mind, and design the protocol to be easily extendible in this direction in the future.

[kovidgoyal]

Sounds good to me.

[kovidgoyal]

I had half an hour so I wrote a first draft of the protocol: https://github.com/kovidgoyal/kitty/commit/67324efae5f68f194df13b134666e394602215e6

Comments are welcome. I have deliberately done the minimum needed, but the design is highly extensible.

[kovidgoyal]

Easier to read link: https://github.com/kovidgoyal/kitty/blob/67324efae5f68f194df13b134666e394602215e6/docs/protocol-extensions.rst#multiple-cursors

[kovidgoyal]

Also ccing @leonerd as he might be interested

[maximbaz]

It is now crystal clear how a single new escape code can achieve all mentioned scenarios, and I personally like this approach. Thanks for drafting the protocol!

[egmontkob]

Thanks for working on this! Here's my first few thoughts:

I haven't met APC so far, so I cannot comment on choosing it. Read: I'm fine with it.

My biggest concern is the use of indices, at first glimpse I dislike it. IMO it imposes an additional unnecessary complexity both for terminal emulators and apps.

• If I were to write a text editor, I don't think I would want to assign IDs to the cursors, just as I don't want to assign IDs to bold or italic or green words etc. I just don't see the point in them. Creating IDs if I don't have them is an additional burden. (The other way around, i.e. not telling them to the terminal emulator even when I have them, is no problem.)

• If I really have to have IDs, and assuming that I use double buffering in my editor, i.e. I have an internal representation how I believe the screen currently looks like, and another one about how I want the screen to look like, and every once in a while I call the method that updates the terminal emulator accordingly, then the concept of IDs make me require either a separate data structure for them; or (as you told mawww he'd still be able to store these as if they were text attributes) 32 bits per cell (to be able to remove them by ID), rather than 1 bit per cell (and remove them at the positions where they are no longer present).

• IDs are also inconvenient for tmux and friends: they'd have to dynamically maintain a mapping from each pane's cursor IDs (as specified by the app running inside) to the IDs use towards the host emulator, to avoid clashes.

I think I'd rather see IDs gone, and cursors just identified by their locations.

A few more corner cases or nitpicks:

• If we stick to IDs, is it allowed to have multiple cursors (with different IDs) at the same position? If so, deleting/moving one retains the other, correct?

• Reset, Clear screen: As the draft evolves, we should become more specific on the escape sequences. E.g. soft reset (<ESC>[!p) just resets the cursor's attributes, I don't think it should influence extra cursors. What about "clear to the right and bottom from the cursor" and similar ones?

• Switching between primary and alternate screen: I'm unsure, e.g. if mc were to use such cursors, it would need to reinstall them if you press <Ctrl>+O twice. (Not sure if it repaints its alternate screen anyway, or just switches back and forth. But what if the normal screen underneath has multiple cursors for a good reason?) I'd probably say each screen should keep their cursors. If this is rejected, then at least it should be emphasized that switching "to" the currently active screen (a no-op) shouldn't wipe out the cursors, I vaguely recall mc emitting unnecessary switches.

• Vertical scrolling: Should, for whatever reason, an application that does partial screen handling (like a shell prompt, or a multiline wget progress bar, etc.) decide to have multiple cursors, they would need them to scroll along with the text, as it's completely out of their control (and their knowledge) when the terminal scrolls. Also, if cat'ing a binary file sets some additional cursors, I'd expect them not to clutter my screen until a hard reset, but move upwards, eventually out of the screen, as I simply tap on Enter. So I vote for scrolling.

• The overhead of the escape sequence is 5 bytes (<ESC>_C, and then <ESC>\), I'm wondering if there should be a shortcut for operations on multiple cursors, with smaller additional overhead than repeating these 5 bytes? (I'm fine without, just asking.)

• I don't get how r=10,c=3 becomes "row 10 and column 2", I guess it's just a typo. In terminal emulators, traditionally rows and columns are numbered from 1, and so shall it be in our new escape codes as well for consistency. If you prefer to use 0-based in the English explanation, then it should go for both axes (i.e. "0-based row 9 and column 2").

• What should "save/restore cursor" do?

[kovidgoyal]

On Thu, Jul 19, 2018 at 09:28:08AM -0700, egmontkob wrote:

Thanks for working on this! Here's my first few thoughts:

I haven't met APC so far, so I cannot comment on choosing it. Read: I'm fine with it.

It is used for kitty's graphics protocol.

My biggest concern is the use of indices, at first glimpse I dislike it. IMO it imposes an additional unnecessary complexity both for terminal emulators and apps.

• If I were to write a text editor, I don't think I would want to assign IDs to the cursors, just as I don't want to assign IDs to bold or italic or green words etc. I just don't see the point in them. Creating IDs if I don't have them is an additional burden. (The other way around, i.e. not telling them to the terminal emulator even when I have them, is no problem.)

• If I really have to have IDs, and assuming that I use double buffering in my editor, i.e. I have an internal representation how I believe the screen currently looks like, and another one about how I want the screen to look like, and every once in a while I call the method that updates the terminal emulator accordingly, then the concept of IDs make me require either a separate data structure for them; or (as you told mawww he'd still be able to store these as if they were text attributes) 32 bits per cell (to be able to remove them by ID), rather than 1 bit per cell (and remove them at the positions where they are no longer present).

If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden. The alternative is, the application has to remember the location of each extra cursor. ids are more robust. They support multiple cursors at the same location. Not to mention that knowing the location is not always easy (think of wide characters). As it is I see a lot of bug reports caused by the fact that most editors use out of date wcwidth() implementations that dont agree with kitty which uses one derived from the unicode 11 standard. This is the main reason I chose to use ids.

• IDs are also inconvenient for tmux and friends: they'd have to dynamically maintain a mapping from each pane's cursor IDs (as specified by the app running inside) to the IDs use towards the host emulator, to avoid clashes.

tmux would have to rewrite the escape codes anyway, to map locations. Rewriting ids shouldn't be a particularly burdensome additional task. Although, given my recent experiences with tmux upstream, I am pretty convinced it should die in a fire. See #413

• If we stick to IDs, is it allowed to have multiple cursors (with different IDs) at the same position? If so, deleting/moving one retains the other, correct?

Yes and yes.

• Reset, Clear screen: As the draft evolves, we should become more specific on the escape sequences. E.g. soft reset (<ESC>[!p) just resets the cursor's attributes, I don't think it should influence extra cursors. What about "clear to the right and bottom from the cursor" and similar ones?

Reset is rs1 from terminfo and clear screen is clear from terminfo. No other escape codes should have any effect.

• Switching between primary and alternate screen: I'm unsure, e.g. if mc were to use such cursors, it would need to reinstall them if you press <Ctrl>+O twice. (Not sure if it repaints its alternate screen anyway, or just switches back and forth. But what if the normal screen underneath has multiple cursors for a good reason?) I'd probably say each screen should keep their cursors. If this is rejected, then at least it should be emphasized that switching "to" the currently active screen (a no-op) shouldn't wipe out the cursors, I vaguely recall mc emitting unnecessary switches.

I'm OK with changing that, let's wait to hear if anyone else has any thoughts on it. And yes, changing to the existing screen is a no-op.

• Vertical scrolling: Should, for whatever reason, an application that does partial screen handling (like a shell prompt, or a multiline wget progress bar, etc.) decide to have multiple cursors, they would need them to scroll along with the text, as it's completely out of their control (and their knowledge) when the terminal scrolls. Also, if cat'ing a binary file sets some additional cursors, I'd expect them not to clutter my screen until a hard reset, but move upwards, eventually out of the screen, as I simply tap on Enter. So I vote for scrolling.

If you allow this then you have to worry about DECOM (i.e. margins). Since index operations are bounded by margins, you would also have to specify the behavior of extra cursors in the presence of margins.

Still it's not an absolute no-no from me. Let's see if other people want it.

• The overhead of the escape sequence is 5 bytes (<ESC>_C, and then <ESC>\), I'm wondering if there should be a shortcut for operations on multiple cursors, with smaller additional overhead than repeating these 5 bytes? (I'm fine without, just asking.)

I dont think this is critical enough to need that. Also what operation would you do in bulk? Currently the only two operations are delete (which you can do in bulk for all anyway) and move which makes not much sense to do in bulk.

In the future if other operations are added, we can revisit.

• I don't get how r=10,c=3 becomes "row 10 and column 2", I guess it's just a typo. In terminal emulators, traditionally rows and columns are numbered from 1, and so shall it be in our new escape codes as well for consistency. If you prefer to use 0-based in the English explanation, then it should go for both axes (i.e. "0-based row 9 and column 2").

Typo. As noted indexing is 1-based.

• What should "save/restore cursor" do?

Nothing, as far as I am concerned. I dont want terminal emulators to have to maintain stacks of large numbers of cursor objects.

[kovidgoyal]

I have made some modifications to the protocol specification based on the feedback. See the cursors branch of this repo for diffs.

[gnachman]

I like having one escape sequence to control everything. Makes it easy to reason about.

Are row and column numbers relative in origin mode?

APC is not ignored by Terminal.app. How does VTE treat it? I generally prefer OSC for this reason. I believe VTE now ignores OSC.

I agree that treating it as a list of cursors makes sense. I can't imagine an editor storing its cursors any other way, but perhaps I lack imagination :)

I think sequences for changing shape and color should be in the first draft. How about <ESC>_Ci=1,s=N,c=#rgb<<ESC>\\ where N corresponds to the Ps of DECSCUSR and #rgb is an SRGB value like #ff8000 or #f80. If you want to add more color spaces later, you can always stick something before the #.

[kovidgoyal]

On Thu, Jul 19, 2018 at 10:13:27PM -0700, George Nachman wrote:

Are row and column numbers relative in origin mode?

No, as noted in the spec DECOM is ignored. Do you think it should not be? An argument can be made either way.

APC is not ignored by Terminal.app. How does VTE treat it? I generally prefer OSC for this reason. I believe VTE now ignores OSC.

What does not ignored mean? What does iterm2 do if it encounters an APC it does not understand?

I think sequences for changing shape and color should be in the first draft. How about <ESC>_Ci=1,s=N,c=#rgb<<ESC>\\ where N corresponds to the Ps of DECSCUSR and #rgb is an SRGB value like #ff8000 or #f80. If you want to add more color spaces later, you can always stick something before the #.

The reason I am reluctant to add shape and color to the first draft is that it adds overhead to the implementation for a feature I have yet to see a use case for, from someone actually intending to use it, as opposed to theoretical usage scenarios.

However, I am open to being convinced otherwise.