Feature request: mouse any-event Tracking (1003)

[AutumnMeowMeow]

Reference: https://www.invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-Any-event-tracking

This would round out mouse support, permitting 1) applications to know the mouse position always, and 2) implement focus-follows-mouse for panes.

[imsnif]

I really like the "focus follows mouse" part, though I think we'll have to give some thoughts about the UX here. I'm not sure it should be on by default, at least... I guess we'll have to play around with it and see how it feels.

Another point that I'm pretty certain about: I think it would be best to only send mouse movements to the focused pane (assuming of course the movements are over said pane).

But regardless - I'd very much love to see this feature in Zellij. I guess we'll also have to place ourselves in any-event tracking mode. I wonder how wide-spread support for this is in terminals?

If you're interested in working on this @AutumnMeowMeow - I'd be happy to point you to the relevant parts of the code-base to get what you need (if you want me to, ofc).

[AutumnMeowMeow]

Another point that I'm pretty certain about: I think it would be best to only send mouse movements to the focused pane (assuming of course the movements are over said pane).

I believe the most common implementation for terminals and embedded terminal panes/widgets is:

• No button down - mouse motion events are only passed to the terminal if the mouse is over that pane.
  • Looking at a few terminals: xterm, mlterm, contour will report events even if the window is not focused; wezterm only reports if the window is focused. I think it would make sense to either follow xterm's example and always send any-event if the mouse is over a pane whether focused or not; or make it configurable with whatever default you want.
• Button down - mouse motion events are passed to the focused terminal, with the coordinates clamped to the terminal dimensions. (So no negative coordinates when click-dragging off screen.) This is the click-and-drag scenario.

I guess we'll also have to place ourselves in any-event tracking mode.

zellij can activate both button-event and any-event in that order on startup, like: "\x1b[?1002;1003h", and then zellij will get any-event if the host supports that, and button-event if the host doesn't support any-event but does support button-event. Easy graceful fallback. (And a good test for terminals that don't loop on all the DECSET parameters. :-) )

I wonder how wide-spread support for this is in terminals?

It's pretty common. All of the terminals listed here with "yes" in the "Mouse Cursor" column support Any-Event, including screen and tmux when their host console supports it. (I believe konsole does now too, and maybe DomTerm -- I haven't updated that page in some time.)

I would be interested in doing this one. My mental model so far is:

• zellij-utils/src/input/event/mouse.rs: add a Motion(Position) to MouseEvent . Follow the compiler messages to all the cases.
• zellij-client/src/os_input_output.rs: modify ENABLE_MOUSE_SUPPORT / DISABLE_MOUSE_SUPPORT to include 1003.
• zellij-server/src/tab/unit/tab_integration_tests.rs: add sgr_mouse_mode_any_event tests
• zellij-client/src/input_handler.rs add MouseEvent::Motion case to handle_mouse_event
• zellij-server-src/pane/terminal_pane.rs: add fn mouse_motion()
• zellij-server-src/pane/grid.rs: add AnyEventTracking to MouseTracking. Follow the compiler messages to all the cases.
• zellij-server-src/pane/grid.rs: add mouse_motion_signal. Looks like I also need to modify all of the click signals to send any-event appropriately too when hold is true and tracking is any-event. (It seems like Hold could be removed in favor of Motion, as a simplifying refactor. But initially I would leave it.)

I have no idea on:

• Where the actual decoding of the host terminal stream into mouse events occurs.
• If any other widgets/plugins/etc. need to be modified to handle a new mouse event.

[imsnif]

I believe the most common implementation for terminals and embedded terminal panes/widgets is:

* No button down - mouse motion events are only passed to the terminal if the mouse is over that pane.

  * Looking at a few terminals: xterm, mlterm, contour will report events even if the window is not focused; wezterm only reports if the window is focused.  I think it would make sense to either follow xterm's example and always send any-event if the mouse is over a pane whether focused or not; or make it configurable with whatever default you want.

* Button down - mouse motion events are passed to the focused terminal, with the coordinates clamped to the terminal dimensions.  (So no negative coordinates when click-dragging off screen.)  This is the click-and-drag scenario.

I just have this feeling that with multiple panes open, having the cursor change the internal cursor location of panes I happen to pass over when trying to focus on a pane beyond them might be a little annoying. I'm definitely on board with having this configurable. Let's try out both options and see what we feel works better as a default?

zellij can activate both button-event and any-event in that order on startup, like: "\x1b[?1002;1003h", and then zellij will get any-event if the host supports that, and button-event if the host doesn't support any-event but does support button-event. Easy graceful fallback. (And a good test for terminals that don't loop on all the DECSET parameters. :-) )

:D

I would be interested in doing this one. My mental model so far is:

* zellij-utils/src/input/event/mouse.rs: add a Motion(Position) to MouseEvent .  Follow the compiler messages to all the cases.

* zellij-client/src/os_input_output.rs: modify ENABLE_MOUSE_SUPPORT / DISABLE_MOUSE_SUPPORT to include 1003.

* zellij-server/src/tab/unit/tab_integration_tests.rs: add sgr_mouse_mode_any_event tests

* zellij-client/src/input_handler.rs add MouseEvent::Motion case to handle_mouse_event

* zellij-server-src/pane/terminal_pane.rs: add fn mouse_motion()

* zellij-server-src/pane/grid.rs: add AnyEventTracking to MouseTracking.  Follow the compiler messages to all the cases.

* zellij-server-src/pane/grid.rs: add mouse_motion_signal.  Looks like I also need to modify all of the click signals to send any-event appropriately too when hold is true and tracking is any-event.  (It seems like Hold could be removed in favor of Motion, as a simplifying refactor.  But initially I would leave it.)

This all seems correct. Also @tlinford rightfully mentioned that we can probably do away with a lot of boilerplate by just passing a MouseEvent around instead of having separate methods of doing it. But we can leave things as they are for now I think.

I have no idea on:

* Where the actual decoding of the host terminal stream into mouse events occurs.

We use Wezterm's input parser here: https://github.com/zellij-org/zellij/blob/main/zellij-client/src/stdin_handler.rs#L47 Here we convert from the above lib's event to our own: https://github.com/zellij-org/zellij/blob/main/zellij-utils/src/input/mouse.rs#L22 - this happens here: https://github.com/zellij-org/zellij/blob/main/zellij-client/src/input_handler.rs#L96

* If any other widgets/plugins/etc. need to be modified to handle a new mouse event.

For now I don't think so. It would be cool to add this as a plugin capability, but I'm overhauling the plugin system soon, so I'd rather wait until that's done with.

[AutumnMeowMeow]

A question to think about for after any-event is in: Will zellij later support an optional floating text mouse cursor? Like this?

https://user-images.githubusercontent.com/99296476/185799572-70b98db6-a159-4618-8f75-6de93b3be199.mp4

In the video above, we have a demo running inside its own terminal window, analogous to how zellij-inside-zellij in a floating window might go. Since both outside demo and demo-inside-terminal are in any-event mode, and each inverts the cell where the mouse is, then when the mouse goes over the inside terminal with being double-inverted the mouse could disappear entirely.

My solution is for an "inner" window to notify the outside world that it will be drawing a text cursor, so the outside shouldn't draw one, via a privacy message:

ESC ^ hideMousePointer ESC \

ESC ^ showMousePointer ESC \

You can see both the implementation and the flaw: when not over the inner terminal, there are two mouse cursors.

Should zellij later on do floating mouse cursor, let me know if you do a different solution for this. zellij has a much larger user base, so I would change mine to match zellij.

[imsnif]

Sure @AutumnMeowMeow. IMO this is not a piece of UI I'd like to have - but we can keep this in mind if this changes.

[jondo2010]

@AutumnMeowMeow any progress on this? I'm forever missing pane-resizing with the mouse #1262 and would need this to impl I believe.

[AutumnMeowMeow]

@AutumnMeowMeow any progress on this? I'm forever missing pane-resizing with the mouse #1262 and would need this to impl I believe.

@jondo2010

Sorry, no progress to report. I have been mostly unable to code for the last year (again). :(

[imsnif]

I hope you are well @AutumnMeowMeow !

[tmpm697]

sr, but would love to able to resize pane with mouse :)

with only this addition is what i really needed.

[AutumnMeowMeow]

@imsnif Hiya! No promises, but I finally have a machine beefy enough to do real Rust. ;-) Crossing fingers my brain is working again, gonna look into this. (BTW I noticed on today's main branch that mouse button releases are only being reported when the mouse has moved, rather than on every release.) Here's to hoping in a little bit we get xterm-matching mouse behavior including AnyEvent...

[imsnif]

Hey @AutumnMeowMeow - I'm very happy to read things seem to be looking up for you! Would love to see your top-notch work and insight in this project once again.

About technical details: yes, we would need to implement AnyEvent tracking in both directions (receiving it from the host terminal and tracking/handling it in the terminal panes). I think you're right about missing the release event - the whole implementation there is a bit hacky and I think we did that to solve another issue. If you do end up getting to it, please change it as you see fit and I'll make sure to solve any other secondary issues this causes (or to point you in the right direction if you prefer). Quite a few things have changed and shifted around, but I'd be glad to point you in the right direction(s) or answer questions if you like. Here or on Discord/Matrix at your preference.

EDIT: forgot to answer your comment about mouse release, added it in-line.

[AutumnMeowMeow]

@imsnif Doing much better so far, hopefully it lasts. 😊

Thank you for the permission to be a bit more invasive. ;-) And I already have my first question. This one is IMHO worth asking here rather than at PR time, because of the design intent nature.

We need to turn the MouseButton single-value enum in mouse.rs:

#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash, Deserialize, Serialize)]
pub enum MouseButton {
    /// The left mouse button.
    Left,
    /// The right mouse button.
    Right,
    /// The middle mouse button.
    Middle,
    /// Mouse wheel is going up.
    ///
    /// This event is typically only used with Mouse::Press.
    WheelUp,
    /// Mouse wheel is going down.
    ///
    /// This event is typically only used with Mouse::Press.
    WheelDown,
}

...into a multi-valued thing, much like the termwiz bitflag in input.rs:

bitflags! {
    #[cfg_attr(feature="use_serde", derive(Serialize, Deserialize))]
    #[derive(Debug, Default, Clone, PartialEq, Eq)]
    pub struct MouseButtons: u8 {
        const NONE = 0;
        const LEFT = 1<<1;
        const RIGHT = 1<<2;
        const MIDDLE = 1<<3;
        const VERT_WHEEL = 1<<4;
        const HORZ_WHEEL = 1<<5;
        /// if set then the wheel movement was in the positive
        /// direction, else the negative direction
        const WHEEL_POSITIVE = 1<<6;
    }
}

Would you prefer that I abstract away the termwiz data structure, i.e. make our own version of multi-value buttons? Or re-use termwiz::MouseButtons and expose where necessary? (Note that the HeldMouseButtons enum has to go away regardless -- it will be cleaner in the end, promise!)

I noticed that there is a comment in the code about termwiz and wasm having some kind of compatibility issue, which has me leaning towards the former (roll our own). Plus termwiz is pretty well isolated already and that definitely looks intentional, another reason I'm favoring rolling our own unless you say otherwise.

[imsnif]

Hi yes, you're right - this is very intentional. Let's please roll our own. The reason is that we've jumped around quite a few STDIN parsing libraries (iirc termwiz is our third) and abstracting these away was the easiest way of doing so - so I'd definitely like to continue in this vein.

In the case of wasm these are not so important, because with the plugins we use a different contract expressed with protobuffers here: https://github.com/zellij-org/zellij/blob/main/zellij-utils/src/plugin_api/event.proto#L176-L191

I think if we decide to add these events to plugins later, we can figure something out in that regard.

[AutumnMeowMeow]

@imsnif

I've got the first compile working now. Some things are broken of course, but we'll get there. :-) Going to create a PR so we can discuss further there -- you will be on the hook for how you want to expose MouseEvent::Motion events within zellij.