Support Pixel Coordinates in Mouse Events

[bbb651]

Is your feature request related to a problem? Please describe. Some terminals offer the ability to report pixel coordinates of mouse events along with the character coordinate. I want to make a UI based on the kitty image protocol (also blocked by #834) where pixel coordinates of events are important.

Describe the solution you'd like I would like crossterm to provide access to these modes and add the pixel coordinates to MouseEvent.

Describe alternatives you've considered if any

Additional context I found this website describing the SGR-Pixels (1016) mode.

[bbb651]

I just realized Event::Resize is also missing pixel sizes (there's a variety of xterm sequences for that, also there's TIOCGWINSZ with the ioctl syscall and SIGWINCH signal on unix). Are these not implemented due platform support? Also is this library for/against implementing image protocols such as the kitty image protocol, iTerm image protocol, sixel, etc? (they are relatively niche in terms of usage and platform support, and are relatively complex and require a cargo feature to avoid image crate dependencies, but I feel like there's a gap in the rust ecosystem for high level terminal image handling).

[bogzbonny]

bump! I would find this useful for the same reason. Basic image interaction in the terminal is an inhibiting feature for making some of my applications fully terminal based

[joshka]

Rather than bumping consider doing the work to work out / design how something like this should be implemented.

Some constraints to keep in mind would be:

• There are a large amount of dependents on crossterm (https://github.com/crossterm-rs/crossterm/network/dependents says 34500 repos), changing a fundamental type like the Resize / Mouse events would be problematic and should be approached with care
• Think about how to implement this behavior in a way that works with terminals that support it and terminals that don't
• Think particularly about various operating systems

Regarding image proto stuff, have you considered using ratatui and ratatui-image? I know that this goes one step further than just raw crossterm, but if your goal is writing working software then this might be a valid approach for your problem.

[bogzbonny]

Yeah I spent a bunch of time going through SGR-Pixels (1016)... the best that I could make out is that, to simply switch to 1016 would replace character positions altogether with pixel positions, making that an optional feature I think would be actually pretty straight-forward however not nearly as useful as having both pixel and character positions (IMO, thoughts?).

Considering that some terminals may not support pixels mouse responses, I imagine that it makes sense to add this under a feature flag (wezterm does, ref). One open question I haven't been able to figure out is - how to get the the height/width of a character from the console to then be able to back calculate the character positions from the pixel dimensions (as to be able to provide pixel and character coordinates). Once this is figured out, everything else seems pretty straight forward. Also I suppose the MouseEvent type would be modified and have a couple extra fields (for pixel coordinates) with this feature flag on.

One more complication which I just thought of is if it would be required to somehow query the terminal for pixel compatibility and only provide the character output if pixel output is not available (even with the feature-flag on).

More relevant discussions: https://github.com/dankamongmen/notcurses/issues/2326

---

edit: from that same document bbb651 posted, cells in pixels are CSI 16 t aka echo -e "\e[16t"

[bogzbonny]

@joshka btw implementing kitty image protocol from scratch isn't that difficult (I don't know about sixels though, I haven't tried), the need for supporting cursor location in pixels is so that one could hypothetically interact with an image while in the terminal - this logic is much more coupled with the inner workings of crossterm

[joshka]

One open question I haven't been able to figure out is - how to get the the height/width of a character from the console to then be able to back calculate the character positions from the pixel dimensions (as to be able to provide pixel and character coordinates). Once this is figured out, everything else seems pretty straight forward. Also I suppose the MouseEvent type would be modified and have a couple extra fields (for pixel coordinates) with this feature flag on.

That part should be simple. Call terminal::window_size() then divide width by columns, height by rows.

[o-sdn-o]

Maybe this will be useful. To get rid of the dependence on the cell size, it is enough to use floating point coordinates with the following format:

fractional_coordinates = integer_cellular_coordinates + coordinates_inside_the_cell / cell_size

This approach allows you to easily convert coordinates to pixel coordinates as well as to cell coordinates back and forth. At the same time, the console application does not need to know the cell size at all. The console application simply uses fractional coordinates as it needs.

I use this in vtm when the console application tracks mouse drags and reports back to the terminal if it is necessary to move the terminal GUI window at the pixel level, xlink: https://github.com/directvt/vtm/issues/571#issuecomment-2073983555, protocol draft: https://github.com/directvt/vtm/blob/master/doc/vt-input-mode.md#mouse

You can play with this by moving the vtm GUI windows on Windows (run vtm.exe, vtm.exe -r term or vtm -r test).