API to receive mouse back/forward button events (mouse buttons 4 and 5)

[dweymouth]

Checklist

• [X] I have searched the issue tracker for open issues that relate to the same feature, before opening a new one.
• [X] This issue only relates to a single feature. I will open new issues for any other features.

Is your feature request related to a problem?

There is currently no way to react to mouse button events other than the first three buttons and the scroll wheel.

Is it possible to construct a solution with the existing API?

No response

Describe the solution you'd like to see.

Introduce a new API for responding to additional mouse button events. New constants should be added to the desktop.MouseButton enum representing the back/forward mouse buttons (button numbers 4 and 5). It's unclear what the names should be. MouseButtonBack/Forward is inflexible, but continuing the existing naming scheme is also awkward (MouseButtonQuaternary, MouseButtonQuinary)?

Also unclear is where the events should be surfaced. A natural option would be through the existing desktop.Mouseable interface and the MouseEvent type, but a potential complication is that the mouse back/forward buttons are usually best responded to at the window level, and with this API any existing Mouseable CanvasObject would "swallow" the event, even if it can't meaningfully respond to it, preventing it from reaching the top-level window canvas. Perhaps exposing a new interface desktop.ExtendedMouseable or something like that so that existing Mouseable objects wouldn't receive them, and then adding new SetOnMouseDown, SetOnMouseUp functions to the desktop Canvas that receive any mouse events (from the normal or extended buttons) that were not sent to a CanvasObject.

[andydotxyz]

Is this a standard? My understanding was that buttons >3 were mapped to OS functions by the mouse driver.

This is a nice addition idea, but it's implementation/naming needs to match a naming / function scheme that works across all OS.

[dweymouth]

I think though I could be wrong that by default those buttons are unmapped and just issue mouseButton events that it's up to the application to handle (which is mapped to forward/back in web browsers and other apps with navigation). But the user can override them at the driver level to have it issue a key press event or other event type to remap the behavior.

I can try to dig a bit more into this later this week. I did see that GLFW defines constants for mouse buttons 1-8

[andydotxyz]

More info would be good - I could not find any consensus online. If we have to expose button numbers so be it, but a consistent intent api would be better.

[dweymouth]

In Win32, mouse buttons 4 and 5 are delivered to the application to respond to: https://learn.microsoft.com/en-us/windows/win32/learnwin32/mouse-clicks

Cocoa dispatches them with NSEventTypeOtherMouseDown and then has a buttonNumber property in the payload: https://stackoverflow.com/questions/69030392/how-to-differentiate-side-and-wheel-buttons-with-cocoa

Qt has APIs for responding to buttons >3 as well: https://stackoverflow.com/questions/30822134/qml-forwards-back-mouse-buttons-handling

So I think it's clear that these events, by default, are dispatched to applications, though many mouse drivers allow them to be remapped. So I think Fyne should have an API to receive these. I like the ExtendedMouseable idea to keep existing Mouseable widgets that don't respond to these from "swallowing" the event and preventing it to reach the window canvas, since as mentioned, by far the most common use case for these is forward/back navigation, which is usually at the top window level in an app. (e.g. doesn't matter what component you click mouse button 4 on, it always goes back)

[andydotxyz]

Yeah. It seems that the intent is not something we can rely on and may just have to fall back to numbering. Where they should land is a tougher one though, and probably impacts how we should implement them too. The "delivered to canvas" is more like a shortcut than a mouse action... and maybe that is what it is? But then again some applications (CAD etc) like to use mouse taps in the context of the widget to control items in the app (like zoom, pan etc?). I wonder if there is any sort of best practice we can pull on here, or perhaps we need the "ExtendedMouseable" to be available on a desktop.Window or desktop.Canvas as well as widgets?

[dweymouth]

Another idea is a new interface that widgets can implement to specify which mouse buttons they can receive, and default for widgets that don't implement it is they receive buttons 1, 2, and 3 only (as is currently the case). Then in the future, in 3.0 we could merge the two interfaces together so any Mouseable widget now has to say which buttons it's interested in

[andydotxyz]

I think that approach only works for 3.0 as in the 2.x if you implement the interface you handle the events - no event handling is optional once you opt in at the code level...

[dweymouth]

my thought was (naming all TBD):

2.x:

// a Mousable component that is not ExtendedMousable only receives buttons 1, 2, 3 // (as is all that is currently supported) Mousable interface { MouseDown(*evt) MouseUp(*evt) }

ExtendedMousable interface { InterestedButtons() []MouseButton } // register to be able to receive additional buttons

3.x:

Combine both functions in Mousable, and force components to be explicit about which mouse buttons they are interested in

[andydotxyz]

This doesn't feel very clean. If you are simply tappable but the thing you are on top of has a popup menu it would be undesirable for a right-click on the child to show the pop-up associated with the widget you are not currently interacting with.

It feels more like these "meta buttons" are special in some way - are they back/forward in the app or are they "apply a fourth action" on a widget. Looking at it that way I wonder if they are more like modifiers or keys than mouse buttons. Whichever way we go I don't think that the API design should break how mouse interactions currently work.

[dweymouth]

They are mostly back/forward within the app but I'm not sure you can make the assumption that they would always be tied to that functionality (games, etc).

But I have no rush for getting this out in a release I guess since it would be easy for me to add temporary support for Supersonic only in a fork.

[pjanx]

Many mice are rather reprogrammable and can send keyboard events. It is safe to assume that buttons 4 and 5 are for history use, but you can theoretically also have more buttons than that (reported as mouse buttons in terms of USB HID).