Support pointer capture

[viridia]

This was discussed on discord, but I'm writing down the details here.

Background: Many of the design choices of bevy_mod_picking are modeled after the web, and more specifically the documented behavior of the HTML DOM. The intent, I believe, is to make it easy for programmers who have web experience to transfer their skills to Rust and Bevy. One feature that is missing, and would be useful, is pointer capture. This API is used in many popular and creative widgets that support dragging: sliders, spinners, split panes, color wheels, hue/saturation pickers, trackballs, knobs; even complex components such as node graph editors.

The "capture" state would associate a given pointer (identified by a pointer id) with a given entity. While captured, all pointer events for that pointer are routed to the target entity, regardless of where the pointer is on the screen.

Pointer capture lasts until one of three conditions occurs:

• The client calls release_capture.
• A pointer "up" event is received.
• A pointer "cancel" event is received. This often happens because the window loses focus. The motivation here is that we'd like the window focus loss to interrupt dragging so that we don't get "stuck" in a dragging state.

A typical slider widget on the web uses pointer capture in the following way:

• on pointer down: set capture to the slider element, and record the current mouse offset.
• on pointer move:
  • if the current element does not have capture, ignore the event
  • if the current element does have capture, compute the new slider value and thumb position based on the current pointer position coordinates and the initial offset recorded from the 'down' event. The slider value is of course clamped to the min/max range of the widget.

Note that because capture is automatically released by pointer up, there's no need for an "up" event handler, unless the widget wants to send a final event to indicate that the user has finished dragging. The 'hasPointerCapture()' method is used to determine whether the slider is in a dragging state, avoiding the need for a boolean state variable.

Also note that this scheme does not use "drag" events (which are kind of a pain in JS because they aren't entirely standard across browsers), but just ordinary pointer events. In general, "drag" events are only needed when dragging between two separate components that have no knowledge of each other.

A slightly more advanced version of this is an widget which supports both double-click and drag gestures, such as a node editor. On the web, the capture state prevents double-click events from firing, so supporting both requires delaying capturing until the drag has moved a few pixels.

[aevyrie]

It sounds like pointer capture is mostly useful for drag events, yet this crate already provides drag events. You mention:

Also note that this scheme does not use "drag" events (which are kind of a pain in JS because they aren't entirely standard across browsers)

Non standardization isn't something we need to worry about. With that in mind, what does pointer capturing enable that can't already be done with the drag events?

[viridia]

The short answer to your question is "nothing" - that is, there's nothing that can be done with pointer capture that can't be done with drag events as you have defined them.

The longer answer is "simplicity". With pointer capture, you have fewer events that need to be subscribed to. A slider widget, for example, only needs to subscribe to Pointer<Down> and Pointer<Move>.

The way that you have implemented drag events is reminiscent of some older UI desktop frameworks in which dragging is a special state handled by the UI framework (this is not a criticism, just an observation). Most widgets on the web today simply use pointer events, and manage the dragging state themselves.

Things get a bit confused because browsers also have something called "drag and drop" (DnD) which is mainly about dragging files from the desktop onto the browser window. This is a OS-level feature where the browser hooks into the DnD events from the native operating system. A "drag" event contains a payload and an icon. The payload consists of one or more data blocks, each of which has an associated MIME type. The icon is an image which is set by the drag source, which is often the icon for the file type being dragged. The recipient of the drag has no control over the data or icon, but can highlight itself (or not) based on whether the MIME type is acceptable. You can drag other things besides files - for example, in the past I've implemented a UI that lets you control the order of columns in a table by dragging.

As you can imagine, this is a fairly heavyweight operation compared to dragging a slider thumb, which is why slider widgets don't use it.

I only mention DnD because the terminology is confusing, what you call a "drag" event is not the same as what the browser calls a "drag" event, which is a DnD event. Other than DnD there are no drag events in the browser.

Some widgets on the web have complex needs that don't fit into the standard model. Here are some use cases:

1) Widgets where the capturing element is not the same as the click element. Imagine for example a node graph. You want to be able to drag a connector from one node to another. The "pointer down" is detected on an input or output terminal, but the element handling the drag is the whole graph, because it wants to know whether the drag is targeting a terminal of a different node. Even in the simple case of a slider, you might click on the thumb, but the widget might want to handle the drag events on the slider as a whole.

I'm guessing that this is possible under your current framework, although I haven't researched how it might work.

2) Widgets where dragging is optional, depending on other factors like modifier keys. This can be done in your current system by setting a state variable that tells the widget to ignore the drag events in some cases.

3) Widgets where the drag operation is delayed. For example, some draggable items are "sticky", that is, they don't start dragging until the mouse has moved a bit. Again, this could be done with your current system by setting a state variable.

4) Widgets that control the appearance of the mouse pointer while dragging. On the web, mouse cursor appearance is tightly integrated with picking behavior. The cursor appearance is a CSS style property, which means that the cursor automatically gets set based on what the pointer is currently hovering over. During a pointer capture, however, the mouse is always considered to be "hovering" over the captured element, so the cursor no longer changes.

Your system doesn't address cursor appearance at all, but eventually people are going to want this because cursor shape is an important visual signal (especially for things like text input fields, which have an I-beam cursor). It's possible to implement a cursor appearance system on top of your framework, but the implementation will be complicated, and this system will need to have its own understanding of something similar to pointer capture. This means additional complexity, since we now have two separate systems that are tracking drag state.

5) One case which pointer capture doesn't handle, which your system does, it multi-target dragging - that is dragging A on top of B. On the web, there are two ways to implement this. If the drag targets are relatively static and are HTML elements, you can use DnD (DnD targets must be an element with a special attribute). Alternatively, you can capture the pointer and then explicitly call the browser picking functions - I've used this technique with node graph editors.

So the bottom line is, anything you can do with pointer capture can be done with your current design - the question is, which approach yields the best / simplest developer experience?

[viridia]

So, now that I've had a chance to dive more deeply into bevy_mod_picking, I can better understand the difference in use cases between dragging and pointer capture. Either that, or cases where drag events could be improved.

Take a widget such as a slider. When a slider is being dragged, all events go to the slider. If, while dragging the slider, you move the mouse over a button or other widget, you'll notice that they don't highlight.

In the pointer capture world, this is easy to explain: when a ui element has capture, other elements on the page don't get in/out events.

However, in a dragging world, you actually do want other elements to get in/out events because they might be a target.