search

rust-windowing / winit

Window handling library in pure Rust
https://docs.rs/winit/
Apache License 2.0
4.76k stars 893 forks source link

Richer Touch events / unified pointer event model #336

Open chris-morgan opened 6 years ago

chris-morgan commented 6 years ago

I note Touch, but it’s rather limited. My Surface Book, for example, has a pressure-sensitive pen, which neither Touch nor Mouse can model; and the latest such pens support tilt as well.

I suspect it’s worth looking into something like the Pointer Events API which browsers have all comparatively recently taken up.

marcianx commented 6 years ago

I'd also like to note https://github.com/tomaka/glutin/issues/916 which I was asked to forward to here:

The Touch event does not include a touch radius or any other obvious way to distinguish finger and stylus touch events. This is particularly useful for palm rejection on large screens when using a stylus. Could such support be added?

tangmi commented 5 years ago

Would a precursor to this be getting pen/tablet support first (https://github.com/rust-windowing/winit/issues/99)?

Edit: some literature that might help implementing pointer events cross-platform: https://gist.github.com/k3a/ef98593a3571b1b399169448327ad333

Edit2: It seems like there needs to be a decision on whether to keep Mouse/Touch/Pen separate or unify them under a "PointerEvent" type. Perhaps winit could redundantly fire the separate events with the unified and let the app decide whether to handle them separately or unified (like how the web pointer events api currently is?)

Osspial commented 5 years ago

@tangmi I think it goes the other way - getting richer touch support will lay the ground for adding pen/tablet events. We've slowly been making progress towards richer touch events, via recent touch pressure PRs for iOS and Windows, and I expect that once more cross-platform support for rich touch events has been added pen/tablet support won't be too much of a technical or API jump.

It seems like there needs to be a decision on whether to keep Mouse/Touch/Pen separate or unify them under a "PointerEvent" type.

That's... an intriguing idea, actually. I'd support unifying pointer events into a single type, especially since that'll improve usability in the naive case where an application is designed for just one input type and given input of a format it didn't necessarily expect.

chris-morgan commented 5 years ago

The Pointer Events API which I suggested as a model is all about unified event types. If you’re not familiar with it, please do read up on it before implementing anything in Winit.

The web still does click events for touch and pen, mousedown et al for pen events, and scrolling/zooming for touch events (adjustable via the touch-action CSS property). Naive users should use those things rather than pointerdown et al. This is a sane model for more than just the web’s backwards compatibility, because touch should generally act differently from a mouse or pen.

tangmi commented 4 years ago

@Osspial I wanted to try pen input on web through winit, so I've hacked together basic support for pointer events in winit-legacy (diff here: https://github.com/rust-windowing/winit/compare/2e11615...tangmi:winit-legacy-2).

Some observations that may already be known:

Notes on this changeset:

Cheers!

Osspial commented 4 years ago

@tangmi Thank you so much for investigating this!

A unified pointer event API was super easy to consume from my app's point of view, although there needs to be some thought done on e.g. what pointerdown (and no button pressed) vs mousedown means, scrolling events, etc.

I think the cleanest way to expose input-specific events would be to have an enum field within Pointer that contains any non-shared data. I don't know what to do about other sorts of events (like scrolling), though - that'll warrant some further thought.

The Windows provides APIs to handle all inputs in a given frame at once (which allows for handling e.g. multitouch gestures). winit (legacy) currently doesn't provide support for this?

I don't fully understand how to call the pointerinfo APIs on windows, so on some hardware pen input skips?

Our touch handling has been improved quite a bit on master, actually (see #991 and #1134). I'm not familiar with the specifics of how the Windows touch API works, but I'd guess that those features would be easier to support with the new implementation. That implementation also seems to handle pointerinfo gracefully.

Switching from multiple input event kinds to a single input event kind was disruptive enough to the API that I decided to leave the mouse events as is to keep support in third-party libraries (like imgui-winit-support). I wonder if the pointer events/richer touch event support should be considered before 0.20 is released?

0.20 is already big enough - I'd rather not expand its scope further and end up in a perpetual alpha. I'm totally fine with removing the old-style mouse events in a future release, though. When we're improving our APIs, we should kill the clunky, thrown-together APIs while we aren't tied to stability guarantees.

tangmi commented 3 years ago

@Osspial: I'm revisiting this because I got some emails on other issues relating to this... Since 0.20 got released, is there any additional blockers preventing this? I think I can port my Windows impl to head and start the investigation/discussion for other platforms?

kirawi commented 3 years ago

@tangmi I was investigating doing that myself before I scrolled down to your comment. Though I'm not very familiar with the overall winit codebase, the design of the existing Touch event/struct is similar enough to your implementation that you could likely reuse much of what's already there and merge your other changes. I'm not too sure on the mouse or touchpad, however.

Here are my notes: