Pen/Tablet Input Support

[ghost]

I propose we support pen tablet input, such as on devices like the Surface Pro/Studio, Galaxy Note, and eventually the Apple Pencil. These devices support things like pressure sensitivity and rotation/gyro. I don't think this is necessarily relevant to games (although it definitely can be), but it's definitely relevant to the larger digital 3D community.

[tangmi]

There is an issue open on imgui (https://github.com/ocornut/imgui/issues/2372) that has a little documentation and discussion on how to handle pen input across platforms that might be relevant when work on this starts.

[DorianRudolph]

I made a preliminary implementation for pen support on X11 here https://github.com/DorianRudolph/winit/tree/stylus

Pen pressure can already be accessed on X11 via the raw axis events, but there you get no information about the range of values.

[lehmanju]

@DorianRudolph What is the state of your implementation? Any help needed? I could look into stylus support for Wayland but first we should probably make sure the API exposed in Winit is correct.

[kchibisov]

@lehmanju if you're interested in contributing Wayland bits, you may add helper for table input to https://github.com/smithay/client-toolkit , since it's something that will be required anyway.

[DorianRudolph]

@lehmanju I'm not working on this right now. I would consider it a proof of concept. The implementation works, but I did not give much thought to the API. It might make sense to implement https://github.com/rust-windowing/winit/issues/336 prior to tablet support.

[kirawi]

A bit confused, but how would you consider it to be not implemented? It looks like it effectively is through Touch.

[tangmi]

Pen input devices can have additional values, such as pen pressure, tilt, and azimuth that are not currently supported in winit. Additionally (and anecdotally), applications sometimes want to be able to distinguish between touch and pen input, even if most interactions end up being the same.

[kirawi]

Touch has the force field which seems to define pressure sensitivity and tilt. I could see the case for the latter and that it only works on IOS and Windows 8+ right now, but wouldn't you still consider that stylus input is mostly supported?

[tangmi]

That's interesting... Looks like it's only used for the Apple Pencil: https://github.com/rust-windowing/winit/blob/b9f3d333e41464457f6e42640793bf88b9563727/src/platform_impl/ios/view.rs#L233

I think we'd still need to hook up azimuth to do that cool tilt shading effect Apple showed off when they released the Pencil...

Although even more broadly than that, I still think it's probably worth it to separate it out into a more generalized Pen event and implement it for all the platforms.

Edit: this may be a breaking change--at least on Windows, if one explicitly asks for pen input, the OS will stop sending them as touch inputs.

[kirawi]

It looks like Windows does have pressure sensitivity support: https://github.com/rust-windowing/winit/blob/66859607a35f7e46a25346511657d3b7ada939ca/src/platform_impl/windows/event_loop.rs#L1482-L1496

Rotation/tilt should be trivial to add as well.

I'm confused why normalize_pointer_pressure() expects to only get a maximum of 1024, though this issue seems to imply that the Win32 Pointer API is limited to that number compared to WinRT. It might be worth investigating that in the future?

Edit: Yes, Win32 is limited to 1024.

[tangmi]

I'll admit the last time I seriously used winit was pre-0.20, and pen support wasn't sufficient for any non-trivial pen-based apps (for reference on what's missing, here's a proof concept supporting pen on Windows and web here https://github.com/rust-windowing/winit/compare/2e11615...tangmi:winit-legacy-2). I don't believe that any significant support for pen/tablet input was added since then, so I think that it's meaningful to keep this issue open.

[kirawi]

Agreed, it doesn't seem very mature yet.

[tasgon]

Looking through @DorianRudolph's PR, it looks like pressure is normalized, so could we use the Touch event in a similar way to how it's done on Windows (instead of adding another event struct)?

[adsick]

Just want to indicate that I wish pen input to be in place

[daxpedda]

For reference, pointer prediction is also a thing:

• Web: https://www.w3.org/TR/pointerevents3/#predicted-events
• Windows: https://learn.microsoft.com/en-us/windows/windows-app-sdk/api/winrt/microsoft.ui.input.pointerpredictor.getpredictedpoints
• MacOS: https://developer.apple.com/documentation/uikit/uievent/1613814-predictedtouchesfortouch

[adsick]

Can we have this in foreseeable future? May I help?

[DorianRudolph]

It seems like the pieces are there for all platforms but wayland. So depending on your needs, you can already use a tablet with either mine or Atilogit's PR. Someone (maybe you) just needs to prepare a production ready PR, I think. Currently both PRs implement the pointer on top of the touch event, which is not ideal, so we should create separate events for pen inputs.

[kchibisov]

Can we have this in foreseeable future? May I help?

I'm not sure how near the future would be, but we lack the general API for that and the API for tablets is really complicated if you've read into what platforms exposed.

[0x182d4454fb211940]

I think I have a rough understanding of how the wayland API works, and I'll try implementing this for wayland soon. It seems like how to report the events to the user hasn't been decided yet, so should I just use what's convenient? That way people can worry about the specifics after most platforms are implemented.

[kchibisov]

The main issue with all of those devices(mouse, touch, pen) is that they are not unified in winit, which a bit annoying. If you have an idea and you want to work on that you could probably try to propose top level API and an example first, without implementation in the backend.

There's also patches to add pen which you could probably take a look at #2647 , #2396, and #1879. So you could give some brief information what other backends have.

There's a tablet pad(or whatever it's called) to consider, because it also has events...

[ActuallyHappening]

Being able to handle pen double-taps on my iPad would be amazing!

[madsmtm]

Thanks for the interest in working on this in https://github.com/rust-windowing/winit/issues/3759 @ActuallyHappening! I think the hardest part about this is not so much the implementation, but more figuring out what the generated WindowEvents should look like - maybe you (or someone else) could start by suggesting an API here, and then we can work from there?

[ActuallyHappening]

I work primarily with the bevy wrappers around winit. From that point of view, the only event I would really care about looks like this (allowing for forwards compatibility):

#[non_exhastive]
#[derive(Debug)[
pub enum PenEvent {
  DoubleTab,
  // Squeeze // todo
}

Thinking about this more, I believe only apple pens have the specific 'Double Tap' feature. Of course there would be documentation to show this. Keeping the enum descriptive (rather than pub struct PenEvent { is_tap: bool }) seems a better design, but my only requirement (for my specific use case) is the ability to know when (an event) I double tap my Apple Pencil. It would also be good to wrap some specific information about the requested action is, see preferedTabAction https://developer.apple.com/documentation/uikit/uipencilinteraction/3039593-preferredtapaction

[madsmtm]

Resolution from the meeting today: We want to start with something matching the web API, specifically pointerType, twist and tilt[X|Y]. @daxpedda wants to try to do this implementation.

On top of that, we could probably add some extra methods to know if an event was e.g. a double tap or a special button press, but we want to start with the above.

[ActuallyHappening]

OK interesting, though in the link to the web API you mentioned there is no functionality specific to Apple Pencils / Galaxy SPens buttons. I.e., after implementing the entire web API, I would still not be able to handle Apple Pencil double tap (or squeeze) events like I want to (see bevy issue and #3759).

Would this API be mutually compatible with the decided web API?

/// Pen events not necessarily tied to a normal touch, move, or drag e.t.c.
pub enum PenSpecialEvent {
  DoubleTap { preferred_action: Option<PenPreferredTapAction> },
  // Squeeze // todo
}

I guess I should explain that the Apple Pencil double tap feature often occurs when the pen is not touching the screen, so information like screen position doesn't make sense for it.

[madsmtm]

Maybe it would help to know what kind of events the pen generates when used in a web page?

[ActuallyHappening]

Ahh I see, trying to mimic the Web API for native. I initially tried to gain pen events on web, but special Apple pen events (i.e. DoubleTap) don't trigger anything on Web, which was one of the initial factors pushing me to use rust for native in the first place, so that I could use the features of my apple pencil. I haven't checked that this is still the case recently however.

Looking over the web api we could merge it with the features I want. From the pointerType property:

The event's pointer type. The supported values are the following strings: "mouse" The event was generated by a mouse device. "pen" The event was generated by a pen or stylus device. "touch" The event was generated by a touch, such as a finger. If the device type cannot be detected by the browser, the value can be an empty string (""). If the browser supports pointer device types other than those listed above, the value should be vendor-prefixed to avoid conflicting names for different types of devices.

I understand more what you mean by extending the official spec now:

On top of that, we could probably add some extra methods to know if an event was e.g. a double tap or a special button press, but we want to start with the above.

This does seem to be a more consistent and unified API.

The only problem I see with trying to add Apple Pencil double tap event support is that the event itself is not associated with any of the touch event types in the web API spec, here. And, it is required that each pointer event from the spec has a specific .clientX, .clientY e.t.c. from the MouseEvent interface however apple pencil double tap events don't have an associated position on the screen, because you are often holding the pencil off the screen when you double tap

[ActuallyHappening]

I'm still eager to implement this, see #3768

[daxpedda]

The only problem I see with trying to add Apple Pencil double tap event support is that the event itself is not associated with any of the touch event types in the web API spec, here.

This is to be expected, there will be a bunch of special events that just don't fit anywhere and maybe are platform-specific with no overlap.

I think to start the design work of post-Web-API-pencil-input we want a proper overview.

1. A list of relevant events for each backend.
2. Find overlap between the backend.
3. Consider which ones to implement or if we want/can let external crates to implement those.

Documentation:

• Android: NDK input
• iOS/MacOS: PencilKit?
• Wayland
  • libinput: API, documentation
  • Tablet protocol
• Windows: GetPointerPenInfo
• X11: x.org is down, couldn't search for it.

[ActuallyHappening]

I'm thinking about my future considerations, I'm wanting to custom build a lot of stuff relevant only to iOS. For that I should really only need a reference to the WinitUIView and WinitUIViewController and the ability to implement protocols on it, but I believe objc2 can only implement objective C protocols in the defining crate.

I'm out of my depth here, but is it possible for winit to expose the ability to add custom views and view controllers to the base / default without messing up any of winits abstractions? That seems the cleanest and best option to me, since I can then maintain my own crates which implement specific features for iOS only rather than peppering winit with PRs for every new iOS specific change.

I think to start the design work of post-Web-API-pencil-input we want a proper overview.

• A list of relevant events for each backend.
• Find overlap between the backend.
• Consider which ones to implement or if we want/can let external crates to implement those.

I can only speak confidently about iOS since it's the only hardware I have. I'm happy to make a list of iOS features I would eventually want:

• Double Tap events + Hover Pose data
• Squeeze events
• Preferred action for tap and squeeze events
• Pen prediction

[ActuallyHappening]

• iOS/MacOS: PencilKit?

I don't think PencilKit can be easily integrated into winit since it relies on a bunch of properties specific to apple's view hierarchy, e.g. inheriting scrolling. To be honest I discounted it immediately when I though that you couldn't get at the underlying data, but that doesn't seem the case

[daxpedda]

I'm happy to make a list of iOS features I would eventually want ...

The goal is to make a more or less complete list. If you include the links to the apple documentation directly, I'm happy to open an issue about platform-specific functionality so we can start collecting and designing an API for them.

[Doublonmousse]

As the windows side of the implementation was mentioned, and the proposed implementation is relying on the Win32 API, I would like to say that the Win32 API is limited in some ways that create unwanted behavior (as in, different from any other implementation, with my main gripe being the absence of information on the button presses when hovering. The result is that buttons are usually not working correctly on windows for any application using gtk3/gtk4 or qt).

I've listed my concerns (and a fix using additional WinRT API calls) on an existing PR here: https://github.com/rust-windowing/winit/pull/2396#issuecomment-2125625651

Other Windows-specific features related to pens are only available using WinRT APIS (haptic feedback, pressure levels with more than 1024 levels)

[Doublonmousse]

Another thing to note on windows is that pen APIs are split between Windows Ink (what we get from the Win32 or WinRT APIS) and Wintab, and these two are still used today.

Windows Ink is used by all modern tablets with a pen digitizer on the screen, whilst Wintab is still widely used by external graphical tablets (wacom, huion). And of course, if your graphical tablet uses Wintab as the API and you're getting events from the Windows Ink API, these events are seen as mouse events, not pen events ...

Usually you have to choose one of the two APIs to catch pen events and restart the app if you want to switch APIs.

https://crates.io/crates/wintab_lite This seems like the only rust crate that deals with wintab right now, although, as I've only looked at the Windows ink part, I'm not familiar with the wintab part. (I know both qt and gtk have a wintab option, so there's other C/C++ code that implement the Wintab API).

[thehappycheese]

https://crates.io/crates/wintab_lite This seems like the only rust crate that deals with wintab right now, although, as I've only looked at the Windows ink part, I'm not familiar with the wintab part. (I know both qt and gtk have a wintab option, so there's other C/C++ code that implement the Wintab API).

Oh hi there :) I am the author of that wintab_lite crate.

FYI i gave up trying to integrate with winit because winit blocks access to the window event loop by making it private.

wintab does provide an alternative polling-based api to obtain events, I think that method sucks. It is prone to overflowing buffers on the driver side which causes inexplicable bad behaviour. For this reason I think winit's choice to hide the event loop is not compatible with wintab... unless i am somehow mistaken.

This is unfortunate, because I find Windows Ink to be highly unreliable and flakey. Wintab has been working solid as a rock for decades.

[Doublonmousse]

Hey. I think part of the reason windows Ink is flaky is the use of the win32 API over the WinRT one (hence why I tried to use WinRT calls in addition to the win32 ones to get the missing info or more info on events, and that gives correct info for pure windows ink devices like a surface pen). That means that for most windows software supporting windows ink that aren't WinUI note taking applications the support is limited and has been for a while now.

But also, as far as external graphics tablets are concerned, it's clear to me the optional windows ink mode is usually less polished and featured, although the fault may be split between drivers manufacturers who already have a good wintab drivers not needing an Ink one, and Windows Ink being less featured than wintab (sometimes using the open tablet driver + windows ink can give you some level of functionality that the official driver lacks in windows ink mode).

But enough ranting, do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here https://github.com/rust-windowing/winit/commit/0ba247b3f4b16783d5daf8a8a4576020dac29f90) could be done instead ? (also if someone know a better way to get pen info from WinRT let me know)

One other point you might have an answer to : on most toolkits that have wintab and windows ink support, you can usually only choose one and need to restart to use the other one (that's the case for both gtk and qt). Is there a fundamental limitation that makes having both activated at the same time impossible ? Cause if you have a device that has both a windows ink device (like a pen digitiser) + an external graphics tablet that uses wintab (and you don't want/can't use the windows ink option), there's always one device that won't work correctly.

Anyway, once the API is decided in #3833 and proper platform-dependent support can be added to winit, I'd like to make sure it doesn't have the usual limitations other windows implementation have on the windows ink side. And if we can do better, why not try to have support for both at the same time (windows ink and wintab) without a need for a restart ?

[daxpedda]

FYI i gave up trying to integrate with winit because winit blocks access to the window event loop by making it private.

wintab does provide an alternative polling-based api to obtain events, I think that method sucks. It is prone to overflowing buffers on the driver side which causes inexplicable bad behaviour. For this reason I think winit's choice to hide the event loop is not compatible with wintab... unless i am somehow mistaken.

We plan to address this with https://github.com/rust-windowing/winit/issues/2120 and the move to a trait-based API, #3432, with the goal that external crates can integrate with Winit exactly like wintab_lite could do.

I was a bit shocked with the terms "private" and "hide the event loop", the event loop is a function and integrating with it requires more API then just "making it public". But maybe I'm missing something, I'm not very familiar with the Windows backend.

But enough ranting, do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here 0ba247b) could be done instead ?

On first inspection I don't think its viable to add Wintab support directly into Winit. We want to limit the scope to cross-platform functionality and only implement and expose whats absolutely necessary.

But as said above, I think its incredibly important that Winit allows external crates to integrate into the event loop exactly for reasons like this. On the other hand, I would love to see Wintab and Windows Ink support in winit-extra (doesn't exist yet).

[thehappycheese]

Do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here https://github.com/rust-windowing/winit/commit/0ba247b3f4b16783d5daf8a8a4576020dac29f90) could be done instead?

I am sure a built in solution would work great :) My plate is a little full at the moment to try help, but feel free to copy-pasta my code all you like. I spent a lot of time adding nice rust documentaiton to it.

Is there a fundamental limitation that makes having both activated at the same time impossible

Not as far as I know... I have a Huion device, the configuration software has a toggle to turn windows ink on and off... as far as I can tell the toggle is broken and does nothing. It still works in windows ink apps. Maybe windows has some fallback and translates wintab into windows ink... i didnt look into it.

I can tell you this though; for about a month windows ink was not working for me. The fix: disable logitech RGB LED driver that I didnt even know existed. Go figure? It did not interfere with wintab though.

We plan to address this with #2120 and the move to a trait-based API, #3432, with the goal that external crates can integrate with Winit exactly like wintabl_lite could do.

I was a bit shocked with the terms "private" and "hide the event loop", the event loop is a function and integrating with it requires more API then just "making it public". But maybe I'm missing something, I'm not very familiar with the Windows backend.

Ah thats exciting to hear! By private I mean that at the time I looked, the values windows passed to the message / wparam / lparam parameters of the callback registered with CreateWindowExA() do not appear to be avaliable through the winit API. I dont remember the exact details... I think winit maybe sets up the window / event loop / polling differently to what I understand.

On first inspection I don't think its viable to add Wintab support directly into Winit. We want to limit the scope to cross-platform functionality and only implement and expose whats absolutely necessary.

Possibly licencing is also an issue for serious projects? Wacom's licence / coppyright notice on thier repo seem contradictory to my primitive understanding of these things.

Thank you all for your hard work on winit though, it is otherwise absolutly fantastic and it sounds like its going to be even better in the future! <3