search

flxzt / rnote

Sketch and take handwritten notes.
https://rnote.flxzt.net
GNU General Public License v3.0
8.19k stars 282 forks source link

Pen buttons not working as intended on windows 11 on Surface pro #785

Closed Doublonmousse closed 3 months ago

Doublonmousse commented 1 year ago

Describe the bug
On windows 11, the primary button of the stylus is not working as intended.

When the button is pressed, this is not detected until the screen touches the screen, even when the pen is in proximity of the screen.

This leads to unwanted behavior :

-When the toggle mode is selected, the button press is registered only when the pen touches the screen. But the select tool is not activated and neither the previous pen tool until the pen is lifted then put back on the screen.

Expected behavior

The correct behavior is the one observed under linux.

On linux, button presses on the pen are registered if the pen is close enough to the screen. When using the primary button as a temporary toggle for the selector tool, the selector tool is active on the first contact of the pen with the screen with the button pressed.

Screenshots

https://github.com/flxzt/rnote/assets/115779707/816bad8a-3b81-4b1a-b6e6-51ad21f68836

Desktop:

Doublonmousse commented 1 year ago

After some debugging following #832 on windows, and knowing that gtk is the one handling pen events and reporting them back to rnote, I found that

Looking at the event log when the pen approaches the screen with the side button pressed : 

2023-10-12T07:17:19.981Z DEBUG rnote::canvas::input > handle pen event element - element: Element { pos: [[322.5976257324219, 105.57954406738281]], pressure: 0.0 }, pen_state: Proximity, event_time_delta: 0ns, modifier_keys: [], pen_mode: Some(Pen)
 2023-10-12T07:17:19.989Z DEBUG rnote::canvas::input > canvas event MotionNotify - gdk_modifiers: ModifierType(0x0), is_stylus: true
 2023-10-12T07:17:19.989Z DEBUG rnote::canvas::input > handle pen event element - element: Element { pos: [[323.49139404296875, 104.00224304199219]], pressure: 0.0 }, pen_state: Proximity, event_time_delta: 0ns, modifier_keys: [], pen_mode: Some(Pen)
 2023-10-12T07:17:19.989Z DEBUG rnote::canvas::input > canvas event ButtonPress - gdk_button: 3, is_stylus: true
 2023-10-12T07:17:19.989Z DEBUG rnote::canvas::input > handle pen event element - element: Element { pos: [[323.49139404296875, 104.00224304199219]], pressure: 0.3369140625 }, pen_state: Proximity, event_time_delta: 0ns, modifier_keys: [], pen_mode: Some(Pen)
 2023-10-12T07:17:20.000Z DEBUG rnote::canvas::input > canvas event MotionNotify - gdk_modifiers: ModifierType(BUTTON3_MASK), is_stylus: true
 2023-10-12T07:17:20.000Z DEBUG rnote::canvas::input > handle pen event element - element: Element { pos: [[323.0182189941406, 104.58059692382813]], pressure: 0.373046875 }, pen_state: Proximity, event_time_delta: 0ns, modifier_keys: [], pen_mode: Some(Pen)
 2023-10-12T07:17:20.079Z DEBUG rnote::canvas::input > canvas event MotionNotify - gdk_modifiers: ModifierType(BUTTON3_MASK), is_stylus: true

the button press happens around the same time as the pen touches the screen (I'm not sure if it is just before or just after), and create the bug (There is no way someone would click on the button at the exact same time as touching the screen).

Contrary to the two-step process of switching to the selector tool in linux, only one of the step seems to be executed.

However, xournal++ comes with a workaround, named Merge button events with stylus tip events (see : https://github.com/xournalpp/xournalpp/issues/4388). It is not perfect as there is no indication that the button press was registered until the pen touches the screen but the tool activates nonetheless.

So two ways could be explored to fix this

I wonder if this is a surface-only issue or a more general Windows ink -> gtk3/4 issue, and would be interested to see if any other surface/windows user see the same issue.

Doublonmousse commented 1 year ago

So I managed to fix it downstream (on a fork) but for now it's a proof of concept than anything else (I'm using unsafe commands and static mutable variables for now as I needed to add a new state variable not linked to glib !).

Effectively, the inputs given by gtk4 are wrong, button presses are not detected whilst the pen is in proximity mode so only two events out of normally 4 are fired. Out of the 4 events that we would expect

Still need more input on whether this is a Surface-only issue or not.

PeterWang-dev commented 1 year ago

So I managed to fix it downstream (on a fork) but for now it's a proof of concept than anything else (I'm using unsafe commands and static mutable variables for now as I needed to add a new state variable not linked to glib !).

Effectively, the inputs given by gtk4 are wrong, button presses are not detected whilst the pen is in proximity mode so only two events out of normally 4 are fired. Out of the 4 events that we would expect

  • ButtonPress on Proximity mode (first press on the button, detected when the pen is close enough)
  • ButtonPress on Pen::Down mode (we touch the screen with the button pressed)
  • ButtonRelease on Pen::Proximity mode (pen leaves the screen)
  • ButtonRelease on Pen::Proximity/Up (button released) I only get the first one (forces Proximity for a pen that is on the screen !) and the third or fourth (to be checked). However, these events can still be uniquely detected (at least on my hardware) so we can force the correct sequence of events to take place with the caveat of not having any "hover" effect as these are not registered.

Still need more input on whether this is a Surface-only issue or not.

Same thing happens on Lenovo Yoga 7i. It seems a general problem, not only surface.

Doublonmousse commented 1 year ago

You commented on https://github.com/flxzt/rnote/issues/705 but the issue is different (persisting between sessions vs some button presses not registered by gtk) so I don't know what to think of this...

I checked and the Lenovo Yoga 7i is not compatible with the microsoft pen protocol, so this points to a larger problem than just surface pens at the very least on the windows side.

Could you get the input debug information or try the "hacky fix" ? This would really help

PeterWang-dev commented 1 year ago

You commented on #705 but the issue is different (persisting between sessions vs some button presses not registered by gtk) so I don't know what to think of this...

Could you get the input debug information or try the "hacky fix" ? This would really help

I'll test and give you some log information later.

I checked and the Lenovo Yoga 7i is not compatible with the microsoft pen protocol, so this points to a larger problem than just surface pens at the very least on the windows side.

Lenovo Yoga 7i is using Wacom AES 1.0 protocol actually. I checked your rnote-ui/src/canvas.rs front-end source code. It seems that you use gtk api to deal with pen input, convert and structurize into the PenEvent defined in rnote-compose/penevents.rs . So if GTK just support AES 1.0, it gotta be work.

By the way, I found more severe problems when using it. I'll check existing issues and may open new ones. Just list potential bugs here first:

  1. Rnote struck and crashed when writing after scaling occasionally.
  2. Performance dropped when dragging the canvas after drawing more and more curves.
  3. The primary butten and secondary button were flipped and did't work properly. e.g.
    • Edited primary button actually changed the behavior of the bottom button instead of expected top button.
    • Secondary button (bottom button actually on the stylus) was set to "Toggle" but automatically untoggled when pressing and releasing the button.

I am not a native English speaker. Hope I described the issue clearly to make me understood. I'll grab some logs later after my today's work. Hope that could help you.

PeterWang-dev commented 1 year ago 
$ RUST_LOG=rnote=debug ./rnote.exe
 2023-10-18T03:26:09.690Z DEBUG rnote > ... env_logger initialized

(rnote.exe:2720): Gtk-CRITICAL **: 11:26:10.610: gtk_widget_class_set_accessible_role: assertion '!gtk_accessible_role_is_abstract (accessible_role)' failed
 2023-10-18T03:27:47.253Z DEBUG rnote_engine::store > state has not changed, no need to record
 2023-10-18T03:35:33.369Z DEBUG rnote_engine::store > state has not changed, no need to record

(rnote.exe:2720): Gdk-CRITICAL **: 11:36:07.969: gdk_button_event_get_button: assertion 'GDK_IS_EVENT_TYPE (event, GDK_BUTTON_PRESS) || GDK_IS_EVENT_TYPE (event, GDK_BUTTON_RELEASE)' failed
 2023-10-18T03:36:11.951Z DEBUG rnote_compose::builders::penpathmodeledbuilder > PenpathModeledBuilder: updating modeler with element failed,
n_steps exceeds configured max outputs per call.

After some tests, the Rnote crashed again with log above. @Doublonmousse

PeterWang-dev commented 1 year ago

The previous comments are off-topic in this issue. Open a new one here. #843

Doublonmousse commented 5 months ago

Upstream issue : https://gitlab.gnome.org/GNOME/gtk/-/issues/6723

The TLDR is that this is a Win32 specific issue (older API with weird limitations, including pen buttons) and unless gtk switches to WinRT to get pen information (and introduces C++ in the process) it won't change anytime soon.

The other not good thing is that the grass is not greener on the other side. Qt has the exact same problem and any rust gui that decides to use win32 instead of winrt will also have the exact same issue

Doublonmousse commented 3 months ago

Fixed (workaround) by #1113