Desktop platforms: Mouse and Keyboard classes or DependencyServices

ToolmakerSteve commented 2 years ago

Description

There should be a standard way to query whether a physical keyboard exists, and if so, to use its functionality.

Similarly, there should be standard mouse detection, including screen position, down/move/up, and view/subview enter/exit/hover/tooltip.

Public API Changes

WPF, .NET Framework, and Windows Desktop all support System.Input.Keyboard class.

Expose this as MAUI.Input.Keyboard? Or in MAUI.Essentials as Device.Input.Keyboard?

And the related Proposal for Mouse Input.

Would it make sense for this to be done as a DependencyService?

Intended Use-Case

Desktop platforms generally have a physical keyboard that does not require on-screen visibility.

Desktop apps traditionally have some "global" keys, that are not tied to a Focused element.

Or if they are associated with current VisualElement, perform some action other than changing Text.

nCastle1 commented 2 years ago

It isn't just desktop where proper mouse support is expected - all major mobile platforms have the concept of keyboard+mouse input. A great app will work just as well on iPad with mouse and keyboard as it does with touch. https://developer.apple.com/design/human-interface-guidelines/inputs/pointing-devices

I think being able to detect hover through a gesture recognizer would be a great start. VisualState support on common controls would be a good next step.

GuildOfCalamity commented 2 years ago

For UX/UI there is a need for an OnHover/MouseOver trigger for Windows machine builds; this could be used in tandem with the {OnPlatform} directive.

jrahma commented 1 year ago

Unfocus hides the keyboard but there is no way to show the keyboard on TapGesture

and why only Desktop? this should also be for mobile.

marcojak commented 1 year ago

Really no one needs it on Windows? Using a keyboard is incredibly important to develop a proper app for Windows, it hsould be part of MAUI or at the least of the community toolkit

cdavidyoung commented 1 year ago

Of course this issue applies to Mac as well. For any desktop app I find myself hovering over controls and I expect little help hints to appear. This is much friendlier than needing to look up controls in documentation.

SonOfKorriban commented 1 year ago

I would also be happy for a solution. I need to react in my MAUI-App, if the mouse hovers a control. In WPF we had the IsMouseOver Event and other useful things. They are really missing here in MAUI.

mobycorp commented 1 year ago

NOTE: The following is just my opinion...

I think the MAUI development team completely missed the boat on what to deliver in the GA release of MAUI... And, since GA, very little has been done to address the complete lack of desktop functionality. Yes, I realize how difficult it is to please everyone, but I, for one, am not pleased at all. A large percentage of @jamesmontemagno and @jfversluis posts center around Android development. These YouTube posts too many times just do a New->MAUI project which never gets as far down in the weeds we, as developers, need. I feel this is a lack of respect for the desktop development community. This is NOT to say that James' and Gerald's YouTube videos are lacking. Rather, their videos are stellar in depth and quality...but, they focus TOO MUCH on mobile development. I'd LOVE to see more videos or pages on your MAUI documentation that discuss the real internals on how to make great applications If you're not going to do that short term, then the YouTube examples need to discuss how to write application-specific code. I simply can't do that with the out-of-the-box MAUI toolset at present.

I get that you looked at the Xamarin code base and you must have felt that converting that first was the path of least resistance to getting MAUI out the door, but this was, IMHO, a huge mistake. I guess the question of how many Xamarin, WinUI, WinForms, and WPF developers there are was not considered. I still think that the desktop community is more highly populated than the mobile community... This is just my guess.

We desktop developers needed a decent data grid. We needed mouse-over events. With mouse-over events, we needed a simple way of changing the mouse cursor. I realize the mobile paradigm doesn't operate on a mouse approach. I get that. But, as a developer, I don't have the time to write a whole bevy of platform-specific behaviors and handlers to do what was simple and out-of-the-box as it was in WinForms and WPF. I waited 2 years for MAUI to come out, but it seems that I'll be waiting another two years for MS and third-party developers to do all of the platform-specific controls and bring MAUI up to par with all of the other platforms (Web, mobile, AND desktop). I have had this happen to me and I've seen it happen to others where Gerald simply dismisses doing the due to offer the requested functionality. Often, the response is, "this is not planned" or "this is too hard" (as was the response to fixing dependency injection for shell pages - it flat out doesn't work so I had to write a huge workaround so that a page's viewmodel can be injected). We desktop developers simply get blown off.

@davidortinau, I have expressed this frustration directly to you in emails, via a video chat between you and I many months ago, and also in Issue posts. I saw you qualify the fact that the developer community should not be required to write a ton of handlers in a recent MAUI Standup so I know you know this. This is sooooooooooooo frustrating and I sincerely feel that some reprioritization needs to happen going forward. I was elated to see mouse-over events have been sort of released, but when I looked into how to change the cursor on enter and leave events, the answer was, "write platform-specific custom handlers" to do the job. This really isn't satisfactory.

kucint commented 1 year ago

I fully agree with your opinion @mobycorp. After long years waiting for MAUI to be released and than again waiting for the fix of many Desktop application issues, the company I am working for decided to abandon MAUI in favor of UNO. Pity.

mouralabank commented 1 year ago

Keyboard events allow users to enter text, navigate through menus and dialogs, and trigger actions or commands within the application. Keyboard shortcuts can also provide a convenient and efficient means of accessing commonly used features or actions.

Mouse events allow users to click on buttons, links, and other interactive elements within the application, as well as to select and manipulate objects on the screen. Mouse movements can also provide visual feedback and help to guide users through the application's interface.

Together, keyboard and mouse events provide a versatile and powerful means of interacting with desktop applications. They allow users to perform a wide range of tasks and actions, and can be customized and configured to suit the specific needs and preferences of individual users.

minze-it commented 1 year ago

I totally agree with this requirement! All platforms supported by dotnet-maui offer keyboard support. Doesnt matter if iPad Pro, a Chromebook with the Android app on it or a Windows PC. Keyboard input makes life so much easier when you have one installed. Not being able to capture that input is a major thing for most business apps!

upswing1 commented 1 year ago

In .NET MAUI, a mouseover event can be useful in scenarios where the user is interacting with the app on a device with a pointing device, such as a mouse or a trackpad. The mouseover event is triggered when the user moves the mouse pointer over an element, and it provides an opportunity for the app to provide visual feedback or additional information about the element.

For example, a mouseover event can be used to highlight a button or provide a tooltip with additional information about the button's function. This can help improve the user experience by providing more information and making it easier for the user to interact with the app.

While mouseover events are not as relevant on touch-enabled devices, they can still be useful in certain scenarios, such as when the user is using a stylus or when the app is being used in a desktop environment. Therefore, including a mouseover event in .NET MAUI can help make the app more accessible and improve the overall user experience.

In Conclusion this issue needs to be addressed ASAP.

danielancines commented 1 year ago

Mouse events, keyboard events are very important to create rich desktop applications, like me, there are lot of people who like use shortcuts instead mouse.

faustmet commented 1 year ago

Some use cases for keyboard interactions that replace mouse controls may be:

Ability to tab through windows in an app
Ability to tab, arrow through controls in a tool, manipulate dropdowns, toggles, edit boxes, and continue to tab or arrow through the tool
Ability to tab or arrow through a dropdown
Ability to tab or arrow through a grid, then manipulate rows in the grid with a different command
Ability to dictate when to keyboard navigate through a single window or tool or all tools in the app
Ability to have keyboard commands that represent commonly used operations such as save, export to file
Ability to simulate a right click event

PureWeen commented 1 year ago

Keyboard

Reading through the use cases here. I'm still a bit curious the scenarios for a generic Keyboard/Mouse class. AFAICT the cases that are outlined here can be achieved through accelerators and PointerGestureRecognizer.

Can folks here expand a bit on their use cases, and where a Keyboard Class would be a better solution then using Accelerators? We've added Accelerators to MenuItems for .NET8 but we're looking at adding the Accelerators API to every single control (like WinUI and Catalyst have). If you're wanting to add shortcuts, then Accelerators are the most a11y compatible way to achieve this.

Some use cases for keyboard interactions that replace mouse controls may be:

Ability to tab through windows in an app

Ability to tab, arrow through controls in a tool, manipulate dropdowns, toggles, edit boxes, and continue to tab or arrow through the tool

Ability to tab or arrow through a dropdown

Ability to tab or arrow through a grid, then manipulate rows in the grid with a different command

Ability to dictate when to keyboard navigate through a single window or tool or all tools in the app

Ability to have keyboard commands that represent commonly used operations such as save, export to file

Ability to simulate a right click event

Same question posed to these requirements. These features are all related to influencing the platform level TabIndexing priorities. On WinUI you'd achieve this through TabIndex/TabStop and then on Catalyst by tapping into the Responder chain. Trying to influence the Tabbing at the level of generally listening to the keyboard feels incredibly difficult because you'd have to disable the platform Tab features and then keep track of your focus and order hierarchies. Similarly, if you wanted to use accelerators, you'd just attach Tab as the accelerator at different points in the hierarchy.

I'm not completely nixing the idea of a generic keyboard class. I'd just like to understand all the use cases because the requests as outlined in this issue feel like they fit better with accelerators, so I'd just like to prioritize for impact. If anyone has a WinUI project using Key events where those are the only way to achieve a given outcome that would also be helpful.

Mouse events

And then with Mouse I'm curious where PointerGestureRecognizer is falling short? The main use case I could see us expanding is adding "Pressed" and "Released" https://github.com/dotnet/maui/issues/13346

Where does PointerGestureRecognizer + TapGestureRecognizer fall short? With those two gestures you can track hovering, clicking (right clicking). You can use PanGestureRecognizer to listen to Pressed and Released (though there are a couple issues we need to fix).

In .NET MAUI, a mouseover event can be useful in scenarios where the user is interacting with the app on a device with a pointing device, such as a mouse or a trackpad. The mouseover event is triggered when the user moves the mouse pointer over an element, and it provides an opportunity for the app to provide visual feedback or additional information about the element.

For example, a mouseover event can be used to highlight a button or provide a tooltip with additional information about the button's function. This can help improve the user experience by providing more information and making it easier for the user to interact with the app.

While mouseover events are not as relevant on touch-enabled devices, they can still be useful in certain scenarios, such as when the user is using a stylus or when the app is being used in a desktop environment. Therefore, including a mouseover event in .NET MAUI can help make the app more accessible and improve the overall user experience.

In Conclusion this issue needs to be addressed ASAP.

AFAIK all of this is achievable with a PointerGestureRecognizer

minze-it commented 1 year ago

We have two use cases that would be way easier to implement with a keyboard class:

Tab through a page containing multiple input fields. One input field is a date/time picker in which users expect to enter the value directly by pressing the numbers on the keyboard.
Tab through a page containing multiple input fields. One input field is a combobox with autocomplete functionality. The user expects to select the item by typing its string representation.

Maybe i'm not familiar enough in the usage of accelerators but i don't see how to achieve that behaviour. And it is not an easy task to explain users "Yeah, you where able to type your date in that box back on windows 1995, but someone considered this to be not useful... so, you have to click the arrow buttons for now."

PureWeen commented 1 year ago

One input field is a combobox with autocomplete functionality. The user expects to select the item by typing its string representation.

One input field is a combobox with autocomplete functionality. The user expects to select the item by typing its string representation.

Can you tell me more about these comboboxes? Are these custom controls you've built? Are you wanting to layer additional behavior on our controls? Are you wanting to change the behavior of existing controls?

Tab through a page containing multiple input fields.

Can you expand on this scenario as well? You can already tab between input fields without any need to implement this yourself. The main feature we're missing around this is the ability to influence tab ordering and scope. https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.controls.control.tabindex?view=winrt-22621 https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.controls.control.tabnavigation?view=winrt-22621

Even with TabIndex, accessibility guidelines typically are very against influencing TabIndex's. The UI should be designed in such a way that everything flows logically enough for the operating to determine the navigation order for you.

minze-it commented 1 year ago

The tabbing through the page is not the problem. That works as expected by default. The issue comes with the input controls. We tried various for each data type so lets take the combobox for example: maui comes with the Picker. But the Picker has no auto completion / filtering. So its impractical when there are many items to choose from. We switched to Telerik RadComboBox. That has an auto completion feature, which is good - as long as you do not really assume using it. And that's up to the way they implemented the auto complete. The use a simple text entry to enable typing an input string. (Due to the lack of a KeyPressedEvent on the View class). As the value changes they filter the items displayed in the RadPopup. But as the text entry is the focused control they have no possibility to respond to the Enter/Return key or the arrow keys to set / select the item. The only workaround is to drop the maui-control at that point and include the WinUi component for the windows build... Having simple keyboard events on the View class would fix that.

GuildOfCalamity commented 1 year ago

@PureWeen Not sure why this needs to be debated; no one want to have to setup a PointerGestureRecognizer for a simple OnMouseOver event which should be natural to the framework out of the box. MAUI offers Windows support does it not? With that support should come all of the underpinnings inherent to that platform; not just mouse and keyboard, but window events including movement, resizing, OnCreated, OnDestroyed, etc.

PureWeen commented 1 year ago

@PureWeen Not sure why this needs to be debated; no one want to have to setup a PointerGestureRecognizer for a simple OnMouseOver event which should be natural to the framework out of the box. MAUI offers Windows support does it not? With that support should come all of the underpinnings inherent to that platform; not just mouse and keyboard, but

Changing the entire Gesture Recognizer system to match the Windows event system is a different conversation. PointerGestureRecognizers exist and they work for all mouse hover events on every platform. Spending the time it would take to change this seems counterproductive to prioritize ahead of quality and new API work.

window events including movement, resizing, OnCreated, OnDestroyed, etc.

These events are already all available on Window. https://learn.microsoft.com/en-us/dotnet/api/microsoft.maui.controls.window?view=net-maui-7.0#events

GuildOfCalamity commented 1 year ago

@PureWeen Sorry, for the window events I was referring more to the ability to move and resize.

PureWeen commented 1 year ago

The tabbing through the page is not the problem. That works as expected by default. The issue comes with the input controls. We tried various for each data type so lets take the combobox for example: maui comes with the Picker. But the Picker has no auto completion / filtering. So its impractical when there are many items to choose from. We switched to Telerik RadComboBox. That has an auto completion feature, which is good - as long as you do not really assume using it. And that's up to the way they implemented the auto complete. The use a simple text entry to enable typing an input string. (Due to the lack of a KeyPressedEvent on the View class). As the value changes they filter the items displayed in the RadPopup. But as the text entry is the focused control they have no possibility to respond to the Enter/Return key or the arrow keys to set / select the item. The only workaround is to drop the maui-control at that point and include the WinUi component for the windows build... Having simple keyboard events on the View class would fix that.

Have you brought these issues up to Telerik? This is another area where I worry that introducing keyboard events wouldn't be completely successful. For example, there's no guarantee that a global keyboard manager would be able to capture keyboard events. Maybe the Telerik control would handle them? I realize WPF has a global keyboard mechanism but does WinUI? When I looked into WinUI the keyboard events were all control based so you're still at the whim of event bubbling mechanics. Technically accelerators could be used to achieve the same behavior really. It's just a declarative mechanism opposed to a reactive mechanism. You'd add arrow key accelerators to the Telerik control, or to the layout control the Telerik control is part of, and then perform actions once those happen. Accelerators are nice because they give the operating system context. They tell the OS "Hey if the users hits this combination of keys a thing will happen" that information can then be propagated out to readers.

As a side note, it's fairly trivial to wire into the platform events on a native WinUI control. You can just grab the PlatformView off the control once it's been created.

control.Handler.PlatformView.KeyPressed (etc..)

PureWeen commented 1 year ago

@PureWeen Sorry, for the window events I was referring more to the ability to move and resize.

Those exist as well https://learn.microsoft.com/en-us/dotnet/api/microsoft.maui.controls.window?view=net-maui-7.0#properties

You can set the X,Y/Min/max/width/height and it'll do things.

It's a little squirrelly on catalyst because of OS limitations but should work pretty well on WinUI

minze-it commented 1 year ago

Technically accelerators could be used to achieve the same behavior really. It's just a declarative mechanism opposed to a reactive mechanism. You'd add arrow key accelerators to the Telerik control, or to the layout control the Telerik control is part of, and then perform actions once those happen. Accelerators are nice because they give the operating system context.

Can you paste an example for that? I couldn't find anything in the docs...

STIGsolution commented 1 year ago

My interest here is in mouse pointer actions. I'm trying to put together a traditional splitter to place between two columns in a grid layout. I want to be able to have a line splitting them and be able to click and drag it to change the width of the columns. So far I've been able to create this by using GestureRecognizers to replicate the click and drag action on the line. From there I was going to resize the columns based on the position of the mouse pointer on completion of the drag. This doesn't work though because I have no way to read the position of the mouse or the dragged object once I complete the drag.

PureWeen commented 1 year ago

My interest here is in mouse pointer actions. I'm trying to put together a traditional splitter to place between two columns in a grid layout. I want to be able to have a line splitting them and be able to click and drag it to change the width of the columns. So far I've been able to create this by using GestureRecognizers to replicate the click and drag action on the line. From there I was going to resize the columns based on the position of the mouse pointer on completion of the drag. This doesn't work though because I have no way to read the position of the mouse or the dragged object once I complete the drag.

Would this issue address everything you need? https://github.com/dotnet/maui/issues/13346

PureWeen commented 1 year ago

Technically accelerators could be used to achieve the same behavior really. It's just a declarative mechanism opposed to a reactive mechanism. You'd add arrow key accelerators to the Telerik control, or to the layout control the Telerik control is part of, and then perform actions once those happen. Accelerators are nice because they give the operating system context.

Can you paste an example for that? I couldn't find anything in the docs...

accelerators were just enabled in .NET 8 and currently are only part of MenuItem (so MenuBar and ContextFlyout), we're looking at applying Accelerators to everything vs exposing a generic keyboard AP. Just getting to know y'all scenarios here to make sure we address this in the most correct way.

As a side note, it's fairly trivial to wire into the Accelerator events on a native WinUI control. You can just grab the PlatformView off the control once it's been created.

https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.uielement.keyboardaccelerators?view=winrt-22621

STIGsolution commented 1 year ago

@PureWeen Thanks! #13346 would be great. I believe I could use e.GetPosition((View)sender); off the PointerReleased event to grab the position. I just subscribed to that as well.

minze-it commented 1 year ago

accelerators were just enabled in .NET 8 and currently are only part of MenuItem (so MenuBar and ContextFlyout), we're looking at applying Accelerators to everything vs exposing a generic keyboard AP. Just getting to know y'all scenarios here to make sure we address this in the most correct way.

Okay, sounds interesting - i'll try check that in a preview version soon...

As a side note, it's fairly trivial to wire into the Accelerator events on a native WinUI control. You can just grab the PlatformView off the control once it's been created.

I know that but until now i was not able to get the native textbox for the dynamically generated text entry inside that telerik control. It's not available by the documented api and seems not accessible through the view tree.

PureWeen commented 1 year ago

I know that but until now i was not able to get the native textbox for the dynamically generated text entry inside that telerik control. It's not available by the documented api and seems not accessible through the view tree.

You can subscribe at a level above the Telerik control. For example, if you add accelerators or subscribe to Key events on the layout the Telerik Control is inside those events should fire. If they don't fire that means the Telerik control is absorbing them and that's something that would probably need to be addressed Telerik side and wouldn't really even be fixed by adding a global keyboard class.

PureWeen commented 1 year ago

Duplicate of #12004

I've created specs for the various requests and conversations in this issue.
We're using #12004 to keep all related keyboard specs organized.

https://github.com/dotnet/maui/issues/16216 https://github.com/dotnet/maui/issues/13346 https://github.com/dotnet/maui/issues/16202 https://github.com/dotnet/maui/issues/16214 https://github.com/dotnet/maui/issues/16217 https://github.com/dotnet/maui/issues/16218

If there is a spec/scenario I've missed, please let me know. If you have thoughts about any of the specs please go into said spec and let me know.

samhouts commented 1 year ago

Duplicate of #12004

dotnet / maui