kekekeks
commented
2 years ago The problem with gamepads is the key mapping. What button is X again?
Open maxkatz6 opened 2 years ago
kekekeks
commented
2 years ago The problem with gamepads is the key mapping. What button is X again?
maxkatz6
commented
2 years ago @kekekeks at least xbox contoller style mapping is expected. Also on windows we don't have native ps4-5 controller support anyway - it is always mapped to xbox buttons. Would like to know if it's different on macOS/Linux.
And in the end it might be an issue only if we want to have full support for gamepad input (just like MonoGame has for example). Though I just checked, they decided also to support only Xbox style buttons - https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Input/GamePadButtons.cs (which might be legacy of XNA).
And for XY focus navigation it should be clear on Windows. And for other platforms/input methods we might want to have customizable mapping service, just like we have for platform-depended hotkeys.
| Xbox/remote mapping to keyboard, quite simple, but sufficient: Keyboard | Gamepad/remote |
|---|---|
| Arrow keys | D-pad (also left stick on gamepad) |
| Spacebar | A/Select button |
| Enter | A/Select button |
| Escape | B/Back button |
maxkatz6
commented
2 years ago It seems HTML JS standard just gave up about xbox-vs-ps mapping and used index based standardization. https://developer.mozilla.org/en-US/docs/Web/API/Gamepad/mapping
And that actually makes more sense.
kekekeks
commented
2 years ago I think we should stick to the HTML JS standard since it covers more platforms and even has some VR/AR controllers for future-proofing.
Also on windows we don't have native ps4-5 controller support anyway
Ah, yes, I remember the horrors of reflashing and effectively disabling your entire bluetooth adapter so it could work with dualshock. Use Windows, they said, gaming works out of the box, they said.
On Linux and macOS those are supported natively by the built-in Bluetooth stack.
ShadowMarker789
commented
2 years ago I would like to put my hand up for work on implementing some sort of gamepad interface.
@kekekeks at least xbox contoller style mapping is expected. Also on windows we don't have native ps4-5 controller support anyway - it is always mapped to xbox buttons. Would like to know if it's different on macOS/Linux.
And in the end it might be an issue only if we want to have full support for gamepad input (just like MonoGame has for example). Though I just checked, they decided also to support only Xbox style buttons - https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Input/GamePadButtons.cs (which might be legacy of XNA).
And for XY focus navigation it should be clear on Windows. And for other platforms/input methods we might want to have customizable mapping service, just like we have for platform-depended hotkeys.
Xbox/remote mapping to keyboard, quite simple, but sufficient:
Keyboard Gamepad/remote Arrow keys D-pad (also left stick on gamepad) Spacebar A/Select button Enter A/Select button Escape B/Back button
Thinking about this, gamepads are typically polled rather than receiving events like one would with WM_* messages. Do we want these controllers always mapping to keyboard inputs? What if an app developer wants to talk to the controller specifically and not have these inputs translated into keyboard events?
I don't want to be super-formal about it, but if you're able to craft up some specifications for what the interface should look like (I can get started making something like the web-API linked above).
I would also ask how should the app developer ask for the controllers/gamepads, did we want to provide a static reference to getting the controllers like with the web API?
AvaloniaGamepadProvider.GetGamepads() returning an array of Gamepad objects matching the web API?
I could definitely get an implementation working in Windows, maybe Linux/Unix (Don't know if libudev works under MacOSX), but would not know how to do that on Android, iOS, and the Web JS/WASM.
Do note that the GDK GameInput API isn't always available on Windows 10, it was included in a newer build and may not be present in older builds. It was not available in Windows 10 IoT 2019 LTSC, it is in Windows 10 IoT 2021 LTSC.
erri120
commented
6 months ago Since handheld devices like the Steam Deck are becoming very popular, I'd be awesome to support for their hardware buttons as well. For the Steam Deck specifically, I'm guessing it appears an Xbox controller for compatibility?
maxkatz6
commented
6 months ago We don’t need full gamepad api to support steam deck, we only need dpad to be mapped on arrow keys. Not sure if steam deck does it already, normal windows doesn’t map gamepad controller like this.
NetterTyp666
commented
1 month ago ShadowMarker789 commented: "I could definitely get an implementation working in Windows..."
Well, that would be all I wish for! It would greatly improve comfort for everyone who runs Ryujinx on a HTPC. Thanks for whatever may be done here.
ShadowMarker789
commented
1 month ago ShadowMarker789 commented: "I could definitely get an implementation working in Windows..."
Well, that would be all I wish for! It would greatly improve comfort for everyone who runs Ryujinx on a HTPC. Thanks for whatever may be done here.
As it happens, I spent a while smacking my head about controller input in another project, trying to run clean minimal controller input without relying on SDL.
Here's the summary of the research and findings.
| DirectInput | XInput | Raw Input | Windows.Gaming.Input | GameInput | |
|---|---|---|---|---|---|
| Availability | Windows 2000 and above | Windows XP and above | Windows 95 and above | Windows 8.1 and above | Windows 10 and above *1 |
| Xbox Triggers | One Axis | Two Axes | One Axis | Two Axes | Two Axes |
| Limitations | One Axis for Xbox Controllers | Only works with Xbox Controllers | Annoying to use, uses the WM_INPUT window message, and only one axis for Xbox controllers. | Only provides values whilst the window is in focus. No background input. | Requires the GDK to develop applications with, the license for GameInput.h precludes redistributing the header or keeping it in source-control. |
| Header | dinput.h | xinput.h | hidpi.h, winuser.h | winrt/Windows.Gaming.Input.h | gameinput.h |
| Header file found in | Windows SDK | Windows SDK | Windows SDK | Windows SDK | Microsoft GDK |
| API Style | NanoCOM | Simple C-API | Windows Messaging and HID | WinRT classes | NanoCOM |
| Bugs? | None Known | None Known | None Known | None Known | Does not work with bluetooth controllers. |
This is a problem. If we want to support reading both triggers on Xbox controllers on Windows, we don't have a good solution for that. This has been such a problem that the work has stalled completely on that other project.
When I reached out to other projects (Such as FNA, etc) about what they do for Controller input - everyone said that they just use SDL.
So, I would like to raise questions for the maintainers - is background input desirable? How should the API look like, should applications want to enable/suppress controller UI navigation? I only know Windows APIs myself, but on Linux one would use libevdev if one didn't want to take a dependency on SDL.
Or... we could just taken a dependency on SDL and call it a day? But that's a big dependency to pull in!
I would need some guidance on what direction the project maintainers want to go into before I make any proof-of-concepts or spawn a draft PR.
Regarding GameInput on Windows, a MSFT Engineer said:
First-party Xbox One gamepads should work in GamePad mode, but only in wired mode. Bluetooth will not work.
Which makes me sad. :(
NetterTyp666
commented
1 month ago Yuzu had the option "Controller navigation" under Controls -> Advanced. This was working perfectly fine in Windows with both Xbox Controllers and Switch Pro Controllers. In both cases I had Bluetooth connection. I do not own other controllers, so I have not tried alternatives.
I'm not sure how it was implemented, it's probably XInput since that was mentioned for another option as well. However, the exact implementation might be checked in the source code and then... well, just replicated.
The d-pad should just navigate the tiles and one of the buttons should start a game. At least for me that would be all that's required. For exiting a game I simply use the utility "Xbox Controller button remapper" to bind ESC key to either the Share or Xbox button. Of course it would be perfect to have a simliar native implementation, but that would really just be a bonus.
I do not know anything about the details but could it be that XInput does not only support Xbox Controllers, but also all other controllers with similar layout (like it did work for me with Switch Pro Controller)?
It would be already a big win if those two controller types work, because that's probably what the majority of players will want to use.
ShadowMarker789
commented
1 month ago Yuzu had the option "Controller navigation" under Controls -> Advanced. This was working perfectly fine in Windows with both Xbox Controllers and Switch Pro Controllers. In both cases I had Bluetooth connection. I do not own other controllers, so I have not tried alternatives.
I'm not sure how it was implemented, it's probably XInput since that was mentioned for another option as well. However, the exact implementation might be checked in the source code and then... well, just replicated.
I do not know anything about the details but could it be that XInput does not only support Xbox Controllers, but also all other controllers with similar layout (like it did work for me with Switch Pro Controller)?
It would be already a big win if those two controller types work, because that's probably what the majority of users will want to use.
I know that controller support is difficult enough that projects will sometimes import the entirety of SDL JUST for its controller support. I'm fairly sure that's how Dolphin runs its controller support. It might just be SDL.
NetterTyp666
commented
1 month ago Not wanting to sound dumb... but what I don't get is:
Within the options I configure which controller I have. Ryujinx then recognizes it and it's working perfectly fine within a game.
Then why can't the same input style not be used within the GUI as well? Why should there be a need to implement another API for that?
ShadowMarker789
commented
1 month ago Not wanting to sound dumb... but what I don't get is:
Within the options I configure which controller I have. Ryujinx then recognizes it and it's working perfectly fine within a game.
Then why can't the same input style not be used within the GUI as well? Why should there be a need to implement another API for that?
It looks to me like Ryujinx is using SDL2 for its Gamepad driver.
https://github.com/Ryujinx/Ryujinx/tree/master/src/Ryujinx.Input.SDL2
If Ryujinx so decided, they could take controller input and inject keystrokes into the Gui application if they wanted to.
For instance, I got this working earlier in a previous version of Avalonia.
https://github.com/user-attachments/assets/d7b1a6fa-13e1-4047-840d-2eb6fd62a5c3
And, correct me if I'm wrong - this is exactly what you want, right?
NetterTyp666
commented
1 month ago More or less, yes!
The D-pad should just navigate the tiles and one of the buttons A/B/X/Y should start a game.
However, it must be ensured that also the tiles that are currently not visible on screen can be reached. So when the user navigates beyond the last tile, the tiles need to move up or down accordingly. Or as an alternative another button could to do the scrolling: for instance use D-pad to move and analog stick to scroll.
And as a bonus - but that would be real luxury - it would be great if an active emulation could be stopped just by using a controller button (e.g. Xbox Controller "Share" or Switch Pro Controller "Home") or a certain button combination.
Goal would be that for regular usage no mouse or keyboard is required. So just have Ryujinx on autostart and in full screen, then just grab a controller to navigate to a game and start/stop it.
I would not expect that everything can be done just by a controller. So when it comes to configuring something within the settings dialogues then that may still be done with a mouse.
ShadowMarker789
commented
1 month ago More or less, yes!
The D-pad should just navigate the tiles and one of the buttons A/B/X/Y should start a game.
However, it must be ensured that also the tiles that are currently not visible on screen can be reached. So when the user navigates beyond the last tile, the tiles need to move up or down accordingly. Or as an alternative another button could to do the scrolling: for instance use D-pad to move and analog stick to scroll.
And as a bonus - but that would be real luxury - it would be great if an active emulation could be stopped just by using a controller button (e.g. Xbox Controller "Share" or Switch Pro Controller "Home") or a certain button combination.
Please keep in mind that Avalonia is a general-purpose Gui solution that runs on many platforms, used by many.
So, any changes to Avalonia need to keep this in mind. The API provided by Avalonia would need to be able to be implemented cross-platform and be maintainable in the future.
Trying to keep the discussion relevant to Avalonia, the Gamepad Navigation:
That already sounds hard. If we're not leveraging a library like SDL2 then cross-platform controller input is very difficult.
NetterTyp666
commented
1 month ago That's totally understandable. But just to mention again: Yuzu was also cross-platform and had such an option implemented.
Also, cross-platform does not necessarily mean that each and every option needs to work on all platforms, right?
But of course, if it does then all the better...
Sewer56
commented
1 month ago If it's any helpful; I made a library for this just over 2 years ago: https://github.com/Sewer56/Sewer56.UI.Controller No Avalonia impl, but it has WPF, which is similar.
It's split into the following:
Core: The core library, contained in Sewer56.UI.Controller.Core.IPlatform: Abstracts a UI platform like WPF or WinForms. (e.g. Sewer56.UI.Controller.WPF).IController: Provides input support to the library. (e.g. Sewer56.UI.Controller.ReloadedInput)Live Demo (original url):
https://github.com/user-attachments/assets/47b747a9-3ee6-4469-ac42-15c99360dc7f
(The above is WPF)
[Feel free to use/study/etc. the code. If Avalonia wants to copy any of it, I'd happily also slap the MIT sticker on it]
Anyway, having previously worked on this problem; there's some other factors to take into account that may not have been brought up. I'll use my own WPF application as an example and hopefully provide some pointers.
When displaying a modal dialog, or another item that overlays the screen, you must be careful as to not make items not reachable by the mouse selectable.
In this specific example, you wouldn't want pressing the DPAD buttons to cause a selection on the tiles below the semi-transparent background.
When implementing this for WPF, I needed to perform Hit Testing. Specifically I tested each of the corners of a control and its middle to determine if they are clickable.
If that is not the case, and the control is not easily reachable by mouse, it should not be selectable.
Gamepad selection works quite differently in general compared to Tab navigation. With Tab navigation, you set a fixed order of elements, while with the D-Pad you generally expect the cursor to go in the same direction as the gamepad input.
Suppose you are at the red dot, and you press DPAD Right (Green Arrow). You need to be able to decide how to 'rank' the individual selectable items that are towards that direction.
The way I did it is this:
Score = Angle * (Distance^2 * DistanceBias)The item with the LOWEST score is selected. If no item matches, I select first item as a failsafe, but in practice that never gets triggered.
The angle in this case is between the two vectors. So if your direction is RIGHT then an angle of 90 degrees means a direction of UP or DOWN. In my case I constrained the angle to 89, but the choice here was arbitrary as to not allow it to go up/down directly. Angles above 90 will of course make you go the other direction which you don't want.
The scoring formula here is arbitrary, and it can be adjusted as needed. But you need to account for both distance and angle to determine what the user wants to select. My implementation is naturally biased a bit towards distance, but it tends to work relatively well.
Also be careful not to divide by 0 here.
Optimal interactions with controls are a tricky subject; the controls can be divided up into:
A/B [Xbox] or Cross/Circle [PlayStation])e.g. Buttons, RadioButtons, Hyperlinks etc.
Pressing A/Cross opens the buttons
Some controls like NumericUpDown(s) cannot be interacted with without some extra logic.
(Arrows here are also very misleading)
Either the user would need to press A to bring up a popup or press/hold a button to 'focus into the control'. It may also be useful to have additional bindings for faster access to these controls. In my case I had a Decrement and Increment button, bound to LB/RB (L1/R1). [You can see that in action in video clip above]
I used that Increment button to let you change the value of ComboBox(es), ToggleButton(s) and NumericUpDown(s) without expanding/opening them. In the case of NumericUpDown, it was the only way to manipulate it, as I didn't have a pop-out window for finer input.
There is also a Modifier key, to get alternative behaviour when pressing buttons on a control.
In my case, you can also use the same Modifier key to use Up/Down to increment/decrement the NumericUpDown, that's the focus into the control part.
Notably, sorting elements in a list in the video above was done by holding the modifier key, and using arrows ... which brings us to ...
When dealing with more complex items like ListItem(s) etc. sometimes you'll want to manually implement some interactions. For example here, pressing A on a row disables/enables the item. You don't really want the user to be focusing on the individual checkbox.
To support these sorts of manual interactions, I had an API called void ProcessInputs(in ControllerState state);, that gets fired after every controller state update (state is different than last).
An additional interaction that may be desired here but needs to be opt-in is closing dialog windows with B/Circle button.
Notably, in Japan the meaning of the X and O buttons on a PlayStation controller are swapped. If you press O, you have to go forward in a menu, not backwards. There may be other cases.
Users also expect to be able to navigate the menu using the left stick. That can be logically translated to simple DPAD inputs, i.e. stick pointing right should do same action as DPAD right.
I will agree with the posts above and would suggest using SDL2.
The reason for me actually isn't to do with difficulty of cross platform controller APIs; I've already written a general purpose controller manager library that lets you easily manage information from multiple APIs, rebind keys, etc. Adding evdev for Linux in addition to existing xinput and dinput modules wouldn't be much work.
The reason is actually to do with controller presets. People today expect to plug in a controller and it should 'just work', regardless of platform or controller. The expectation today is that you shouldn't have to manually bind controller buttons.
SDL2 has something called the Game Controller Database, it's rather massive; that alone, enabling simple 'plug and play' is what makes it stand out. I wish I knew about this 2 years ago.
As for how to bundle SDL2, it's generally a rather heavy dependency as a whole. If it were up to me, I'd make a Rust or a Zig project (they're both easy to cross compile with) that exposes only the slice of the controller API that Avalonia would need. This way; all of the unrelated non-controller code not used is not included in the shared library (.so, .dll etc.).
As for polling, I'd just poll at the beginning of each 'display frame'. I don't know the details of Avalonia's rendering stack, but what I essentially mean is at the same rate as the display/monitor, at the start of a new 'frame'. For efficiency, only do processing if there's a change in controller state since last.
Just some odd things I experienced implementing this in WPF.
Focus Visuals (a.k.a. selection/highlight box)
In WPF, focusing an element did not update the visuals on the control. You could only make the selection box appear by using a private API, which was called by the keyboard Tab handler.
When a user focuses a control from code, there should be a way to display the focus selection.
Styling Focus Visuals (a.k.a. selection/highlight box)
In WPF, the focus visuals were painfully hard to style, as it often delved into the territory of making entire custom control templates from scratch. Many controls had multiple focusable elements too, that were not tab stops; and did not render focus visuals. These should not be picked up by a controller.
In Avalonia, this appears to be much easier, with the selector syntax.
Anyway, these are some notes having done this in the past. Hopefully they're useful. There's a bunch of considerations in here to be made, both API and UX wise.
I think the best approach is first making sure Avalonia has the right APIs to support this sort of use case (for example, public APIs to properly set focus, indicator included, which WPF didn't have). After that, something like Controller Support could be added via an extension package like AvaloniaUI/TreeDataGrid.
Having the controller behaviour opt-in in some manner would also help us keep binary sizes small. At least personally I don't want to have code I'm not using present in NativeAOT builds.
Is your feature request related to a problem? Please describe. Controller (xbox controller or remote control) should be treated as valid user input that should be mapped into XY focus navigation. So users can use Avalonia applications using only their controllers.
Additional context UWP APIs: High level integration of controller with app navigation (similarly how it's done with keyboard input): https://docs.microsoft.com/en-us/windows/apps/design/input/gamepad-and-remote-interactions Lower level Gamepad API: https://docs.microsoft.com/en-us/uwp/api/windows.gaming.input.gamepad?view=winrt-22000
I guess it's very low priority requests, but still would be nice to have somewhere in the future. Majority of UI frameworks don't treat controllers as they should and from my knowledge it works only with UWP apps.