Allow using game key bindings to interact with skip / pause overlays

[ticotaco72]

when you cover skip overlay with cursor you should be able to click this not only by pressing mouse button, but keyboard button assigned to clicking hitobjects too; cause it works in that way in stable

[smoogipoo]

I brought this up with @peppy a while back and believe the consensus was against this.

[ghost]

This feels like a bug.

The interface should be consistent: all other buttons can be clicked with the keyboard mouse buttons, so the skip button should too.

Also, it's nice and convenient to be able to use the keyboard mouse buttons.

I don't understand why this is seen as a bad thing.

[smoogipoo]

"all other buttons"? such as?

We may want a "select" button or something like that which maps to a click but that is usable interface-wide. I have also personally struggled with this.

Edit: Turns out this is already a thing, just needs to be used more.

[ghost]

@smoogipoo

"all other buttons"

Huh, up to this point I always thought 'z' was literally like clicking my mouse on buttons like the big "osu!" button, the "play", "solo", buttons, but it looks like actually just pressing 'z' from the main menu takes me through those buttons.

I tested just now and couldn't click e.g. "editor" or "settings" with 'z'.

In that case, I think this is also poor behaviour. I want to be able to click all interface buttons without using mouse left-click.

[peppy]

We need to add "select" support to a lot of places clicks are supported (or consider some other automatic mapping of select to left click or similar).

[ghost]

I also wish it were possible to choose beatmaps from the list using 'z' and 'x'.

Currently when I press these keys it types into the search. I have to use my mouse buttons to click.

[peppy]

See above.

[Loreos7]

Since the lazer is an update for osu!stable and after it's going to be proposed a problem may arise as it's in header of the topic. It is really convenient to select options by pressing key 1 and key 2. Yes, you can change binds in the settings, but after lazer release it can't suddenly disappear. Perhaps it's worth adding a function in the settings that allows key 1 and key 2 to be "select" by default?

[peppy]

That is the intended direction. There will be a number of "stable compatibility" features like this, probably grouped under their own section in settings, specifically for users that prefer legacy stable behaviours.

The reason it hasn't been implemented yet is that it's a non-trivial implementation. Gameplay specific keys are per-ruleset, and also isolated to the gameplay context right now. Making this work in a sane way requires some serious thought.

[ohitstom]

i hate to do this but can we get an update on this issue? @peppy, I'm sure I'm part of the minority who has a pen with non functioning buttons and who uses a keypad.

However in stable it makes it impossible to do things like interact with the pause menu or skip the intro without putting my pen down and picking up my mouse.

More ui should accept z and x inputs if those keys aren't usable at that current moment for anything else

[peppy]

More ui should accept z and x inputs if those keys aren't usable at that current moment for anything else

So are you okay with not being able to type "z" or "x" at song select to search for a song? How about into chat? Where is the line drawn?

[peppy]

If we scope this change down to the two most asked-for scenarios (which will give us parity with stable) – skip overlay and pause/fail overlay – I can imagine two possibilities of how this can be implemented.

Method 1: Local key-to-click mapping

Here's a working non-optimised proof-of-concept of the first:

diff --git a/osu.Game/Rulesets/UI/DrawableRuleset.cs b/osu.Game/Rulesets/UI/DrawableRuleset.cs
index 4aeb3d4862..85555a1e03 100644
--- a/osu.Game/Rulesets/UI/DrawableRuleset.cs
+++ b/osu.Game/Rulesets/UI/DrawableRuleset.cs
@@ -46,11 +46,6 @@ public abstract partial class DrawableRuleset<TObject> : DrawableRuleset, IProvi
         public override event Action<JudgementResult> NewResult;
         public override event Action<JudgementResult> RevertResult;

-        /// <summary>
-        /// The selected variant.
-        /// </summary>
-        public virtual int Variant => 0;
-
         /// <summary>
         /// The key conversion input manager for this DrawableRuleset.
         /// </summary>
@@ -413,6 +408,11 @@ protected override void Dispose(bool isDisposing)
     [Cached(typeof(DrawableRuleset))]
     public abstract partial class DrawableRuleset : CompositeDrawable
     {
+        /// <summary>
+        /// The selected variant.
+        /// </summary>
+        public virtual int Variant => 0;
+
         /// <summary>
         /// Invoked when a <see cref="JudgementResult"/> has been applied by a <see cref="DrawableHitObject"/>.
         /// </summary>
diff --git a/osu.Game/Screens/Play/GameplayMenuOverlay.cs b/osu.Game/Screens/Play/GameplayMenuOverlay.cs
index 0680842891..a03abebcb3 100644
--- a/osu.Game/Screens/Play/GameplayMenuOverlay.cs
+++ b/osu.Game/Screens/Play/GameplayMenuOverlay.cs
@@ -14,6 +14,7 @@
 using osu.Framework.Input.Events;
 using osu.Framework.Localisation;
 using osu.Game.Beatmaps;
+using osu.Game.Database;
 using osu.Game.Graphics;
 using osu.Game.Graphics.Containers;
 using osu.Game.Graphics.Sprites;
@@ -22,6 +23,7 @@
 using osuTK;
 using osuTK.Graphics;
 using osu.Game.Localisation;
+using osu.Game.Rulesets.UI;

 namespace osu.Game.Screens.Play
 {
@@ -142,6 +144,31 @@ public int Retries
             }
         }

+        [Resolved]
+        private RealmAccess realm { get; set; } = null!;
+
+        [Resolved]
+        private DrawableRuleset drawableRuleset { get; set; } = null!;
+
+        protected override bool OnKeyDown(KeyDownEvent e)
+        {
+            string rulesetName = drawableRuleset.Ruleset.ShortName;
+
+            var bindings = realm.Realm.All<RealmKeyBinding>()
+                                .Where(b => b.RulesetName == rulesetName && b.Variant == drawableRuleset.Variant);
+
+            foreach (var binding in bindings)
+            {
+                if (binding.KeyCombination.IsPressed(KeyCombination.FromKey(e.Key), KeyCombinationMatchingMode.Any))
+                {
+                    Buttons.FirstOrDefault(b => b.IsHovered)?.TriggerClick();
+                    return true;
+                }
+            }
+
+            return base.OnKeyDown(e);
+        }
+
         protected override void PopIn()
         {
             this.FadeIn(TRANSITION_DURATION, Easing.In);

Caveats:

• mouse cursor doesn't animate
• buttons are activated on key down, not up.
• activates on all ruleset key binds

Method 2: Global support

This would involve adding an API to DrawableRuleset to expose actions which should map to mouse in the UI, then adding support at somewhere like OsuUserInputManager to do the mapping in a more correct way (similar to touch mapping to mouse, I guess).

I think this one might work better? Any thoughts?

[bdach]

(2) is probably better, but I do have to wonder, what an "action that should map to mouse" would be in something like catch? I guess for taiko / mania any action could be that, but for catch, I dunno? Any action again? None?

I don't like the first option due to the way it's reinventing several already-existing components ((Databased)KeyBindingContainer to be specific).

[peppy]

catch would likely be the dash key if anything.

I don't like the first option due to the way it's reinventing several already-existing components

It's feasible to make the DrawableRuleset.CreateInputManager() public and hook it up that way (is what I was originally trying for), but making it work in the multiple destinations is a bit harder. I think at least it would need a new flow to handle OnPressed without knowing the generic type of the actions.

That said, the global version is also going to have to figure that out.

[ohitstom]

More ui should accept z and x inputs if those keys aren't usable at that current moment for anything else

So are you okay with not being able to type "z" or "x" at song select to search for a song? How about into chat? Where is the line drawn?

I think following stable would be best to start, but if there was some way to see what elements the mouse is currently "hovering over" so to speak and only block inputs to search if that were the case I think that could also work without interrupting the user

no clue if that's stupid or not though, starting with parity with stable would be great though simply being able to skip intros and interact with pause menu is a must for me

maybe if there was some sort of way to know if z or x are usable on the current page and if not reallocate them to function as mouse buttons?

[smoogipoo]

I'm not sure that I want this to be a thing for all rulesets... And I'm not sure that I want clicks to be triggered at mouse positions on the screen from a keypress. I'm fairly certain this is a very special case in osu-stable which is only really applicable to (in that it matches bindings to) the osu! ruleset.

For the special case, is there a reason why the select keybinding can't already be used? Setting it to match my osu! keybind allows it to be used in the gameplay overlay screens:

I am also aware that we are in a bit of a unique position where a significant portion of the userbase plays with tablet and would like to control their game without ever clicking their pen, so is there a reason why we couldn't have a separate binding similar to the "select" one that does this? Perhaps replacing the "select" binding altogether? At least this way it wouldn't introduce the per-ruleset hackery.

[peppy]

is there a reason why the select keybinding can't already be used?

If you give it a try you'll soon see the issue. It works in some places globally but not all, creating a really inconsistent experience.

It would also require each and every user to set this up, which I'd hope to not be the case.

[smoogipoo]

It works in some places globally but not all

As I said, this is a special case. The select binding already matches very well with what's possible in osu-stable (it actually works in a few more areas than stable), and the main issue here is interaction with the skip/pause overlays (see: title of issue).

It would also require each and every user to set this up, which I'd hope to not be the case.

As we've discussed before, we'll eventually have a config migration (first-run) step. Part of that process could be to forward your osu! keybinds to Select.

Imo it's better to have it be set up explicitly than the very weird (and hidden) behaviour of using your ruleset's bindings for game-wide interactions. There are also cases where players have accidentally pressed their keybind after getting to the pause screen, only to have their session restarted, and this could not be fixed if it were done the proposed way.

Nevertheless, for as long as you want ruleset bindings to take on this role, there can never be a global "click-via-keypress" action. It cannot exist in cases like song select. I'm not sure that this could ever really work well which is why I said "I'm not sure that I want clicks to be triggered at mouse positions", but it absolutely cannot work with this implicit role.

[peppy]

I dunno, I'll test again and report back as I found enough weirdness when testing using "Select" that it looked like a dead end.

[peppy]

Ah, the most immediate thing was that the "Skip" button would still not be clickable (half of the issue here). I guess we could make "Select" work for that, although it might be a bit weird since it already has its own action?