[Feature]: multi-dimensional keymapping

[FlyMeToTheMoonAndLetMePlayAmongTheStars]

Is your feature request related to a problem?

Example game: Genshin Impact

Current problems/limitations: the existing keymap is strictly static and 2-dimensional, as in you can interact with it in x- and y- dimensions such as by placing buttons relative to the x- and y- axis of the window. Now this is great and a straightforward solution to emulating touch input from an actual touchscreen device, however, this also significantly constrains ease of use due to the buttons being static on a single layer of the keymap and being fixed a single x,y coordinate.

Impacts to the gameplay experience: Games have dynamic interface elements, often in different menus. The inherent limitation with a 2D static keymap is you will have to constantly toggle the cursor to properly select menu items that may be significantly different from each other depending on the context.

Existing workarounds/attempted fixes: A way I've tried to solve the aforementioned issues is adding additional buttons in the keymap to account for the more consistent menu elements, but the limitations of this is that the user will have to memorize all the extra keys and which UI element it corresponds to. Alternatively, I've tried mapping the same key input to multiple different locations, but this comes with its own problems such as when you press the key it disables camera control, possibly due to the game recognizing this action as an attempt to zoom in or out.

Describe the solution you'd like

Proposed solution: add a 3rd z-dimension to the keymapping editor and keymapping interaction to allow for the existence of multi-layer keymaps where specific keys can trigger contextual buttons on a separate layer that do not conflict with any other buttons of the same key value on other layers. Additionally, allow for the addition of multiple contextual layer exit trigger keys, which when any valid keys are pressed will return to the 'base' keymapping layer. The different layers can also be cycled through via specific keyboard shortcuts without having to press specific trigger keys. One layer created on top of the base layer can also nest its own layers to cover more diverse in-game menus, pop-ups, etc. On top of having certain keys that can exit the contextual layer, maybe it can also be based on a set number of mouse clicks that will return to the base layer and enable keymapping/hide cursor.

Scenario A: Domain menu F can be mapped as a trigger key to activate another keymap layer (or another key to avoid potential issues). When the player presses this to see the domain menu, X and esc will now be mapped to the button on the top right corner, i will mapped to the domain details button below it, LMB can be mapped to the co-op matching button at the bottom right, and RMB can be mapped to the start button. These keys can be mapped as trigger keys that exit the contextual keymap layer or serve as triggers for more layers. Additionally, numbers 1 to 4 can be mapped as non-trigger keys so the player can select different domain levels. This solves the need to use the cursor each time as well as the need to implement numerous additional buttons on a single keymap layer to account for all this, coding the keys to match their UI element names will also be more intuitive to use.

Let's say for example the user presses LMB to use matchmaking, this key can trigger another layer on top of the previous one, where the user can now press layer exit keys: LMB or esc to cancel or RMB to confirm. These keys can be individually set to exit to the base layer or another layer. If the user cancels, it should return to the prev layer, if confirms, should return to the base layer as they will enter a co-op instance and normal exploration/combat controls will now be on screen.

Scenario B: quest menu Here on a separate layer from the base, N can be mapped to the navigate button (which could also be the pointer navigation key on the base layer), and S can be for the story quests button. And of course X and/or esc to exit and return to base layer.

Scenario C: instrument gadgets Another key such as i can be mapped on the base keymap layer in addition to G or Z (this is to prevent issues/make distinction between using regular gadgets vs. instrument gadgets that bring up a whole new screen with note buttons). In this layer, the PC equivalent controls can be better emulated by binding Q to U to the top row of instrument buttons, A to J to the middle, and Z to M to the bottom. To restore base layer controls, the user in this case can press esc as the trigger key.

Anything else?

I've not yet thought about how this might actually look to the user. Perhaps another feature such as visible keymap hints should exist in conjunction with a multi-layered keymapping system to make things less obscure.

For layer exit keys that return to the previous or base layer, maybe it can also be triggered by a set amount of mouse clicks that will automatically lock keymapping

Maybe it could work and look something like this:

Some potential problems and limitations with this solution is that multiple different keys may have to be mapped to the main engage interaction "F key" area. Since the same key position is also used to pick up drops (though this can be changed to a different location to pick up all drops), and interact with certain things like open treasure chest, talk to NPCs. Another thing is that the keymap layer may go out of sync for unpredictable in-game scenarios that disrupt the typical sequential nature of interactions, one example is in Genshin co-op you could be using the musical instrument layer, but a teammate triggers nearby combat (or use Albedo's pedestal) which cancels you out of the instrument screen. In this case the layer will be out of sync because it cannot restore itself to the base layer. Some ways around this would be to press keyboard shortcuts (maybe cmd + 1 to 9) to quickly switch between layers, and have the keymap hint UI active with the layer exit keys better differentiated in terms of visuals so in case something like this happens the user can manually press the exit key to restore normal combat/movement keymaps. I think having on-screen keymap hints would be vital to this feature and probably should be implemented prior or in-tandem to this.

Issue Language

• [X] Yes my issue is written in English

[FlyMeToTheMoonAndLetMePlayAmongTheStars]

One additional keymapping feature that might work well with this multi-layered keymapping idea is allowing the creation of certain spots that: when clicked on while the cursor is released, will automatically lock the cursor. This would work well for instances like the inventory in Genshin, where no amount of keymapping we make can possibly account for everything the user might want to do in the inventory.

Imagine this scenario: the opening the inventory is mapped to a cursor release key B (see https://github.com/PlayCover/PlayTools/issues/61 for more info on this idea), the user presses this key and their cursor is automatically released so they can click and scroll whatever while in the inventory, this release key also links to another layer containing a cursor lock area at the top right corner (where the X is located for closing the inventory). After clicking within the valid circular area, the cursor locks itself and the player can resume using normal comat/exploration controls without needing to manually press the option key. The goal is to try and make it as seamless and as natural as possible.

This trigger area can be a fixed size or maybe even adjustable down the road.

I didn't feel the need to create another post for this since it's more of an ancillary thing to the multi-layered approach.

[FlyMeToTheMoonAndLetMePlayAmongTheStars]

Two additional interactions I thought of that could support this idea:

1. Have an option toggle where users can enable keymap pressed feedback. Like when you press on a key it briefly flashes the keymap circle matching that key located on screen. Perhaps this be toggled on/off on a per keymap button basis. (Not sure how this might impact performance, if any)
2. In addition to the idea above, maybe could allow for one keymap value to be mapped to two different actions if the user holds down on the key for a few seconds. For example: in Genshin you could map the F key to both interact with NPCs and certain things, but also give it a function where holding the F key down for 3 seconds triggers some other keymap layer. I think this might help with addressing the issue of needing one keymap value (most notably F) to account for multiple different game UI interactions.

[FlyMeToTheMoonAndLetMePlayAmongTheStars]

Another interesting idea is being able to create sequential keymap actions, like if one key is pressed after another (different key values ofc) within the next few seconds, it could trigger another keymap layer.

For example you could press ctrl + 1 to tag in another char and use their burst.

Better yet, maybe even pressing a key can temporarily switch to another layer as long as the key is held, and back to the previous when the key is released. This could allow for additional layer of interaction and may even solve conflicting keys issue I mentioned previously.