Incorrect keyboard mapping using a QWERTZ (de_DE) keyboard using an Android Moonlight client

FlorianWolters commented 2 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Is your issue described in the documentation?

[X] I have read the documentation

Is your issue present in the nightly release?

[X] This issue is present in the nightly release

Describe the Bug

After connecting (using direct LAN connection) via the "Moonlight" app to the "Sunshine" instance running on my Windows PC, the keyboard behaves as it is QWERTY (en_US), e.g. pressing the Z and ? ß \ keys (and some others) on my keyboard generate incorrect output. I've tested this with a text editor.

On the Windows host only one language "Germany" and one keyboard layout is "installed".
The input works properly if using a German QWERTZ (de_DE) keyboard attached to the Windows PC that is running "Sunshine".

Expected Behavior

I identify multiple approaches:

A setting force_client_keyboard is added to "Sunshine" that automatically detects the client keyboard and does not required any additional configuration (e.g. modifying keybindings).
The functionality behind the setting always_send_scancodes is updated in a way that it also works with non en_US keyboard. As it is now, deactivating this option seems to fix the key mappings, but it makes PC Gaming unusable, e.g. holding the SHIFT key does not work, using the NUM LOCK does not work as with a hardware keyboard, etc.
The documentation provides working keybindings settings for commonly used keyboard layouts. As it is now I was not able to manually map some of the incorrectly mapped keys.

Additional Context

There are multiple (closed) issues that seem to be related to this issue:

493
780
989
1392
https://github.com/loki-47-6F-64/sunshine/issues/82

I did comment on some of these. Since I couldn't come up with a working solution yet, I've created this new separate issue. My comments can be found here:

Host Operating System

Windows

Operating System Version

10.0.22631.3447

Architecture

64 bit

Sunshine commit or version

0.23.1

Package

Windows - Winget (Third Party)

GPU Type

Nvidia

GPU Model

NVIDIA GeForce RTX 4070

GPU Driver/Mesa Version

552.22

Capture Method (Linux Only)

No response

Config

min_log_level = 3
keybindings = [
    0x5a, 0x59,
    0x59, 0x5a
]
adapter_name = NVIDIA GeForce RTX 4070
audio_sink = High Definition Audio Device
fps = [30,60,120,144,240]
ping_timeout = 5000
virtual_sink = Steam Streaming Speakers
encoder = nvenc
global_prep_cmd = [{"do":"\"C:\\Windows\\System32\\cmd.exe\" /c \"D:\\Users\\florian_wolters\\Documents\\Sunshine\\command_preparations\\do.cmd\"","undo":"\"C:\\Windows\\System32\\cmd.exe\" /c \"D:\\Users\\florian_wolters\\Documents\\Sunshine\\command_preparations\\undo.cmd\"","elevated":"true"}]
key_rightalt_to_key_win = enabled
address_family = both
resolutions = [
    720x480,
    960x540,
    1080x2400,
    1280x720,
    1366x768,
    1600x900,
    1920x1080,
    2560x1440,
    3200x1800,
    3840x2160
]
gamepad = x360

Apps

No response

Relevant log output

I don't know what could be relevant for this issue.

cgutman commented 2 months ago

What version of Android is your client running?

The API for getting layout-independent key codes on Android is quite new unfortunately (Android 13 and later). This was a longstanding gap in the Android ecosystem for applications that wanted scancode-like behavior.

When operating on Android 13 or later, Moonlight will normalize scancodes as expected which should lead to the expected behavior out of the box for non-QWERTY layouts. However, on earlier Android versions, Moonlight won't be able to get the normalized scancodes for key presses, so it will tell Sunshine that it couldn't normalize the key.

When a key can't be normalized and Always Send Scancodes is on, Sunshine will just make a guess that the key was a US English keyboard and generate the corresponding scancode.

If Always Send Scancodes is off, Sunshine will just send the non-normalized key press without a scancode because it doesn't know which keyboard layout to get the corresponding scancode from. It may be possible in this case to try and query the host's current keyboard layout to find the key being pressed and generate that scancode, but this will still break in the case that the host and client keyboard layouts don't match.

moi952 commented 2 months ago

What version of Android is your client running?

The API for getting layout-independent key codes on Android is quite new unfortunately (Android 13 and later). This was a longstanding gap in the Android ecosystem for applications that wanted scancode-like behavior.

When operating on Android 13 or later, Moonlight will normalize scancodes as expected which should lead to the expected behavior out of the box for non-QWERTY layouts. However, on earlier Android versions, Moonlight won't be able to get the normalized scancodes for key presses, so it will tell Sunshine that it couldn't normalize the key.

When a key can't be normalized and Always Send Scancodes is on, Sunshine will just make a guess that the key was a US English keyboard and generate the corresponding scancode.

If Always Send Scancodes is off, Sunshine will just send the non-normalized key press without a scancode because it doesn't know which keyboard layout to get the corresponding scancode from. It may be possible in this case to try and query the host's current keyboard layout to find the key being pressed and generate that scancode, but this will still break in the case that the host and client keyboard layouts don't match.

Thanks for the explanation, I have the same problem on my nvidia shield (android 11), wouldn't it be possible to add a configuration in sunshine to force a keyboard type?

Nexus-505 commented 1 month ago

I am having the same problem that my keys are changed, is there any way to solve the problem of keyboard config?

LizardByte / Sunshine