On iOS Safari mobile keyboard, SDL keyboard fails to return any of :()$&@"?!'[]=\' (15 characters)

[tommythorn]

This is a really odd bug and I apologize for not having a tiny repro, but you can see this by going to https://tommythorn.github.io/virtual-nascom/mobile.html tapping Reset and trying to type ( ) $ or & (other keys are fine).

Verified on an iPhone running iOS 15.0.2. Source was compiled with Emscripten 2.0.31-git as installed with Mac Homebrew.

Note, it works fine on all the desktop browsers I've tried, it only the iPhone mobile keyboard (I haven't yet tried a bluetooth keyboard).

[tommythorn]

Here the complete list of issues I found with the iOS mobile keyboard (US layout) and the events that gets returned:

: scancode 51 sym 59 = ';' mod 0
( scancode 38 sym 57 = '9' mod 0
) scancode 39 sym 48 = '0' mod 0
$ scancode 33 sym 52 = '4' mod 0
& scancode 36 sym 55 = '7' mod 0
@ scancode 31 sym 50 = '2' mod 0
" scancode 52 sym 39 = ''' mod 0
? scancode 56 sym 47 = '/' mod 0
! scancode 30 sym 49 = '1' mod 0
' scancode 52 sym 39 = ''' mod 0
[ scancode 47 sym 91 = '[' mod 1
] scancode 48 sym 93 = ']' mod 1
= scancode 46 sym 61 = '=' mod 1
\ scancode 49 sym 92 = '\' mod 1
' scancode 52 sym 39 = ''' mod 1

It looks like the problem with all of these is that the modifier should have been negated. Not sure why the issue only shows up on iOS. ADD: it not possible to work around as the scancode is the same, eg. [ and { both return scancode 47 sym 91 mod 1.

[PeterJensen]

@tommythorn I made a small(er) reproducer. It shows the info from the keydown event on both the DOM side and the SDL side. Does this still show the issue with iOS?

https://peterjensen.github.io/virtual-nascom/sdl-test.html

I added Makefile.sdl to build both the native version and the .wasm/.js version.

To build native: $ make -f Makefile.sdl To build .wasm/.js: $ make -f Makefile.sdl wasm

All the necessary files can be accessed via the gh-pages branch on: https://github.com/PeterJensen/virtual-nascom

Hope this helps

[tommythorn]

Thanks Peter that’s awesome. Yes it works great. In the left example I hit colon (:) and the right semicolon (;) (EDIT: the pictures go swapped, but no matter - the two appears as the same)

[PeterJensen]

@tommythorn So, I guess this is not really an Emscripten bug. If the DOM keydown event is the same for both, there's no way the SDL mapping can figure out how to map to the right SDL event. I'm thinking it's a problem with the DOM keydown event generation for iOS

[PeterJensen]

Oh, for the 9/( key it should work. Here you getting different scancode/sym values

[tommythorn]

pay close attention to the key from the DOM, it’s different.

Here’s the other example I meant, 0 and ):

[PeterJensen]

@tommythorn

Looks like the code that handles the DOM keydown to SDL keydown event is here: https://github.com/emscripten-ports/SDL2/blob/e11155a97bb67e5e6702a88e1799ea135bcfed0d/src/video/emscripten/SDL_emscriptenevents.c#L478

I'm guessing that the SHIFT keydown event isn't firing with the iOS mobile keyboard. Maybe a possible solution is to expand the logic in the above function to look at the .key value of the DOM event and if it matches a key that requires SHIFT to be down, then fire two SDL events; one for the shift and one for the key matching the .key character.

EDIT: Alternatively, we can intercept the keydown and keyup events in our JS code and if the userAgent is iOS/Safari, we can fire a 'pseudo' SHIFT keydown event for the characters requiring shift on a 'normal' keyboard and then fire the event for the corresponding non-shifted key

[PeterJensen]

@tommythorn

I made the keydown and keyup JS handler issue a fake shift down/up event for some of the troublesome keys. By default it doesn't do it, but if you hit the 'Enable fake shift key' button it will. Does that work on iOS?

https://peterjensen.github.io/virtual-nascom/sdl-test.html

[tommythorn]

Ah great, thanks. Sorry I've been too busy to follow up. I'm not 100% what the expected behavior is, but once you enable your fake shift, then ')' shows up with a mod: 1, but still the same scancode and sym as '0'. Without the fake shift key both have the exact same values for scancode, sym, and mod.

[PeterJensen]

If you're getting the same scancode/sym/mod values as you get on the desktop, I think this should work.

[tommythorn]

I just checked and indeed, on the desktop I get the same so you are right.

Am I understanding you correctly that you suggest a workaround rather than a fix in Emscripten?

[PeterJensen]

Yes, I don't think it's Emscripten's responsibility to account for weirdness in how mobile keyboards are dealt with by the mobile browser. For example, some of the special characters, that would require a shift keydown on a normal keyboard can be generated without use of shift on these specialized mobile keyboards; like the one shown in the screen dumps above

[tommythorn]

FWIW, there's precedence for working around iOS issue, like https://github.com/emscripten-ports/SDL2/commit/da0ede5e21187195a0d59b7db8ce42ff7092e166