Emscripten: support SDL_StartTextInput and ShowScreenKeyboard

[ericoporto]

Looking in SDL_video I can't quite find this implemented, and it appears it doesn't work.

Apparently three ways to go here:

• A. Some way to use the mobile OS Keyboard
• B. Implement a self rendered self managed SDL On-Screen Keyboard
• C. Add facilities so people can plug their own, with a Custom On-Screen Keyboard API

About A, quoting myself

The way this is done on Android, IIRC, is there’s a hidden text input box, and the focus is set to it and SDL picks up things from it and once we are done entering text then the focus from it goes away.

I believe the same thing has to be done on the Web/Emscripten version, because I believe there’s no other way to show a keyboard and get it’s input in the mobile browsers. Now I don’t think this is currently implemented in the current Emscripten port yet.

From discourse: https://discourse.libsdl.org/t/emscripten-on-screen-keyboard/42576

---

Notable examples

• A Unity thread about this, it appears they went with A https://forum.unity.com/threads/keyboard-input-webgl-mobile.505193/page-2
• RenPy used B, a virtual keyboard in the engine: https://github.com/renpy/renpy/pull/2336
• GameMaker apparently uses C https://manual.yoyogames.com/GameMaker_Language/GML_Reference/Game_Input/Virtual_Keys_And_Keyboards/Virtual_Keys_And_Keyboards.htm

Additional context

• SDL2 from Emscripten Ports fork original issue https://github.com/emscripten-ports/SDL2/issues/80
• https://groups.google.com/g/emscripten-discuss/c/owG1YiKKmT8
• https://twitter.com/danlucraft/status/1225763866765905922
• https://groups.google.com/g/emscripten-discuss/c/FN-8Tdxi1D8
• Qt used a virtual keyboard of their own: https://bugreports.qt.io/browse/QTBUG-70828?focusedCommentId=424928&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
• more Qt info: https://github.com/therecipe/qt/issues/830
• https://stackoverflow.com/questions/4609765/manually-triggering-the-iphone-ipad-ipod-keyboard-from-javascript
• https://stackoverflow.com/questions/6837543/show-virtual-keyboard-on-mobile-phones-in-javascript
• interesting implementation details of a version of A https://github.com/kou-yeung/WebGLInput/blob/master/Assets/WebGLSupport/WebGLInput/WebGLInput.jslib
• on making a custom keyboard: https://web.archive.org/web/20221203184949/https://ryanpcmcquen.org/code/2020/11/26/porting-a-c-game-to-webassembly.html
• terminal project with custom keyboard: https://github.com/tschak909/platotermjs

[rtrussell]

The potential difficulty I see with this is that on Android and iOS it's reasonable to assume that there is a touchscreen, and therefore for SDL_StartTextInput() to show the On Screen Keyboard (which is what happens now).

But on Emscripten you cannot make any assumption about the presence of a touchscreen: if the browser is running on a mobile device there probably will be, but if it's running on a desktop or laptop device there is probably a physical keyboard.

So should SDL_StartTextInput() display the On Screen Keyboard (which is what you would want on a mobile device) or open the Input Method Editor (which is what you would want if there's a real keyboard)?

[ericoporto]

Apparently in the past this was discussed, I added some links above on the matter. Notable is RenPy approach was to reimplement it's own keyboard in their engine (B).

In the Unity thread above it appears Unity managed to figure some way to use the system on-screen keyboard (A). It also appears that the method leveraged a html input text box but it also had differences between iOS and other systems.

on Emscripten you cannot make any assumption about the presence of a touchscreen: if the browser is running on a mobile device there probably will be, but if it's running on a desktop or laptop device there is probably a physical keyboard.

The method with the hidden input doesn't care for this as the browser will figure out what to do here. Other methods though, will require some work there.

[ericoporto]

Been experimenting a bit with the idea of having an input text field to type text that shows up and hides, but when the browser is fullscreen, the canvas element is selected for fullscreen, this means that the input field won't show up. Not sure what to do in this case.

[ericoporto]

Found someone who made this

https://github.com/PeterJensen/virtual-nascom/blob/ad344093b6df445a796ad04df035daf0b5887723/virtual-nascom.html#L211

[takase1121]

Hi, I just implemented the approach in https://github.com/PeterJensen/virtual-nascom/blob/ad344093b6df445a796ad04df035daf0b5887723/virtual-nascom.html#L211, and I want to give some observations I had when doing this.

1. Android keyboards (mostly) GBoard seems to be really messy in this aspect; it sends a mixture of InputEvent and KeyboardEvent, including keyboard events for enter and backspace, but not when editing autocorrect text. When doing that, it sends an InputEvent event with deleteContentBackward inputType. When autocorrect text is cleared, it started sending KeyboardEvent again.
2. There seems to be no way to check if the control / shift key is held without other keys being pressed. For example, ctrl will not work, but ctrl+a will send a single KeyboardEvent with ctrl:true. This might be an issue for some applications.
3. I am able to emulate SDL_SetTextInputRect() by making the hidden textarea fixed and moving it around with style.left and style.top. However, this is an imperfect solution because I can't display composition text (SDL Emscripten does not send text input events).

Note that the "autocorrect text" here seems to be different from normal IME composition (or even IME composition with similar mobile keyboards, e.g. GBoard Pinyin / Handwriting). It is committed instantly, but depending on what the user chooses, it can replace the previous word with the corrected one. I am not sure if SDL's current TextInput API can handle "deleting previous word".

My opinion is that approach C would be the most desirable here. Perhaps SDL can try to provide a default implementation with hints to get the hidden textarea (and disable it if this is not configured). The hidden textarea approach will not work when canvas is in fullscreen, so we need user drawn on-screen keyboard for that. This would be a lot like the current IME API with custom composition selection dialog.

My implementation can be found here: https://github.com/lite-xl/playground/blob/e92109646f0dfbe6111b5264181495a82a69e99a/core/connector.c#L186 https://github.com/lite-xl/playground/blob/e92109646f0dfbe6111b5264181495a82a69e99a/shell/js/main.js#L433 https://github.com/lite-xl/playground/blob/e92109646f0dfbe6111b5264181495a82a69e99a/shell/index.html#L25

[ericoporto]

The hidden textarea approach will not work when canvas is in fullscreen

Uhm, this is because we fullscreen the canvas itself, but maybe we can use a div that contains both and fullscreen that, would it work? For this we would have to drop using the Emscripten canvas API or show this issue to upstream and get their opinion. Paging @sbc100

[curiousdannii]

The VirtualKeyboard API might eventually be helpful, but for now it's Chromium only.

[curiousdannii]

I implemented my own version of approach A, and gave the code here: https://github.com/emscripten-core/emscripten/issues/21960#issuecomment-2118810416

[icculus]

I think that the VirtualKeyboard API is the only clean way forward on this; all the rest of these solutions look extremely fraught with danger to me.