Text Input for edit_string/property/et.al, virtual keyboards, and SDL

[ewmailing]

I'm banging my head on how to handle with text input via SDL & Nuklear in a way that works across all platforms. The big problem is mobile (e.g. iOS, Android) where there is no physical keyboard and a virtual on-screen keyboard must be used. However there are subtle related issues even for desktop. Even though I'm talking about SDL, I suspect this is a common problem across all libraries.

To jump to the end, I'm thinking Nuklear needs a callback for when a edit field first enters edit mode so the correct SDL (or whatever) functions can be invoked to activate the appropriate input mode.

So more specifically, SDL offers these 3 really important functions for text input. SDL_StartTextInput() SDL_StopTextInput() SDL_IsTextInputActive()

For all platforms, this will turn or off SDL's Text Input (and Text Editing) modes. This was intended to handle more complicated text scenarios where multiple keypresses may make a glyph. The current SDL adaptor for Nuklear depends on this being activated. As far as I can tell, this mode is by-default-on for most desktops, but default-off for mobile. But it is worth noting that users may toggle this for their own needs, presumably something where they only care about raw key presses, maybe suitable for standard game controls via keyboard. So this is one reason why I think Nuklear needs a callback, so SDL_StartTextInput() can be called at the right time if the rest of the app needs it off.

For platforms with virtual keyboards (i.e. mobile), SDL_StartTextInput() and SDL_StopTextInput() automatically bring up and hide the on-screen keyboard. So being able to call this at the right time is essential for getting keyboard input to work at all.

So basically, the workflow needs to be:

1. User mouse clicks or touches the editable field
2. Some callback gets invoked so SDL_StartTextInput() can be called.
3. User types in stuff and is sent properly to Nuklear (the (evt->type == SDL_TEXTINPUT) case in the current adaptor).
4. User leaves the textfield editing mode somehow (clicking/touching out).
5. Another callback gets invoked so SDL_StopTextInput() can be called as appropriate. (Depending on needs, this could be left on.)

Notes: A. It is worth noting that we must only call SDL_StartTextInput() at the beginning of the edit, and not every frame. The reason is that for virtual keyboards, there is a dismiss button which is not a regular button that sends events. It is solely responsible for dismissing the keyboard. So if we accidentally keep calling SDL_StartTextInput() every frame, every time the user tries to close the keyboard so they can try to get out the text editing mode, the keyboard will be forced back up and the user will be trapped.

B. Because the user can dismiss the keyboard manually, always calling SDL_StopTextInput() may or may not be necessarily in the final callback.

C. There is an additional tricky layout problem with on-screen keyboards. If the Nuklear edit box is under the keyboard, you can't see what you are typing. Maybe there is a way the 2 callbacks can be used to either move the field so it is visible, or maybe easier, introduce a pop-over that displays/echoes the current text in the field.

I know callbacks suck, but I can't really think of any other good way of doing this. Perhaps you have some better ideas? But if not, would you be willing to support such a thing in Nuklear?

Thanks

[vurtun]

Everything what you talk about already exist, discussed here: #19 Since the first discussion is quite a long time ago and has no example I will just provide one:

int len = 0;
char buffer[256];
nk_flags event = nk_edit_string(ctx, NK_EDIT_FIELD, buffer, &len, 256, 0);
if (event & NK_EDIT_ACTIVATED) {
    SDL_StartTextInput(...);
}
if (event & NK_EDIT_DEACTIVATED) {
   SDL_StopTextInput(...);
}
if (event & NK_EDIT_ACTIVE) {
   ...
}

[ewmailing]

Thank you. That did the trick.

But any advice to what to do about nk_propertyf and friends? (Are there any other widgets that have textfields?)

Also, do you have any recommendations on how to best deal with Problem C?

[ewmailing]

Also, is there a way to set the cursor to be at the end of the string when NK_EDIT_ACTIVATED happens? Again, with Problem C, since I cannot see/interact with the textfield while the keyboard is up, I am unable to clear any pre-populated text because the cursor is at the front of the string, but my keyboard on iOS only has a backspace. In the worst case, if the cursor could be moved to the end of the string, the user can delete the existing string using backspace. But right now, since it is at the beginning, backspace has no effect so I can't remove any characters.

[ewmailing]

Sorry to pile on, but I hit one more related problem. On iOS, the backspace key event triggers extremely fast, so fast that both the key down and key up events seem to happen in the same event loop.

So using the traditional SDL pattern, like you use in your own example:

    nk_input_begin(ctx);
    while (SDL_PollEvent(&evt)) {
        if (evt.type == SDL_QUIT) goto cleanup;
        nk_sdl_handle_event(&evt);
    }
    nk_input_end(cox);

I seem to be getting both the down and up events for the backspace in the same loop. My backspaces never actually do anything, so it is my belief that Nuklear is not queuing up the events, but instead the up event overwrites the down event before Nuklear processes it, effectively causing my backspace to never happen in Nuklear.

I also tried holding down the backspace to try to split the down/up events into separate loops, but it appears that the iOS built-in key repeat is so fast, I keep getting pairs of down/up, so again, I can never get backspace to do anything.

Any ideas on how to deal with this? I don't think I can change the standard SDL PollEvent pattern to process only one event per-game-loop instead of all in SDL's queue because it would be too slow. Similarly, I don't think I can update Nuklear for every event I get in the while loop because that would be too slow too.

[ewmailing]

Quick follow up on the paired backspace keydown/keyup issue. I looked at SDL's iOS implementation for keyboard handling and realized that the pairing always happen to due to implementation requirements. In CocoaTouch, the virtual keyboard events need to be handled by a higher-level text input system. SDL has to decompose the text string differences and turn them into keydown and keyup events. Since Cocoa only gives this one general text event, SDL must submit both the keydown and keyup immediately one after the other which is why they always come in pairs. SDL is doing the right thing here and doesn't have any flexibility in the matter. So any suggestions on what to do on the Nuklear side so it doesn't lose the keydown event and ignore everything?

[vurtun]

But any advice to what to do about nk_propertyf and friends? (Are there any other widgets that have textfields?)

The only widgets using text input are properties and nk_edit_xxx. As for how to handle touch on properties I was not sure so far that properties actually would make sense to have on touch devices. The logic to differentiate between dragging, button presses and text input is quite precise and fiddly for small property widgets and I am not sure it is user friendly for touch input.

Also, do you have any recommendations on how to best deal with Problem C?

A solution for problem C I is something I don't really want to specifically solve in the library since it seems to me quite touch screen dependent. As for a specific solution would be a little bit more complicated. What I would do is either a.) create a custom touch text input UI (unlikely) or b.) temporarily (contextual) window to blend over the UI and add the text input to it. Sorry I don't have a perfect solution for this at the moment.

Also, is there a way to set the cursor to be at the end of the string when NK_EDIT_ACTIVATED happens?

I added an additional flag to optionally cursor jump to the end of the text field on activate: e69aee692283176e7d58268ce55a5c9beac1da08

[vurtun]

I seem to be getting both the down and up events for the backspace in the same loop. My backspaces never actually do anything, so it is my belief that Nuklear is not queuing up the events, but instead the up event overwrites the down event before Nuklear processes it, effectively causing my backspace to never happen in Nuklear.

Nuklear keeps track of up and down even over one frame but I did not actually check it. I hopefully fixed at least key up/down pairs in one frame with this commit.

[ewmailing]

So the iOS keyboard backspace is now working. I'm not sure if it was this or #222, but it works.

Can you elaborate on your: b.) temporarily (contextual) window to blend over the UI and add the text input to it

So when the user starts editing a text field, how would I build a new contextual window that can be seen, and yet doesn't interfere with the text field being edited?