Displaying the Android soft keyboard causes serious rendering issues when combined with controller input

[leonstyhre]

Hi!

I have come across a very strange issue when using the Android soft keyboard (on-screen keyboard). The keyboard itself works perfectly fine, and it displays and disappears as it should. My application (ES-DE) supports both touch input using an overlay as well as controller and keyboard input. If the touch overlay is used everything works perfectly and there are no issues, but if using a physical keyboard or a controller there is a green border displayed along three sides of the screen after closing the soft keyboard, and the bottom of the screen has a number of pixels that don't get rendered at all.

Please see the attached screenshots which demonstrate the problem. Navigating back to the text input field makes the keyboard appear again, and the green borders are removed. But as soon as the keyboard is closed again the borders come back and they will remain indefinitely, requiring an application restart to fix.

I have spent a lot of time trying to figure out this issue and what I've concluded is that it has nothing to do with the keyboard itself, as commenting out the code that spawns it still makes the green border appear. What is causing the problem is when calling mSurface.requestFocus() in SDLCommandHandler(). So commenting out this line will temporarily fix the issue. However input focus is then not moved back to the main window so the application doesn't work correctly. As well switching to the home screen and back to the app introduces the green border as requestFocus() is called via onResume().

Adding mLayout.removeView(mTextEdit); to the end of the ShowTextInputTask::run() method will make the keyboard get displayed briefly, then immediately disappear and the green border is not shown. So something with the DummyEdit View is causing the problem but it's really bizarre as it only happens if you navigate using a controller or physical keyboard and not if using the touch input overlay. It's almost as if any controller input consumed by mTextEdit is introducing the problem. I guess something with the dummy/invisible DummyEdit View is somehow confusing the SDLSurface but this way beyond my knowledge of SDL to figure out.

I'm having real difficulties describing this problem as it's so strange, I hope the above makes sense. Unfortunately it's quite serious as it makes the Android soft keyboard unusable with my application as most users will probably use some kind of controller rather than the touch overlay.

Let me know if you want me to test something further to try to pinpoint the cause of the issue.

Thanks in advance!

[leonstyhre]

@slouken Would it be possible to look into resolving this? It may seem like a small issue but it's making the Android soft keyboard unusable unfortunately. I'm very happy to test things to try to resolve the problem, what I have been doing so far is turning my application inside out trying to fix it but it really seems to be an issue in SDL and I don't have enough experience with your library to pinpoint and resolve the problem there. Thanks! :)

[slouken]

What device and version of Android are you seeing this on? I'm testing with checkkeys and it's bringing up a smaller side keyboard on Android 14.

[AntTheAlchemist]

Testing it on Android 12, with game controller & physical keyboard. Working okay for me.

However, I notice in your screenshot, you have a bar at the bottom, and a down arrow, which I guess is for closing the keyboard. I'm not seeing that bar at the bottom. My keyboard is anchored to the bottom of the screen. I think this maybe related.

And I've seen that green border thing before, when fudging around with keyboard focus in Java. I was trying to make an invisible button to focus on so AdMob banners could "find" it and give keyboard focus back to SDL. It creates the green boarder when I manually give focus back to the SDL surface. The touch keyboard doesn't make this happen my end though, so something funky is going on due to the extra keyboad nav bar at the bottom of your screen. What Android device is that?

[leonstyhre]

Thanks for looking into this issue! The problem is seen across all devices and Android versions I've tested with. For example I just reproduced it on a Samsung Galaxy S23 (Android 14), a Google Pixel 4a (Android 13), a Google Pixel Tablet (Android 14) and an Ayn Odin 2 (Android 13).

As described in my original report, it only happens if you're using a physical controller and not if using touch input only on the device, and it also only happens when leaving the text input field so the Android soft keyboard gets hidden.

If you want to I can provide you with an APK for ES-DE so you can test it for yourself, it's very easy to reproduce the problem and it's also been experienced by other people than myself as it seems to be a universal issue on all Android devices when using the Android soft keyboard.

The steps required to reproduce the problem in ES-DE are:

• Connect an external controller to your device (I tested with an Xbox 360 USB controller and the built-in controller of the Ayn Odin 2)
• Start ES-DE, open the main menu, then select the UI settings menu and disable the setting "Enable virtual keyboard" almost at the bottom of the menu (this will enable the Android soft keyboard instead of ES-DE's built-in virtual keyboard)
• Go back to the main menu, then select "Other settings" and then "Game media directory" which is the second entry from the top, as this is a text input field the Android soft keyboard will be displayed
• Press the down button on the controller which will jump from the text input field to the "Save" button and which will also hide the Android soft keyboard, this will introduce the issue

[slouken]

Yes please, if you can provide an APK that would be helpful.

[leonstyhre]

@slouken As the ES-DE Android release is a paid app I don't want to share it publicly here on GitHub, but I've sent you a download link now via email!

@AntTheAlchemist I couldn't find your email address, but if you also want the APK then please send me an email and I'll reply with the download link!

[AntTheAlchemist]

@leonstyhre I've added an email address to my profile. I'll take a look, thanks.

I've made an SDL3 app which tests all the input features of SDL. The keyboard test has an option to open and close the software keyboard. Can you see if you you're having the same issues with my app while a gamepad is connected? Might give us more of a clue. https://play.google.com/store/apps/details?id=uk.co.powgames.gamecondiag&hl=en (EDIT... wrong URL given, just corrected it!)

[leonstyhre]

@AntTheAlchemist Thanks, I've sent you an email now with the APK download link.

I've also tested your app and it does not create the green border around the window when closing the Android soft keyboard, but I guess this apps works very differently than ES-DE which is rendering using OpenGL so not sure how conclusive such testing would be.

[AntTheAlchemist]

@leonstyhre I'm using SDL's 2D renderer. Are you rendering using gl functions? What are you doing java-side that might affect how the surface works?

I'm trying your apk out. What steps do I need to do to bring up the software keyboard? Do I need a game ROM to get passed the directory creation step?

[leonstyhre]

I'm using OpenGL, you can find the code here: https://gitlab.com/es-de/emulationstation-de/-/tree/master/es-core/src/renderers?ref_type=heads

I don't do anything in the Java code that could have any effect on this, it's completely unrelated to the renderer. There must be some kind of internal SDL issue causing this problem.

I described the steps a few comments above on how to trigger the issue. But yes you need to add at least one game to start ES-DE. You can just make it an empty dummy file like ROMs/megadrive/dummy.zip or similar.