search

GSConnect / gnome-shell-extension-gsconnect

KDE Connect implementation for GNOME
GNU General Public License v2.0
3.15k stars 255 forks source link

Remote keyboard only active when editing text #892

Open felipemarins opened 4 years ago

felipemarins commented 4 years ago

Describe the bug

Remote keyboard don't stay active even if the "Handle remote keys only when editing" option is disabled

Steps To Reproduce:

  1. Active KDE Connect Remote Keyboard on Android system settings
  2. Go to plugin's settings
  3. Disable the "Handle remote keys only when editing" option
  4. Try to use remote keyboard
  5. See that isn't active

Expected behavior

Remote keyboard should stay active even if isn't selected any text entry (If it isn't possible, sorry for misunderstand the plugin option)

Screenshots

image

Receive remote keypresses plugin options

System Details:

GSConnect environment (if applicable):

Additional Notes:

My tablet touchscreen is broken, the only way that I can control the tablet is from touching on a thin line that is working, and controlling from a keyboard with micro usb. I wanted to control from my PC, but I have to disconnect the micro usb keyboard, so I can't accept the authorization dialog for usb debugging. So I was trying to use the remote keyboard. (Sorry for any english mistake)

andyholmes commented 4 years ago

Hi, thanks for reporting.

This may be a bug in either the Android app or GSConnect. I haven't really tested this use-case, so I will try to reproduce and debug soon.

felipemarins commented 4 years ago

I don't know if it's important, but it's Android 4.4.2

andyholmes commented 4 years ago

That's a good thought (there is some code where that might matter), but I just had a look and there are no SDK-versioned guards in the Android app plugin I see.

Actually, looking briefly I don't see that the preference is actually hooked up to the plugin anywhere...

felipemarins commented 4 years ago

So it might be a bug in the Android app and not in GSConnect?

andyholmes commented 4 years ago

It seems like that may be the case. I've asked in the KDE Connect developer channel about this, and I'll update you when I hear back.

felipemarins commented 4 years ago

Ok, thanks

andyholmes commented 4 years ago

Ok, so after talking with the upstream developers, this seems to be a case of the protocol being a little ambiguous.

The short version is that Android sends us a flag (keyboardstate = true/false) letting us know if the keyboard is ready to accept input, but what it actually means is "keyboard is visible on the screen". But it won't tell us if the preference allows for keypresses when the keyboard is not visible.

So hopefully we can improve that behaviour in the long term, but in the short term I will modify GSConnect to consider that to be more of a warning, rather than an error.

felipemarins commented 4 years ago

Thank you for explaining and taking the time for check this!

andyholmes commented 4 years ago

I've just pushed a release candidate here: https://github.com/andyholmes/gnome-shell-extension-gsconnect/releases/tag/v40-rc1

You can start using that (and testing :wink:) by following the instructions in the Wiki.

felipemarins commented 4 years ago

I will try :)

felipemarins commented 4 years ago

It's working everywhere now! (But it's not working on the usb debugging authorization dialogue, that is where I wanted to work. I guess Android prevents from third party keyboards working on that :( )

felipemarins commented 4 years ago

And it doesn't work on the notifications area and on any window that overlaps another (as a dialogue or a menu. So maybe this is the cause that I can't work on the authorize dialog), but I think it may be improved on long-term.

felipemarins commented 4 years ago

I think that all these problems are things that has to be improved in the Android app, so maybe I have to report it to them instead of here.

felipemarins commented 4 years ago

I will list here some of the problems that I have noticed (if there is any that is unrelated to GSConnect, you may disconsider):

andyholmes commented 4 years ago

Well, this issue I'm going to leave open to track the "keyboardstate" protocol problem.

Those particular key issues, I'll have to poke around a bit when I have time. They could be in GSConnect, because handling keys like Esc, Enter and so on can get a bit tricky because we're trying to "steal" them from the desktop.

It could also be some keys aren't being mapped correctly by the Android app, because for example Enter and keypad Enter (the part of your keyboard like phone/calculator) aren't necessarily the same.

I just don't know yet, but feel free to keep posting your results; it saves me time :)

felipemarins commented 4 years ago

Well, today I tried to test again, but I'm not getting do it to work. (it's in the same state that was in the previous version) I will keep trying here for a while.

felipemarins commented 4 years ago

If I disable the option "Physical keyboard" on the Android settings, it works until a small number of keys is inputted. If I press some keys in the micro usb keyboard, it enables again, but after a small number of keys inputted, it disables. If I unplug the micro usb keyboard, it enables, but works the same way, it disables after a small number of keys inputted. The same happens if I pull the notifications area.

felipemarins commented 4 years ago

I don't know how I managed to do it to work yesterday.

felipemarins commented 4 years ago

If I move through the interface with the arrow keys of the micro usb, it enables again, but works the same way. I noticed that pressing Ctrl + Right causes the open application to crash.

felipemarins commented 4 years ago

I actually was able to make it work with a functionality of the Hacker's Keyboard, that opens the keyboard manually through the notifications area, so I open the Hacker's Keyboard and change to the KDE Connect Remote Keyboard while it's open. Sorry for not report it. I will try to enable without this workaround. (Hacker's Keyboard is open source, maybe the developer of the Android app can implement it)

felipemarins commented 4 years ago

Without the Hacker's Keyboard workaround, it can be enabled by selecting some text entry (as the Google app) and it's ready, it works. Please, disconsider all today's reports. (but the Ctrl + Right really causes the open app to crash)

felipemarins commented 4 years ago

All yesterday's reports are valid without the Hacker's Keyboard workaround, and the Ctrl + Right causing applications to crash too

felipemarins commented 4 years ago

It can be enabled by selecting some text entry (as the Google app) and it's ready, it works.

In my smartphone (LG K10 with Android 6.0), it didn't work. Maybe it only work if initially the micro usb keyboard was plugged, but I'm not being able to connect it to the LG K10 to test.

ferdnyc commented 4 years ago

I was going to say, if you were initially testing with Android 4.4.2, then unfortunately the odds are very good (or, bad?) that things will be worse with newer builds, if anything.

Android 8 - 10 really clamped down hard on a lot of potential security/permissions exploit vectors, and preventing alteration of any sort of system resources from third-party software (which KDE Connect of course is), when those changes can't be verified as coming directly from the user and not the software itself, was a major focus. That's how we lost access to the clipboard-mirroring functionality, it's why we'll almost certainly never be able to support any sort of device-screen remoting, and I wouldn't be surprised at all if it's the reason why this functionality ends up becoming increasingly crippled over time.

It's important to keep in mind that the difference between a microUSB keyboard and the KDE Connect remote keyboard is that the microUSB keyboard is a directly-connected hardware interface — it's for all intents and purposes a system keyboard, and its input is processed directly by the Android OS with elevated permissions compared to the input method apps.

But when GSConnect is sending keyboard events over the network to the KDE Connect app, there's no way the OS on the device side could possibly hope to verify that it's actually you, the authorized device owner, doing the typing. The possibility exists, and the potential is real (even if the likelihood is incredibly low) for those events to be fake keyboard inputs generated by an unknown (and potentially malicious) third party.

Ultimately the KDE Connect remote keyboard is just another untrusted, third-party Android input method app (a soft keyboard), so it's subject to all the limitations of every other soft keyboard. The same way you can't, say, use the on-screen keyboard to navigate the notification shade, you can't use KDE Connect's either. In fact, you can't do anything with the KDE Connect remote keyboard that you can't do with an on-screen keyboard — which is a lot.

In particular and especially, since roughly Android 7 or 8 all of the permissions dialogs are going to be off-limits to input methods for security reasons. (The same way that all browser web extensions are blocked from running on Chrome's internal about: pages, for example.) Because it's obvious how tempting that could be, for some less-than-upstanding app developer who wants to sneak a "feature" into their input method so maybe it "helps" unsuspecting users, say by auto-accepting any distracting authorization pop-ups — if something like that were possible.

A quick few experiments on my Android 9 phone seems to indicate it is not, though. The moment a system dialog or menu takes focus on my phone, the remote input session effectively goes completely dead. With v40-rc1 the remote input no longer gets disconnected like it used to with v39, but it's still completely unresponsive to any input until I dismiss the on-screen system menu/popup/interface and drop back to interacting with unprivileged user-space apps.

(If Hacker's Keyboard really is able to do an end-run around some of those protections, even on more recent/secure Android releases, then I wouldn't be at all surprised if the app eventually ends up getting pulled from Google Play temporarily, until that can be addressed by either Google or the H K developers.)

andyholmes commented 4 years ago

Good news is @nicolasfella submitted an MR yesterday, so I'll give that a test later tonight and maybe we can revert the workaround in v40-rc1.

For things like focusing the notification tray, it might be worth looking up that Android-x86 project since they could have a list of secret hotkeys that aren't well known.

felipemarins commented 4 years ago

(If Hacker's Keyboard really is able to do an end-run around some of those protections, even on more recent/secure Android releases, then I wouldn't be at all surprised if the app eventually ends up getting pulled from Google Play temporarily, until that can be addressed by either Google or the H K developers.)

I only tested on Android 4.4.2 (and the result was better without the Hacker's Keyboard), but I will test on Android 6.0 (these are the only available for me at the moment)

ferdnyc commented 4 years ago

For things like focusing the notification tray, it might be worth looking up that Android-x86 project since they could have a list of secret hotkeys that aren't well known.

Oh, my phone has that built in. But, tellingly, this list is found at Settings > General management > Language and input > Physical keyboard > Keyboard shortcuts. Not a one of 'em works from the software input apps (including KDE Connect's): Screenshot_20200716-040107_Settings