Feature request: New action(s) - click/tap items in other apps

[sudomain]

Great app. I use a closed source app (Automate) for custom keyboard shortcuts, but if KeyMapper implements this feature I can replace Automate:

Action(s) that allows for clicking/tapping items in other apps. There can be a button to record the interaction in the other app so KeyMapper knows what to click. This will be good for apps that use Android provided buttons, switches, and menus. To implement this, I'm guessing that while KeyMapper is running in the background it will need a way to keep track of the current app activity so that custom shortcuts wont bleed between apps.

Another idea would be if KeyMapper fails to detect any clicking actions while recording the interaction, it could fall back on clicking a user designated X,Y coordinate on the screen. This might be useful for apps such as games that don't use Android provided buttons, switches, and drop menus.

[sds100]

actiontype = SCREEN_INTERACTION

performAction on accessibility node. Let them tap on a view in an app and Key Mapper will detect this. throw action error if isn't in the correct app.

• [ ] Have textinputlayouts with EditTexts and Switches inside to pick..

  These are all compulsory...

  • [ ] action (click, long click, scroll forward, scroll backward, focus, select, dismiss)
  • [ ] package name
  • [ ] UI element text optional...
  • [ ] UI element id (viewIdResourceName). Not all views have this and only show this option on API 18+.
  • [ ] UI element hinttext
  • [ ] UI element tooltipText

[sudomain]

Thank you for looking into this feature request, as well as the tap coordinates action (it's very cool). How feasible would an action that clicks a user provided/cropped .png file of an UI element be? E.g. the user gives Key Mapper a cropped screenshot of a button to tap, Key Mapper searches for it on the screen (assuming it has the accessibility service), and if found Key Mapper taps the center of it?

[sds100]

It is technically possible (not sure on the latest Android versions) but there is no elegant way to do it. Lets say the screen is just a picture, finding another picture of random resolution and accuracy in another picture isn't trivial. What is the use case for providing a cropped .png of a UI element? If the UI element is a standard UI element that subclasses View.java then the accessibility can find it with the accessibility nodes API thing.

[sudomain]

Sorry I was unclear in my last reply, I thought "UI element" was a generic term rather than specific to Android. Some usecases I can think of that don't involve Android UI elements are tapping a button on a webpage (e.g. GH notifications) or a VNC client connected to a remote host. A .png search feature could be used as a fallback when a normal Android UI isn't being used

[sds100]

Yeah, UI element is a generic term. I'm not sure which terminology to use so I'll just use "UI element". The user will have to be a bit familiar with how accessibility service nodes work to understand how this feature works. It is up to the app developers to make their custom Views/UI elements behave nicely with accessibility services, which is why UI elements in some apps can't be interacted with by an accessibility service. The standard views already behave nicely like Buttons and RecyclerViews. Good point, there are useful use-cases for this but I am very unlikely to pursue it since it isn't trivial and there isn't much demand for it.

[sudomain]

I thought I'd throw the idea out there. Thank you for the work you put into this great app.

[sds100]

Yup, thanks for the ideas