search

appetizerio / replaykit

[DEPRECATED] Command line tools for recording, replaying and mirroring touchscreen events for Android
Apache License 2.0
245 stars 61 forks source link

How to identify application activity or element id on click #13

Closed djcm75 closed 6 years ago

djcm75 commented 6 years ago

I would like to know which element or button was clicked, I require this for a custom application. when I execute the command

./appetizer trace info mytrace.trace

I get this result {"contacts": 1, "description": "", "height": 1280, "length": 5.08226399999694, "records": 4, "valid": true, "width": 720} Is it possible to capture/display the element id or application activity that was launched ?

mingyuan-xia commented 6 years ago

Right now, replaykit only records touch points (X, Y, timestamp). It seems like we need the uiautomator to report the current widget tree and locate the widget from that with the screen position. Not sure if we can find one for this purpose. In addition, this might only work for taps. I have no idea how to deal with dragging, unconventional gestures (random ones), multi-touch gestures, etc.

djcm75 commented 6 years ago

I understand uiautomator can solve the problem but since it's an automated task, I do not want to use uiautomator to manually identify element. Is there any tool or is it possible to use monkey runner to identify elements that are clicked on the device ?

mingyuan-xia commented 6 years ago

I mean adb uiautomator dump or the AccessibilityNodeInfo it uses at low level. You can code up a script/program that capture real-time widget tree for cross-referencing. As for monkey-runner, AFAIK, 1. it is out-dated; 2. it is unstable sometimes, get stuck when connecting to a device; 3. you might not want to deal with the funny Jython when developing on top of that.