search

openstf / minitouch

Minimal multitouch event producer for Android.
Other
639 stars 236 forks source link

Android 10 support #49

Closed histronger closed 4 years ago

histronger commented 5 years ago

Do you guys have plan to support Android 10? Since version 10 have been quite popular? Thank you!

issue-label-bot[bot] commented 5 years ago

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.81. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

aristeia commented 5 years ago

+1, this is a big blocker for me. I would love to know if anyone has any plans for how Android 10 support could be achieved. The engineers I work with believe a fundamental rearchitecturing of minitouch would be necessary due to a change in how permissions are structured for writing to IO driver files:

android/source/system/sepolicy/prebuilts/api/28.0/public/shell.te android/source/system/sepolicy/prebuilts/api/29.0/public/shell.te
allow shell input_device:dir r_dir_perms; allow shell input_device:dir r_dir_perms;
allow shell input_device:chr_file rw_file_perms; allow shell input_device:chr_file r_file_perms;
neverallow shell input_device:chr_file no_w_file_perms;
Litsvin commented 5 years ago

Hi, guys!

Any updates in plans for android 10?

pcrepieux commented 4 years ago

Hi @Litsvin , There are 2 opened PR that you might give a look at:

As libevdev cannot access input devices anymore, the suggested workaround is to rely on STFService apk to inject the touch events through the Android InputManager (minitouch in this case, simply forwards the event to STFService)

ktrothschild-liifzeng commented 4 years ago

in that case，we need to develop a jar/dex run on /data/local/tmp which executed by app_process, and open a simple server on the jar/dex, then listen touch/tap/drag events send by websocket side, and transfer those events to MotionEvents, inject then to android side.

stoefln commented 4 years ago

@pcrepieux I see that both PRs above are merged. I would like to help, but I am not sure how. Should we just add instructions on how to run STFService (since it's needed on Android 10 devices) or should we also package the service, so minitouch "just works".

koral-- commented 4 years ago

Sorry for the delay. PRs were merged and dependency releases (minicap, minitouch, APK) are in progress.

pcrepieux commented 4 years ago

@stoefln I understand your point. The primary goal was to be able to have openstf controlling Android 10 devices. It should be possible to extract the MinitouchAgent from STFService.apk and put it in its own apk (or jar). It could actually completely replace the minicap binary but I didn't want to take the risk of breaking something. I'll give it a try but not sure when ...

koral-- commented 4 years ago

All the related PRs were merged and Android 10 support is available on openstf/stf HEAD.

stoefln commented 4 years ago

I don't get why this was closed. I don't see how people could use minitouch on Android 10 the way it is documented now... Isn't there something missing guys? Or am I missing something?

koral-- commented 4 years ago

In case of minitouch alone nothing has changed. As stated in Readme you can build and run (current HEAD of master). That version is also available inside stf docker image.

Prebuilt minitouch binaries are available in minitouch-prebuilt-beta npm package. This one is not documented but also not officially supported.

stoefln commented 4 years ago

@koral-- Sorry, but I am still confused. Here is what I know: I use minicap without STFService in my application. Starting with Android 10 I realised that there is a problem which is not easily fixable. AFAIK I need to additionally run STFService, so MiniCap can forward commands to STFService in case of Android 10. But this is far from intuitive and even now, I am not completely sure if I got everything right. (You are saying nothing has changed for minitouch...huh?) If I am right, we should have at least a paragraph in the MiniTouch readme which tells people that minitouch only runs with STFService on Android 10 devices.

pcrepieux commented 4 years ago

@stoefln , Minitouch cannot support Android 10 due to a new security policy. To overcome that a fallback has been added to forward touch events to STFService.apk. Latest changes in STFService.apk aim at managing those events at the framework level. Right now if you are using minitouch (outside of openstf scope), you simply have to:

As explained here, it could be improved to ease minitouch standalone usage but the priority (at least mine) was to have it running together with openstf. Meanwhile, I agree a small paragraph could clarify a bit more, will try to do it. You could also submit a PR for this once you verified the above steps are correct (hoping they are...).

stoefln commented 4 years ago

@pcrepieux ok, basically it was clear to me already before, but now I am sure I got it. I just wondered about why this ticket was closed.. I will try to get it running and send you a PR.. Thanks

koral-- commented 4 years ago

OK now I understand, reopening.

stoefln commented 4 years ago

https://github.com/openstf/minitouch/pull/58 was merged, so I guess this can be closed as well.

man4567890 commented 4 years ago

I think the same problem is on LGE phones, 64 bit and 32 bit phones

2020-03-25T09:01:36.028Z INF/device:plugins:identity 30751 [LMXXX] Solving identity

2020-03-25T09:01:36.031Z INF/device:plugins:solo 30751 [LMXXX] Subscribing to permanent channel "LbG8mL2R19k0nvP/7sYg/tX3OnE="

2020-03-25T09:01:36.032Z INF/device:plugins:screen:stream 30751 [LMXXX] Starting WebSocket server on port 7476

2020-03-25T09:01:36.476Z INF/device:resources:minitouch 30751 [LMXXX] Installing "/usr/lib/node_modules/stf/node_modules/minitouch-prebuilt-beta/prebuilt/armeabi-v7a/bin/minitouch" as "/data/local/tmp/minitouch"

2020-03-25T09:01:36.667Z WRN/device:plugins:data 30751 [LMXXX] Unable to find device data { serial: 'LMXXX',
platform: 'Android',
manufacturer: 'LGE',
operator: '',
model: 'LM-K300',
version: '10',
abi: 'armeabi-v7a',
sdk: '29',
product: 'mdh5lm',
cpuPlatform: 'mt6765',
openGLESVersion: '3.2',
marketName: 'mdh5lm',
display:
{ id: 0,
width: 720,
height: 1520,
xdpi: 294.9670104980469,
ydpi: 311.35400390625,
fps: 61.2400016784668,
density: 2,
rotation: 0,
secure: true,
size: 5.458133099625045,
url: 'ws://10.119.102.86:7476' },
phone: { network: 'UNKNOWN' } }
2020-03-25T09:01:36.670Z INF/device:plugins:touch 30751 [LMXXX] Touch origin is top left
2020-03-25T09:01:36.670Z INF/device:plugins:touch 30751 [LMXXX] Requesting touch consumer to start
2020-03-25T09:01:36.671Z INF/device:plugins:touch 30751 [LMXXX] Launching touch service
2020-03-25T09:01:36.689Z INF/device:plugins:touch 30751 [LMXXX] Connecting to minitouch service
2020-03-25T09:01:36.745Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "open: Permission denied"
2020-03-25T09:01:36.745Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Unable to open device /dev/input/mice for inspectionopen: Permission denied"
2020-03-25T09:01:36.745Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Unable to open device /dev/input/event2 for inspectionopen: Permission denied"
2020-03-25T09:01:36.746Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Unable to open device /dev/input/event3 for inspectionopen: Permission denied"
2020-03-25T09:01:36.746Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Unable to open device /dev/input/event0 for inspectionopen: Permission denied"
2020-03-25T09:01:36.746Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Unable to open device /dev/input/event1 for inspectionUnable to find a suitable touch device"
2020-03-25T09:01:36.746Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "using Android InputManager"
2020-03-25T09:01:36.861Z INF/device:plugins:touch 30751 [LMXXX] Reading minitouch banner
2020-03-25T09:01:36.904Z INF/device:plugins:touch 30751 [LMXXX] minitouch says: "Connection established"
2020-03-25T09:01:36.911Z INF/device:plugins:vnc 30751 [LMXXX] Starting VNC server on port 7478
2020-03-25T09:01:36.913Z INF/device:plugins:browser 30751 [LMXXX] Loading browser list
2020-03-25T09:01:36.934Z INF/device:plugins:browser 30751 [LMXXX] Updating browser list
2020-03-25T09:01:36.936Z INF/device:plugins:mute 30751 [LMXXX] Will not mute master volume
2020-03-25T09:01:37.521Z INF/device:resources:minirev 30751 [LMXXX] Installing "/usr/lib/node_modules/stf/vendor/minirev/armeabi-v7a/minirev" as "/data/local/tmp/minirev"
2020-03-25T09:01:37.777Z INF/device:plugins:forward 30751 [LMXXX] Launching reverse port forwarding service
2020-03-25T09:01:37.805Z INF/device:plugins:forward 30751 [LMXXX] Connecting to reverse port forwarding service
2020-03-25T09:01:37.938Z INF/device 30751 [LMXXX] Fully operational
2020-03-25T09:02:17.948Z IMP/device:plugins:group 30751 [LMXXX] Now owned by "xxx"
2020-03-25T09:02:17.948Z INF/device:plugins:group 30751 [LMXXX] Subscribing to group channel "27wYow6vQ7KHx53MvmzFQw=="
2020-03-25T09:02:18.815Z INF/device:plugins:screen:stream 30751 [LMXXX] Setting frame producer projection to 313x583
2020-03-25T09:02:18.816Z INF/device:plugins:screen:stream 30751 [LMXXX] Requesting frame producer to start
2020-03-25T09:02:18.816Z INF/device:plugins:screen:stream 30751 [LMXXX] Launching screen service
2020-03-25T09:02:19.061Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "PID: 16104"
2020-03-25T09:02:19.062Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: Using projection 720x1520@276x583/0"
2020-03-25T09:02:19.062Z INF/device:plugins:screen:stream 30751 [LMXXX] Connecting to minicap service
2020-03-25T09:02:19.064Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:241) Creating SurfaceComposerClient"
2020-03-25T09:02:19.065Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:244) Performing SurfaceComposerClient init check"
2020-03-25T09:02:19.065Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:255) Creating virtual display"
2020-03-25T09:02:19.065Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:261) Creating buffer queue"
2020-03-25T09:02:19.108Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:264) Setting buffer options"
2020-03-25T09:02:19.108Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:268) Creating CPU consumer"
2020-03-25T09:02:19.108Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:272) Creating frame waiter"
2020-03-25T09:02:19.109Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "INFO: (external/MY_minicap/src/minicap_29.cpp:276) Publishing virtual display"
2020-03-25T09:02:19.109Z INF/device:plugins:screen:stream 30751 [LMXXX] minicap says: "Vector<> have different types (this=0xbec76160, rhs=0xbec76210)"
2020-03-25T09:02:19.395Z WRN/device:plugins:screen:stream 30751 [LMXXX] Shell keeping minicap running ended unexpectedly
2020-03-25T09:02:20.689Z INF/device:plugins:screen:stream 30751 [LMXXX] Disconnecting from minicap service
2020-03-25T09:02:20.690Z INF/device:plugins:screen:stream 30751 [LMXXX] Stopping minicap service
2020-03-25T09:02:20.691Z FTL/device:plugins:screen:stream 30751 [LMXXX] Frame producer had an error FailError: Failure: 'closed'
at /usr/lib/node_modules/stf/node_modules/adbkit/lib/adb/parser.js:183:29
at runCallback (timers.js:810:20)
at tryOnImmediate (timers.js:768:5)
at processImmediate [as _immediateCallback] (timers.js:745:5)
2020-03-25T09:02:20.691Z FTL/util:lifecycle 30751 [LMXXX] Shutting down due to fatal error
2020-03-25T09:02:20.697Z INF/provider 20200 [] Cleaning up device worker "LMK300QKAMLFVGFAB6"
2020-03-25T09:02:20.698Z ERR/provider 20200 [] Device worker "LMXXX" died with code 1
2020-03-25T09:02:20.698Z INF/provider 20200 [*] Restarting device worker "LMXXX"
2020-03-25T09:02:21.484Z INF/device:support:push 31264 [LMXXX] Sending output to "tcp://127.0.0.1:7116"
2020-03-25T09:02:21.487Z INF/device 31264 [LMXXX] Preparing device
2020-03-25T09:02:21.858Z INF/device:support:sub 31264 [LMXXX] Receiving input from "tcp://127.0.0.1:7114"
2020-03-25T09:02:21.859Z INF/device:support:sub 31264 [LMXXX] Subscribing to permanent channel "*ALL"
2020-03-25T09:02:21.976Z INF/device:support:properties 31264 [LMXXX] Loading properties
2020-03-25T09:02:22.168Z INF/device:support:sdk 31264 [LMXXX] Supports SDK 29
2020-03-25T09:02:22.169Z INF/device:support:abi 31264 [LMXXX] Supports ABIs armeabi-v7a, armeabi
2020-03-25T09:02:22.920Z INF/device:resources:minicap 31264 [LMXXX] Installing "/usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi-v7a/bin/minicap" as "/data/local/tmp/minicap"
2020-03-25T09:02:22.920Z INF/device:resources:minicap 31264 [LMXXX] Installing "/usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi-v7a/lib/android-29/minicap.so" as "/data/local/tmp/minicap.so"
2020-03-25T09:02:23.287Z INF/device:resources:service 31264 [LMXXX] Checking whether we need to install STFService
2020-03-25T09:02:23.471Z INF/device:resources:service 31264 [LMXXX] Running version check
2020-03-25T09:02:24.250Z INF/device:resources:service 31264 [LMXXX] Installing STFService
2020-03-25T09:02:27.975Z INF/device:resources:service 31264 [LMXXX] STFService up to date
2020-03-25T09:02:27.977Z INF/device:plugins:service 31264 [LMXXX] Launching agent
2020-03-25T09:02:29.228Z INF/device:plugins:service 31264 [LMXXX] Agent says: "Starting minitouch agent"
2020-03-25T09:02:29.271Z INF/device:plugins:service 31264 [LMXXX] Agent says: "Listening on @stfagent"
2020-03-25T09:02:29.277Z INF/device:plugins:service 31264 [LMXXX] Launching service
2020-03-25T09:02:29.278Z INF/device:plugins:service 31264 [LMXXX] using 'start-foreground-service' command for API 29
2020-03-25T09:02:29.282Z INF/device:plugins:service 31264 [LMXXX] Agent says: "InputClient started"
man4567890 commented 4 years ago

ooh sorry did not see the issue is closed

nanoscopic commented 4 years ago

@man4567890 I get the exact same errors; "Unable to find device data", following be a series of minitouch permission denied errors. I get it for 2 different Android 10 phones: A Essential PH-1 phone, and a Samsung Galaxy Note 9.

man4567890 commented 4 years ago

Thanks @nanoscopic , are you aware of any kind of fix for this? It's driving me insane, but I don't have the technical abilities to fix this...

nanoscopic commented 4 years ago

Thanks @nanoscopic , are you aware of any kind of fix for this? It's driving me insane, but I don't have the technical abilities to fix this...

Those "errors" are actually expected and appear even when everything is "working properly". You do have additional errors though besides the ones I mentioned. "Frame producer died" specifically should not happen and indicates that you won't get video in stf.

I can verify though that Android 10 support is working though, and this ticket is correctly closed. You should open a new ticket with your other specific errors, and then see who can help you there.

You should mention in your ticket how you setup your system, specifically which commit of the project you are using, what platform you are running on, and how you are starting everything up. Also mention what model phone you are using and the android version.

instasck commented 2 years ago

@stoefln , Minitouch cannot support Android 10 due to a new security policy. To overcome that a fallback has been added to forward touch events to STFService.apk. Latest changes in STFService.apk aim at managing those events at the framework level. Right now if you are using minitouch (outside of openstf scope), you simply have to:

As explained here, it could be improved to ease minitouch standalone usage but the priority (at least mine) was to have it running together with openstf. Meanwhile, I agree a small paragraph could clarify a bit more, will try to do it. You could also submit a PR for this once you verified the above steps are correct (hoping they are...).

as far as I know inject inputs from "InputManager" is not the same as direct touch screen. The main different is that each touch report pysicalId==0 which is detectable from app side as not a real touch HW. I want to ask- is it the same as adb input tap 0,0 image