Idea: implement input via monkeyrunner for low latency remote control

[clapbr]

I'm currently searching for a good solution to control both my android devices: a slower Bravia and a Shield TV Pro.

Of course we have adb input but it is slow as heck on both my devices (~2s) because it spawns a new jvm process for each sent command- it's good enough for single commands like pause or volume control but it sucks for d-pad navigation for instance. I'm disabled and have to use all my devices without any physical control so all my remotes should be exposed in some way in Home Assistant (lovelace cards, services, alexa commands etc).

For Bravia specifically we already have three different API's but even those have it's flaws (ie: actually harder to keep the server alive and responsive than adb mostly due to TVs lack of memory). If you've ever had to power cycle or reboot your TV because sony PSK API refuses to answer because it crashed you know what I mean.

So, after some github searching I've found this neat java app called ADBKeyMonkey that uses the native monkeyrunner API to forward local input (which is a different thing than monkey, usually used to launch intents/app). It just requires a normal adb connection, and it's very low latency.

I don't wanna mess with java and the official monkeyrunner python module only works with pre-historic Jython 2.5. I played with it a bit and was set on writing a tiny websocket server script as a demo but the Jython requirement still sounded bad. Luckily someone made this compatibility layer and after a bit of fiddling I'm now writing a few sample scripts in Python 3 and figuring out how to properly keep the connection alive based on ideas from the ADBKeyMonkey.

I'm a newbie coder but I hope this gives some ideas for the brighter minds, so drop your thoughts.

[JeffLIrion]

You know a lot more about this than me!

I haven't heard of people complaining about ADB being slow with Nvidia Shield devices. I have no experience with Sony Bravia TVs.

You're welcome to submit a pull request, or fork the repo and the HA component and modify them however you'd like.

[clapbr]

You know a lot more about this than me!

Trust me, I don't!

I've hacked up a simple proof of concept script with a slightly modified monkeyrunner module to see how fast this method can actually be on real devices. Keeping monkey+adb alive is tricky and I'm doing it by force now because my coding skills are very limited but it works as a demo. Results are promising, although these are only request execution times they reflect a lot of the actual latency difference.

nuc@nuc:~$ time adb shell input keyevent KEYCODE_DPAD_UP

real 0m1,364s user 0m0,006s sys 0m0,008s nuc@nuc:~$ time curl "http://127.0.0.1:5000/?key=KEYCODE_DPAD_UP" Command sucessful real 0m0,035s user 0m0,022s sys 0m0,004s

[JeffLIrion]

I took a look at this.

I don't see why a Flask server is necessary. If you want to implement monkeyrunner functionality for this package, you would need to modify https://github.com/nlitsme/PythonMonkey so that it can use an ADBPython or ADBServer instance. Both of those classes implement a shell method, and from a quick glance, that should be sufficient to run the monkeyrunner code in that repo.

I think the only file that you really need from that repo is monkeyrunner.py. So copy that into the androidtv folder, modify it so that it can use an ADBPython or ADBServer instance, instead of using the ADB class from adblib.py, and comment out or delete stuff that you don't need or haven't gotten working yet. Once that's done, you can create instances of MonkeyRunner and MonkeyDevice and call MonkeyDevice.press directly, like you did in your demo.

[stainlessray]

While I realize this thread is pretty old, I wanted to corroborate that the shield tv commands are pretty slow through the pipeline. My experience is consistent with the above statement. It's fine for single commands. Navigation is one-off ux at best.

The coding for the solution above is beyond me at the moment, but I'm more than happy to help, and test if anyone decides to work on this. I would love to help.

[JeffLIrion]

@stainlessray I don't think it's ADB that is the bottleneck. I rewrote the HA component and the backend libraries to be async, thinking maybe that was the issue, but it didn't make much difference, if any.

In a terminal, outside of HA, I used this async Python code to send 100 shell commands to a Fire TV stick: "echo x" where x went from 0 to 99, and I checked the ADB shell responses. Everything finished successfully in about a second. But with HA, I can only send about 1 command per second.

EDIT: the "echo x" shell commands ran quickly, but I tried some navigation commands and they are slow. So ADB is the bottleneck, after all.

If you want to try the async version of the integration, here it is. You have to use the Python ADB implementation; using an ADB server is not supported.

https://github.com/JeffLIrion/ha-androidtv/blob/async/custom_components/androidtv

[JeffLIrion]

I looked into this a bit.

From https://android.googlesource.com/platform/development/+/master/cmds/monkey/README.NETWORK.txt:

INITIAL SETUP
--

Setup port forwarding from a local port on your machine to a port on
the device:

$ adb forward tcp:12345 tcp:12345

Start the monkey server

$ adb shell monkey --port 12345

Now you're ready to run commands

The instructions use port 1080, but I changed it to 12345 to match what's in the linked monkeyrunner Python repo. In terms of implementing this, I think the steps are:

1. Setup port forwarding of port 12345 (or whatever) on your computer to port 12345 on the Android device.

   • If you're using an ADB server, this can be done via the forward method
   • If you're using adb-shell, I think you need to do:

   # tell the device to listen to your local port
   adb_device._service(b'reverse', b'tcp:12345;tcp:12345')
   
   # forward your local port to the device
   # TODO

   • Reference: ADB services

2. Launch monkey. This mostly amounts to sending the shell command monkey -v --script-log --port 12345 (source).

3. Send monkey commands to port 12345 on your computer (source)

[JeffLIrion]

Another idea is to convert input keyevent commands to sendevent commands (more info).

Steps:

1. Send this ADB command to your device (source):

   ( getevent ) & pid=$!; ( sleep 8 && kill -HUP $pid ) 2>/dev/null & watcher=$!; if wait $pid 2>/dev/null; then echo 'your command finished'; kill -HUP -P $watcher; wait $watcher; else echo 'your command was interrupted'; fi

2. Before that command times out (8 seconds), hit a button on your remote.

3. Convert the output to sendevent commands using the script linked in this article.

[JeffLIrion]

https://github.com/home-assistant/core/pull/35707