search

openstf / stf

Control and manage Android devices from your browser.
https://openstf.io
Other
13.39k stars 2.79k forks source link

KaiOS support #1063

Open pcrepieux opened 5 years ago

pcrepieux commented 5 years ago

KaiOS (https://www.kaiostech.com) is an OS targeting "smart feature phone" like the Nokia 8110 and it is based on some low level parts of Android. We've been (a coworker and I) experimenting a bit with this device and OpenSTF lately and it makes us think that adding its support is - at least partially - feasible.

User interactions

There is no touchscreen on kaios devices but a physical keyboard and a keypad for navigation. Using a libevdev based approach (like minitouch does), it is possible to emulate user interactions with those hardware buttons.

Screen capture

There is an android::ScreenshotClient available on the device that provides RGB_565 buffers (requires root privilege). With little changes to the current minicap project, it is possible to get a remote screen.

STFService

As there is no Android framework on the device, there is no mean to install an apk on KaiOS. However, using C++ protobuf library, it is possible to write a native application that would be similar to this Android service.

Demo

We started tinkering around with those ideas and ended up with a first draft which looks like this:

kaios

We are willing to continue a bit our journey with KaiOS, but before going further we'd like to have your feedbacks, and know if there is some existing guidelines to properly integrate a "new" device support on OpenSTF.

issue-label-bot[bot] commented 5 years ago

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.90. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

thinkhy commented 5 years ago

@pcrepieux I heard about Kaios before but have no chance to buy one to experience it. Recently @mrx1203 added iOS support for STF project, I think you can refer to his commit: https://github.com/mrx1203/stf/commit/155c8366759d4f8bbfc0434fa2f1e71340c3c54a

pcrepieux commented 5 years ago

@thinkhy, thanks for pointing me to @mrx1203 commit. It looks like it will definitely help 👍

sorccu commented 5 years ago

I think this is super cool and it would be great to have it in. However, my main concern is ongoing maintenance. We have neither the expertise nor the time to keep KaiOS support up to date. I would need you to take responsibility of making sure it keeps working with latest KaiOS and STF versions, and if/when you become unable to do so, to hopefully find someone else who might be able to help. Push access to relevant repositories is of course possible. If you can make at least a reasonable level of commitment to maintaining the feature, I’m open to assist in reviewing and merging the code.

pcrepieux commented 5 years ago

Thanks for this feedback. I fully understand your concerns and while we (@jmauffret and I) are pretty confident that we will be able to spend enough time on that, we are still discussing the level of commitment that would be manageable. I prefer to be cautious to ensure we'll provide a realistic answer.

I’ve just started to look at the iOS support as suggested by @thinkhy, so hopefully it will help to figure out what’s the best way to integrate kaios changes smoothly. If you don’t mind, a first step where you would review the upcoming additions we are considering on the main stf repository would be very appreciated. That will also let us ensure we have the project structure right and have a better idea how much effort the maintenance would require.

One important thing is that it has only been tested on a Nokia8110. I still have to figure out whether the different KaiOS devices in the wild have the exact same set of features. My stand is that it would anyway be useful but depending on your expectation that might be not enough (to be confirmed).

I'll keep you posted on the progress.

pcrepieux commented 5 years ago

Most of the plugins work just fine (shell, filesystem,...) but I basically need to change minicap and service (or so...). In order to reuse the android codebase as much as possible, would it be acceptable if:

 module.exports = syrup.serial()
+.define(function(options) {
+  return syrup.serial()
   .dependency(require('../../support/adb'))
-  .dependency(require('../../resources/minicap'))
+  .dependency(require(options.os === 'kaios' ? '../../resources/minicap-kaios' : '../../resources/minicap'))
   .dependency(require('../util/display'))
//.................
-  })
+  }).consume(options)
+})

Any suggestion/criticism is welcomed.

Olivia26-2020 commented 4 years ago

Hi @pcrepieux

I have tried your approach on my Nokia 8180 and build with Android AOSP 5. and compile (https://github.com/Olivia26-2020/Myrepo/blob/master/minicap_mmb29m.cpp) file against AOSP 5 it build very well but I not getting the desired result like you have shown in your image. Can you please let me know where I am doing mistake. Below are the error which I am getting while connecting with Minicap app.js file. Thanks in advance. PID: 25889 INFO: Using projection 240x320@240x320/0 INFO: (jni/minicap/JpgEncoder.cpp:64) Allocating 232452 bytes for JPG encoder INFO: (jni/minicap/minicap.cpp:464) New client connection INFO: (external/MY_minicap/minicap/src/minicap_mmb29m.cpp:135) ScreenshotClient::update() result BAD_VALUE ERROR: (external/MY_minicap/minicap/src/minicap_mmb29m.cpp:137: errno: None) ScreenshotClient::update() failed BAD_VALUE ERROR: (jni/minicap/minicap.cpp:502: errno: None) Unable to consume pending frame

(https://github.com/Olivia26-2020/Myrepo/blob/master/SurfaceComposerClient.cpp)

While building I have put logs in Android.mk file including ./external/MY_minicap/minicap/Android.mk ... external/MY_minicap/minicap 21 LOCAL_SRC_FILES PATH src/minicap_mmb29m.cpp 21 src/minicap_mmb29m.cpp

wzrf commented 4 years ago

Hi! Really interesting project, I hardly see people dig into this KaiOS system, I'm really interested in this. BTW, how is the progress right now? I can see from the demo that it is working, so I guess the STF-service.apk for KaiOS and minicap-KaiOS are all working fine? Care to share a bit?

pcrepieux commented 4 years ago

@Olivia26-2020 Are you able to get root privileges on your phone ? If so, it's worth a try. I had to run it as 'root' on my 8110.

pcrepieux commented 4 years ago

@wzrf It's still at an early stage. There is nothing like apk on KaiOS, so the agent is replaced by a native binary. The very first lines of code are available here (https://github.com/Orange-OpenSource/KaiOS-STF-Agent). There is only the GET_DISPLAY request implemented right now which is the base minimum to have stf detect the device. As you can guess there is still a lot to do

wzrf commented 4 years ago

Hi @pcrepieux

I have tried your approach on my Nokia 8180 and build with Android AOSP 5. and compile (https://github.com/Olivia26-2020/Myrepo/blob/master/minicap_mmb29m.cpp) file against AOSP 5 it build very well but I not getting the desired result like you have shown in your image. Can you please let me know where I am doing mistake. Below are the error which I am getting while connecting with Minicap app.js file. Thanks in advance. PID: 25889 INFO: Using projection 240x320@240x320/0 INFO: (jni/minicap/JpgEncoder.cpp:64) Allocating 232452 bytes for JPG encoder INFO: (jni/minicap/minicap.cpp:464) New client connection INFO: (external/MY_minicap/minicap/src/minicap_mmb29m.cpp:135) ScreenshotClient::update() result BAD_VALUE ERROR: (external/MY_minicap/minicap/src/minicap_mmb29m.cpp:137: errno: None) ScreenshotClient::update() failed BAD_VALUE ERROR: (jni/minicap/minicap.cpp:502: errno: None) Unable to consume pending frame

(https://github.com/Olivia26-2020/Myrepo/blob/master/SurfaceComposerClient.cpp)

While building I have put logs in Android.mk file including ./external/MY_minicap/minicap/Android.mk ... external/MY_minicap/minicap 21 LOCAL_SRC_FILES PATH src/minicap_mmb29m.cpp 21 src/minicap_mmb29m.cpp

Hello, I just want to ask if the minicap.so is working now? And I guess the binary minicap you are using is the original android-21/bin/minicap? Or did you build a new Binary too? I am interested in KaiOS, it would be really helpful if you can share a bit with me.

pcrepieux commented 4 years ago

@wzrf indeed only the minicap shared library needs to be modified, the minicap binary is untouched. Are you running it as root ? At the time this "proof of concept" was made, it was required. NB: as you may have noticed, this work has been put on hold for a while due to other priorities but I hope we will find the time/resources to continue.