[ENHANCEMENT] minicap written in pure public Android Java APIs

[sirmordred]

I have working project (for Android Q(10) and up) that captures internal audio as well as device screen on android, i dont know the current STF provides streaming internal audio from connected android device to browser app, but i hopefully think that it can be implemented i think

also as far as i read from minicap readme, it says it will be changed with more upstream solution(directly capture android screen via MediaProjection APIs)

socket streaming in android works flawlessly too as you know(based on pure java api)

i can work on this if its worthwhile what do you think, what are the future plans?

cc @sorccu @koral-- @pcrepieux

[deliberist]

I'm not sure what the maintainers plans are, hopefully one can chime in as you've CCed them. But I would like to put it out there that I desperately need Android 12 support in Device Farmer.

I'm not too concerned with how minicap implements that, but I would certainly appreciate a quick bare-bones solution even if it meant it needed incremental refactoring. I'm just not in a position to implement Android 12 support myself at the moment. So if someone already had the knowledge to do it, I'm hoping the maintainers would gladly accept any PR you might have.

[pcrepieux]

@sirmordred, the plan is indeed to use MediaProjection in some cases. The number one would be Audio which is currently not supported at all. Then, maybe for video. But for now the current solution has the advantage of avoiding a system popup. I guess some kind of hybrid approach would make sense because we could accept the popup for the audio media remotely.

That being said, apart from finding a bit of time which is not always easy, the work on Android side shouldn't be huge if there is a mean to play audio buffer within the browser without too much troubles. I don't know that part well enough to say. If we go with something like webRTC or something equivalent, that could be much harder to fit in the current stf frontend.

Ideas are welcome.

[pcrepieux]

@rbprogrammer Android 12 is supposed to be well supported. The kotlin version of minicap helped to get rid of the AOSP source code for the compilation process and is expected to work even with developper preview or beta (which was often not possible with the native minicap).

I can try to help fix your issue if you provide me more details.

[sirmordred]

@pcrepieux hmm got it, thank you so much for the explanation, and i must say that if we are able to initiate some adb commands after installing an our helper apps/apks(STFService.apk for example) into target android devices, then our minicap rewritten with android APIs will not require system popups (there are ways to avoid popups and with that solutions, my project also works silently without needing user interaction at all 👍 ) and you say its worthwhile to work, i mean rewriting minicap via java/android, i'll try to demonstrate it ASAP then

for audio yeah i dont know either frontend of STF and what it will require for adopting that audio playback feature, other core developers may chime in after we have working solution maybe

[deliberist]

@pcrepieux Android 12 support in Device Farmer sounds fantastic. But I probably forgot to mention that I'm working with the Docker images pushed to Docker Hub. The latest versions, the last time I checked, did not include a version of minicap built for Android 12.

[stoefln]

@pcrepieux you were referring to a kotlin version of MiniCap. Is this available publicly? Could not find it... I am also looking for Android 12 support for MiniCap.

[pcrepieux]

Sorry for the late response. In case it still might be useful, minicap apk code is here : https://github.com/DeviceFarmer/minicap/tree/master/experimental

[pcrepieux]

@rbprogrammer, are you referring to stf docker image ?

[deliberist]

@pcrepieux, yep. I have not checked if there were any updates to the image on Docker Hub. But yeah, I was talking about: https://hub.docker.com/r/devicefarmer/stf

[pcrepieux]

@rbprogrammer you should have a minicap.apk in the noarch directory of the minicap-prebuilt node module. STF tries a binary version first, and fallback to the apk if there is an error. You can ask STF to use the apk first with --screen-grabber parameter or the SCREEN_GRABBER environment variable (value can be minicap-apk or minicap-bin). Here is a link on the PR introducing this behavior : https://github.com/DeviceFarmer/stf/pull/262