Open tkkcc opened 5 months ago
we use git protocol to do this for its atomic multi-file sync and free hosting. jgit is very small, total dex 1M, while git2 need 6.1M * 2 3.3M * 4 with vendored ssl, and we have to disable ssl check.
another lib to try is gitoxide
should stick with shizuku and libsu. These two share common functionality (the shell user permission).
benefits:
drawbacks:
should stick with wasm, benefits:
drawbacks:
market / billing / online device limitation
different from ascript: total free if the plugin doesn't use our server. free to make standalone app. 1% billing for using our service.
different from lanren: better local ocr and captcha bypass
we choose the native way currently, for more freedom
Attempt to implement dev tool for GameBot, support connect to multiple devices, color/img picker in fetched screenshot, node inspect in fetched node tree, OCR and object detection test. So there must be a server on mobile, and user use a local front end to communicate. Front can be gui or web browser. It must support win, linux and macos. Similar debug tools exist in https://ascript.cn/, https://www.nspirit.cn/, and http://lrappsoft.com/.
plugin entry is future, execute by tokio::spawn, so plugin is in same or different threads, and in same process. we force plugin developer to use async rust, but it's not nice, game bot doesn't need much concurrency. the ui part may need async runtime, or not.
same as the thread way, we need dynamic loading, and has memory leak possibility.
plugin entry is block function, execute by thread::spawn or tokio::spawn_blocking, so plugin is in separate thread in same process.
to make plugin cancelable(stop/restart by user or plugin it self), for sync api, we need to pass cancel_token and insert check in critical api. for async api, plugin entry can use tokio::select! with cancel_token.
we must do dynamic loading in rust side, use rust libloading or dlopen2. because android's System.load and System.loadLibrary can't reload library without app process restart(so all plugins restart)
on plugin reload/restart, there could be leaked resources, even leaked threads
the only clean way to update and reload a library is process restart. otherwise static in library will leak.
need to compile with ndk, extra environment requirement for plugin development.
plugin run in separate process. we can run command in kotlin or rust or startActivity with different process. but the last way consumes more memory(compose ui activity need 100MB).
we don't need dynamic loading. we can easily start and stop process, and restart without memory leak possibility (assume we don't leak memory in our host).
plugin developers don't need to download ndk source, and cargo-ndk, only rust.
need to compile with ndk.
we need more time for ipc, binder or socket or pipe or shared memory? tarpc with bincode over unix domain socket need 50ms~100ms to transfer 2880x1620x4 bytes. ndk shared memory only available since android 8.0
host app may not need jni and rust, all in kotlin seems ok.
currently we use UiAutomation.takeScreenshot, it works with logical size and dpi change, performance is close to MediaProjection+ImageReader and better than looping screencap.
Why not virtualDisplay+MediaCodec like scrcpy: need more time to encode and decode on ByteBuffer, lossless MediaFormat is not availeble on all devices. need preset bitrate.
Why not virtualDisplay+MediaRecorder: it directly writes to output, need to decode frames
Why not virtualDisplay+SurfaceTexture+glReadPixels: it's complicated. need to copy from gpu to cpu finally, as we don't know how to do template matching with GLES api.
Why not virtualDisplay+ImageReader: createVirtualDisplay need hidden api, to avoid front service notification and request permission. scrcpy use two hidden methods and FakeContext to create virtualDisplay, while UiAutomation way is simple and not relies much on hidden api. there is memory leak on surfaceflinger on some x86 emulators: avd <10, genymotion <10, but not on leidian 7&9. the leak is observable when using scrcpy on screens with fast changing content, like stopwatch.
Why not UIAutomation: on genymotion 7, the cpu usage is higher than ImageReader way(15% vs 21%), as it's working more with software Bitmap, while ImageReader way only need buffer copy(gpu2cpu + cpu2cpu). on genymotion 11~13 and my linux, UiAutomaiton way has fast memory leak(visible on host htop), but not on any other devices: avd 15 on linux, mumu 12, genymotion 13 on win, chinac 7. the leak happens on hardware Bitmap copy to software. UIAutomation way is bit slower than displayProjection.
phenomenon:
trying the virtualDisplay+ImageReader way, but after fetching new screenshots for several minutes, there is no more. it's even magical that if we allocate bitmap or bytebuffer during fetching, i can only get 1 or 2 screenshots.
attempts and thought:
final solution:
we are not holding reference to display, so it's can be gc and stops producing new screenshots. but it works for a while! the more we allocating, the faster it getting gc.
we should recheck scrcpy's code. it holding ref correctly. we are doing premature optimization!
GameBot is designed to work for years without human operations. GameBot should be self upgradable, because mobile games iterating fast. GameBot relies on AccessibilityService and MediaProjection, and use hidden api via HiddenApiRefinePlugin, and gain permission via Shizuku and libsu
plugin / hot update / hot fix / code push / dynamic loading / scripting
plain DexClassLoader/PathClassLoader
host apk --DexClassLoader/PathClassLoader--> plugin apk/jar/dex
compile andorid library into dex and load via DexClassLoader, the drawback is all dependencies must be bundled into host apk, library can't add new dependencies, and host apk almost can't use r8. library's resource can't be simply load, but kotlin i18n solutions can be used, like lyricist. fat jar solutions may help, like shadow
compile andrlid application into apk and load via PathClassLoader, this is what mediabox does. plugin can use new dependencies, but r8 almost unusable.
Shadow / Tinker
hacky and complicated, may be try BlackShadow
above two solutions violate google play policy:
flutter
shorebird
react native
react-native-code-push
above two solutions is complicated as GameBot need deep integration with android system api, so we need to export java/kotlin function into dart and js. we also need to ship their runtime library. can use rich ui component. flutter doesn't support x86. react native does't support 120fps
js embedded
rhino, used in autojs and it's forks nodejs, used in autojs boa quickjs deno
python embedded
jython chaquopy rustpython, compilable but runtime exception
lua embedded luajava mlua
wasm embedded wasmtime wasmer
above solutions are simpler and safer, but we need to export java/kotlin function into vm, if using compose instead of view, we need to implement some kind of serializable data structure. we need to transfer image byte into vm efficiently. first three solutions have zero compile time, but need to struggle with dynamic typing. for wasm, we can use rust, but have to drop support for 32 bit platforms.
lib size for x86_64:
simplar projects
kotlin/java: FGA, granblue, RobotHelper js: hamibot, autojs6, autox, 自动精灵, EasyClick, 冰狐, aibote, 云控, aiwork lua: 懒人精灵, 节点精灵, 触动精灵, 触摸精灵, 积木编程, 鱼叉助手, 飞天助手, 叉叉助手, autolua python: ascript, aibote, 小派精灵 VBScript/Q: 按键精灵 rpa: 影刀, UiBot, ...