search

remote-android / redroid-doc

redroid (Remote-Android) is a multi-arch, GPU enabled, Android in Cloud solution. Track issues / docs here
4.14k stars 297 forks source link

Running Redroid on Raspberry Pi #67

Closed kebugcheckex closed 2 years ago

kebugcheckex commented 2 years ago

I'm running Redroid on Raspberry Pi 4B. The OS is Ubuntu 20.04 (arm64).

Starting the container with default param works but the graphics is pretty laggy. It turns out that the rendering is done on CPU, instead of GPU. RPi has a pretty powerful GPU but I'm wondering how I can make use of it in Redroid.

Additional information:

If this is not yet supported by Redroid, what is the path towards supporting it? Thanks.

zhouziyang commented 2 years ago

redroid only add intel / amd / virtio gpu support currently.

for other gpus, you can modify BoardConfig.mk, and rebuild redroid images. Sent from my iPhone

On Dec 10, 2021, at 11:43 AM, Xinyu Chen @.***> wrote:



I'm running Redroid on Raspberry Pi 4B. The OS is Ubuntu 20.04 (arm64).

Starting the container with default param works but the graphics is pretty laggy. It turns out that the rendering is done on CPU, instead of GPU. RPi has a pretty powerful GPU but I'm wondering how I can make use of it in Redroid.

Additional information:

If this is not yet supported by Redroid, what is the path towards supporting it? Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/remote-android/redroid-doc/issues/67, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAP5VMRUGJY7TR4ASDAOOHTUQFZPRANCNFSM5JYE2K2Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

mythsman commented 2 years ago

I'm working on Raspberry 4B too , and I'm also tired of the terrible performance without gpu support....

Looking forward to your support 😄

zhouziyang commented 2 years ago

@mythsman no Raspberry devices by hand. I add more GPU / SoC support in redroid11 , and you can build by yourself (or wait for images get updated by end of this month)

mythsman commented 2 years ago

@zhouziyang Grateful for your work , I don't have much experience working on Android , but I let me try it 👍

mythsman commented 2 years ago

I saw you just merged that commit into redroid-11.0.0 , so I built that tag. I compiled it with redroid_arm64-userdebug successfully ,but that container cannot run on raspberry pi and just crashed (CrashLoopBackOff) ...

I don't know what I can do to fix ....

mythsman commented 2 years ago

The only operation I did which was different from your instruction is that I typed

sudo tar --xattrs -c vendor -C system --exclude="vendor" . > redroid.tar

then upload that redroid.tar to my raspberrypi

and then typed cat redroid.tar |docker import -c 'ENTRYPOINT ["/init", "qemu=1", "androidboot.hardware=redroid"]' - redroid

mythsman commented 2 years ago

After trying compling it with that commit reverted , I also got an image that always crashes .... I thought maybe I have done sth wrong ....

zhouziyang commented 2 years ago

my fault, please update /device/redroid repo, and build again. BTW, can you provide the drm driver name? (cat /sys/kernel/debug/dri/*/name)

mythsman commented 2 years ago

my fault, please update /device/redroid repo, and build again. BTW, can you provide the drm driver name? (cat /sys/kernel/debug/dri/*/name)

In my raspi ? It doesn't contains that dri directory in /sys/kernel/debug ...

zhouziyang commented 2 years ago

debugfs mounted? what's the HOST OS? (ubuntu?)

try inside container, debugfs mounted there

mythsman commented 2 years ago

my HOST OS: Linux version 4.19.95-19 (root@runner-857a6918-project-16016-concurrent-0) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)) #1 SMP Sun Feb 7 08:55:02 UTC 2021

In my HOST OS , /sys/kernel/debug is an empty dir...

zhouziyang commented 2 years ago

try mount debugfs in Host: mount -t debugfs none /sys/kernel/debug/

mythsman commented 2 years ago

cirrus dev=0000:00:02.0 unique=0000:00:02.0

mythsman commented 2 years ago

BTW , I found that in my redroid.tar (from sudo tar --xattrs -c vendor -C system --exclude="vendor" . > redroid.tar ) , there was no vendor/ directory or system/vendor directory ... I thought maybe there was some problems...

mythsman commented 2 years ago

I just try running x86_64 in my HOST OS , and got :

[Fri Dec 24 14:21:06 2021] init: mkdir("/dev/pts", 0755) failed File exists
[Fri Dec 24 14:21:06 2021] init: mount("proc", "/proc", "proc", 0, "hidepid=2,gid=" MAKE_STR(AID_READPROC)) failed Device or resource busy
[Fri Dec 24 14:21:06 2021] init: mount("sysfs", "/sys", "sysfs", 0, NULL) failed Device or resource busy
[Fri Dec 24 14:21:06 2021] init: mount("selinuxfs", "/sys/fs/selinux", "selinuxfs", 0, NULL) failed No such file or directory
[Fri Dec 24 14:21:06 2021] init: mknod("/dev/kmsg", S_IFCHR | 0600, makedev(1, 11)) failed File exists
[Fri Dec 24 14:21:06 2021] init: mknod("/dev/random", S_IFCHR | 0666, makedev(1, 8)) failed File exists
[Fri Dec 24 14:21:06 2021] init: mknod("/dev/urandom", S_IFCHR | 0666, makedev(1, 9)) failed File exists
[Fri Dec 24 14:21:06 2021] init: mknod("/dev/ptmx", S_IFCHR | 0666, makedev(5, 2)) failed File exists
[Fri Dec 24 14:21:06 2021] init: mknod("/dev/null", S_IFCHR | 0666, makedev(1, 3)) failed File exists
[Fri Dec 24 14:21:06 2021] init: init first stage started!
[Fri Dec 24 14:21:06 2021] init: 8 output lines suppressed due to ratelimiting
[Fri Dec 24 14:21:06 2021] init: init second stage started!
[Fri Dec 24 14:21:06 2021] init: Failed to read /vendor/etc/selinux/plat_sepolicy_vers.txt: No such file or directory
[Fri Dec 24 14:21:06 2021] init: Could not read vendor SELinux version
[Fri Dec 24 14:21:06 2021] init: InitFatalReboot: signal 6
[Fri Dec 24 14:21:06 2021] init: #00 pc 00000000000ee80d  /system/bin/init (android::init::InitFatalReboot(int)+205)
[Fri Dec 24 14:21:06 2021] init: #01 pc 000000000006f490  /system/bin/init (android::init::InitAborter(char const*)+32)
[Fri Dec 24 14:21:06 2021] init: #02 pc 0000000000019cfc  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_3::__invoke(char const*)+60)
[Fri Dec 24 14:21:06 2021] init: #03 pc 00000000000192a0  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+368)
[Fri Dec 24 14:21:06 2021] init: #04 pc 00000000000f18e9  /system/bin/init (android::init::SelinuxGetVendorAndroidVersion()::$_2::operator()() const+601)
[Fri Dec 24 14:21:06 2021] init: #05 pc 00000000000f1677  /system/bin/init (android::init::SelinuxGetVendorAndroidVersion()+39)
[Fri Dec 24 14:21:06 2021] init: 7 output lines suppressed due to ratelimiting
[Fri Dec 24 14:21:06 2021] vetha084c9c: renamed from eth0

which was almost the same with that in my raspi

zhouziyang commented 2 years ago

cirrus dev=0000:00:02.0 unique=0000:00:02.0

this GPU is not supported.

mythsman commented 2 years ago

cirrus dev=0000:00:02.0 unique=0000:00:02.0

this GPU is not supported.

Well , I don't think it was the point , because in my HOST OS(redhat) , I can run the online version of redroid-11.0.0 at least ...

mythsman commented 2 years ago

Sorry for disturbing again , I thought I just found where the bug was ....

In my HOST OS( redhat ) ,the tar version is 1.26 ,which will drop the vendor dir .... while in another OS which has a tar with version 1.30 , it won't drop the vendor ....

mythsman commented 2 years ago

I have successfully built the newest redroid-11.0.0 , but it seems still laggy in my raspi ... 😭

zhouziyang commented 2 years ago

cirrus dev=0000:00:02.0 unique=0000:00:02.0

this GPU is not supported.

Well , I don't think it was the point , because in my HOST OS(redhat) , I can run the online version of redroid-11.0.0 at least ...

I means GPU accelerating not work on cirrus. You can try on SW rendering instead.

mythsman commented 2 years ago

cirrus dev=0000:00:02.0 unique=0000:00:02.0 was from my redhat machine , not my raspi ...

mythsman commented 2 years ago

After I change my raspi-config to vc4-fkms-v3d , I successfully found /sys/kernel/debug/dri dir .

The output for cat /sys/kernel/debug/dri/*/name is :

v3d dev=fec00000.v3d unique=fec00000.v3d
v3d dev=fec00000.v3d unique=fec00000.v3d
vc4 dev=gpu unique=gpu

The output for glxinfo -B is : 

name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Broadcom (0x14e4)
    Device: V3D 4.2 (0xffffffff)
    Version: 20.3.5
    Accelerated: yes
    Video memory: 7377MB
    Unified memory: yes
    Preferred profile: compat (0x2)
    Max core profile version: 0.0
    Max compat profile version: 2.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.1
OpenGL vendor string: Broadcom
OpenGL renderer string: V3D 4.2
OpenGL version string: 2.1 Mesa 20.3.5 (git-42a4d76fcc)
OpenGL shading language version string: 1.20

OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.3.5 (git-42a4d76fcc)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

Now , the acceleration seems to work ( because the cpu was not so high when I did my operation ) , but the rendering result was corrupted:

Xnip2021-12-26_14-44-21
zhouziyang commented 2 years ago

make sure gralloc is gbm (getprop ro.hardware.gralloc) since using software video codec, you can try to add SW usage to buffers allocated by gbm gralloc. Good Luck~~

mythsman commented 2 years ago

Well , ro.hardware.gralloc is gbm for sure , but it seems too hard for me to do sth like add SW usage to buffers allocated by gbm gralloc ...

rsinglaenph commented 2 years ago

@mythsman i followed your steps to create the .tar file and tried to run it on an ubuntu vm. However, the container stops immediately, its not able to find /init. What am i doing wrong?

mythsman commented 2 years ago

@mythsman i followed your steps to create the .tar file and tried to run it on an ubuntu vm. However, the container stops immediately, its not able to find /init. What am i doing wrong?

Thought you have fixed your problem , while I still could not fix my problem 😭 .

rsinglaenph commented 2 years ago

yes, luckily it worked for me on ubuntu vm, i am going to try with arm64 build now on raspberry pi3. But my OS is Yocto based linux distribution. Lets see how this goes.

FallenChromium commented 2 years ago

Just wanted to chime in to report that I have similar issue here. Not sure how to resolve it yet: image

FallenChromium commented 2 years ago

since using software video codec

I am not sure this is the case, because Mesa actually supports V3D driver for Videocore IV (RPi 3) and Videocore VI (RPi 4), unless it was explicitly disabled

zhouziyang commented 2 years ago

redroid use software media codecs currently. can you try redroid12 with VNC support (no SW media codecs involved)?

docker run -itd --rm --memory-swappiness=0 --privileged \
    --pull always \
    -v ~/data:/data \
    -p 5555:5555 \
    -p 5900:5900 \
    redroid/redroid:12.0.0-latest \
    redroid.vncserver=1

# connect VNC via <HOST-IP>:5900
FallenChromium commented 2 years ago

Hey @zhouziyang, thanks for the support :) Unfortunately Android 12 doesn't fully boot on Raspberry Pi 4, here's what I've got. It looks like it's soft-rebooting every ~20 secs, but I'm not sure why. lmkd issue was also present in Android 11, so any help on the GPU and lmk will be appreciated! logcat.txt ps.txt dmesg.txt P.S I doubt it's "low on memory", in htop those processes don't use much more than 1GB of RAM and I still have 2GB free. Any hints on adding SW codecs usage on A11?

zhouziyang commented 2 years ago 
## system_server died, did you mount same folder to containers?
01-16 17:33:35.363  1570  1570 E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: main
01-16 17:33:35.363  1570  1570 E AndroidRuntime: java.lang.IllegalStateException: Lost network stack
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.net.ConnectivityModuleConnector.maybeCrashWithTerribleFailure(ConnectivityModuleConnector.java:335)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.net.ConnectivityModuleConnector.access$400(ConnectivityModuleConnector.java:51)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.net.ConnectivityModuleConnector$ModuleServiceConnection.onServiceDisconnected(ConnectivityModuleConnector.java:281)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.app.LoadedApk$ServiceDispatcher.doConnected(LoadedApk.java:2070)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.app.LoadedApk$ServiceDispatcher$RunConnection.run(LoadedApk.java:2110)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.os.Handler.handleCallback(Handler.java:938)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.os.Handler.dispatchMessage(Handler.java:99)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.os.Looper.loopOnce(Looper.java:201)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at android.os.Looper.loop(Looper.java:288)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at com.android.server.SystemServer.run(SystemServer.java:902)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at com.android.server.SystemServer.main(SystemServer.java:609)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at java.lang.reflect.Method.invoke(Native Method)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
01-16 17:33:35.363  1570  1570 E AndroidRuntime:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:981)

for lmkd, please ensure CONFIG_PSI enabled.

01-16 17:29:40.975   100   100 E libpsi  : No kernel psi monitor support (errno=2)
01-16 17:29:40.975   100   100 I lowmemorykiller: No kernel memory.pressure_level support (errno=2)
01-16 17:29:40.975   100   100 E lowmemorykiller: Kernel does not support memory pressure events or in-kernel low memory killer
01-16 17:29:40.975   100   100 I lowmemorykiller: exiting
FallenChromium commented 2 years ago

did you mount same folder to containers?

Indeed I did. Duh, that was stupid. Android runs fine now, I can connect with a VNC client and I see actual UI elements, but it is super, super slow.

please ensure CONFIG_PSI enabled.

Unfortunately it is not enabled in the default Pi kernel, and building my own made it stop booting properly, for a yet unknown reason. Is failure of lmkd a major contributor to very poor performance on RPi (I can't even open the notification shade, launcher restarts itself and it takes about a minute for it to actually load up icons), or are there other reasons? Could it be optimized in any way to run smoother on a Raspberry Pi 4 (4GB RAM)? My usecase: I am trying to run a single application which is not resource intensive, but I couldn't run WayDroid on RPi, because images do not build with relevant GPU drivers enabled (https://github.com/waydroid/waydroid/issues/225). I understand that redroid may not be the best solution for my problem, but I don't see any alternative solutions for running an android app inside Raspberry Pi OS yet :(

zhouziyang commented 2 years ago

can you check gles driver used in your env? docker exec <container> dumpsys SurfaceFlinger | grep -i gles

I'm afraid you have to optimize by yourself (no RPi devices by hand...)

FallenChromium commented 2 years ago

can you check gles driver used in your env?

GLES: Broadcom, V3D 4.2, OpenGL ES 3.1 Mesa 22.0.0-devel Looks good to me, it's exactly what it should use.

FallenChromium commented 2 years ago

Okay, I've ran it on another distro which has CONFIG_PSI and swap enabled (Armbian), and it runs decently fast now, I can install and run an app (it doesn't work properly though, black screen after starting screen. Could it be because of the TCP|networking errors in the logcat? Does internet work inside the container at all?). This is a part of my logcat with errors: https://pastebin.com/UGpvn3nC.

zhouziyang commented 2 years ago

internet (ethernet) do works in redroid. check via dumpsys connectivity:

....
Active default network: 100

black screen just related to your app? (launcher or some other apps shows correctly?)

FallenChromium commented 2 years ago

black screen just related to your app? (launcher or some other apps shows correctly?)

Yes, correct, Android UI, while works slowly, shows up and functions properly.

P.S Figured it out. The app has privacy protection, so on screenshots, it's showing black screen on my phone, too! Any way to make the dummy screen work without making Android think it's a screenshot?

FallenChromium commented 2 years ago

Android 12 + latest scrcpy from github works fine on Raspberry Pi 4B, but unforunately it still didn't solve the app's black screen for me, it turned out that Android 12 hardened the FLAG_SECURE logic, so scrcpy can't show screenshot-protected apps on Android 12 and later. In the end, I use Armbian Focal + Android 11 + scrcpy, and it works fine for my usecase. Very slow, but I need it only for 1 app, so I guess it is fine. As of the original issue: A12 and A11 both use V3D driver for acceleration in my case, but ANGLE and, subsequently, Vulkan, render some things using SwiftShader for some reason, creating CPU spikes. It is something beyond my research\knowledge area, but if I can give any useful info on this, I'd be glad to do so.

image
zhouziyang commented 2 years ago

@FallenChromium thanks for your feedback. as for "CPU spikes", it's may be due to media.codec process. And I'm plan to optimize (check #94 ). Suggest to implement a full solution for your platforms (instead of using scrcpy / vnc):

zhouziyang commented 2 years ago

stale

kkkkkk812 commented 2 years ago

Okay, I've ran it on another distro which has CONFIG_PSI and swap enabled (Armbian), and it runs decently fast now, I can install and run an app (it doesn't work properly though, black screen after starting screen. Could it be because of the TCP|networking errors in the logcat? Does internet work inside the container at all?). This is a part of my logcat with errors: https://pastebin.com/UGpvn3nC.

can you tell me how to set the CONFIG_PSI?

zhouziyang commented 2 years ago

It's kernel configuration. need recompile your kernel.

a279437145 commented 1 year ago

is this problem solved? trying to run redroid on raspberry pi 4,it can run on raspberry pi OS with a custom kernel(CONFIG_PSI=y),but the display is corrupted when using hardware rendering(software rendering works fine but is very slow) image debug.sh: tmp.1GxepwBuoJ.tar.gz kernel config: config.txt

tried to run on armbian(22.08 CLI kernel 5.15.y) , but that kernel doesn't have CONFIG_PSI enabled by default

zhouziyang commented 1 year ago

possible due to SW media codecs. how about run

docker run ... \
redroid/redroid:12.0.0-latest \
androidboot.use_redroid_omx=1

No Raspberry Pi devices by hand, cannot test myself.

mythsman commented 1 year ago

I tried this command on my raspberry pi 4b : shell sudo docker run -itd --rm --privileged -p 5555:5555 redroid/redroid:11.0.0-latest androidboot.use_redroid_omx=1 and it shows as below:

https://user-images.githubusercontent.com/11818556/194085069-c0901853-577f-4cc3-a110-994fd078e63b.mov

Mostly , it goes well . 😄

zhouziyang commented 1 year ago

@mythsman yes, HW decoder needed as well.

a279437145 commented 1 year ago

thanks for your great work , it's working as expected now

a279437145 commented 1 year ago

now scrcpy works , but if i use adb shell screencap -p /sdcard/screenshot.png , the result will be corrupted again , why is this?

zhouziyang commented 1 year ago

now scrcpy works , but if i use adb shell screencap -p /sdcard/screenshot.png , the result will be corrupted again , why is this?

seems something wrong with the GBM gralloc.

how about:

docker run ... \
redroid/redroid:12.0.0-latest \
androidboot.use_redroid_omx=1 \
ro.hardware.gralloc=cros