Open dev-techmoe opened 8 months ago
How about run redroid via docker
/ podman
directly on your host?
Since binderfs
supported by your platform, -v /dev/binderfs/binder:/dev/binder ...
bind mounts are not needed.
BTW, check #411 for lxc
support.
How about run redroid via
docker
/podman
directly on your host? Sincebinderfs
supported by your platform,-v /dev/binderfs/binder:/dev/binder ...
bind mounts are not needed.BTW, check #411 for
lxc
support.
Hey @zhouziyang , thank you for your reply.
I use lxc because Proxmox VE (PVE) has a built-in lxc support, It has a very easy experience for controlling resource usage of containers, such as CPU core, RAM usage, etc, I need to make sure my server doesn't go boom because it has a very poor hardware platform 🤣.
I think I need some more tips for debugging redroid because I can't find any useful logs of this issue, could you give me some hints for debugging in this case? Thank you.
Well, I will take a look at lxc
/ libvirt
support.
Stay tuned~
Check https://github.com/remote-android/redroid-doc/blob/master/deploy/lxc.md
@zhouziyang thank you for taking time on this, I tried this and got lxc container up, but I can't get network connection in container and also can't get the adb connection.
Here is some logs
# lxc-info redroid11
Name: redroid11
State: RUNNING
PID: 44564
# nsenter -t `lxc-info redroid11 -p | awk '{print $2}'` -a ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 08:30 136:4 00:00:00 init second_stage androidboot.hardware=redroid androidboot.redroid_gpu_mode=guest root 3 1 0 08:30 ? 00:00:00 init subcontext u:r:vendor_init:s0 14 root 5 1 0 08:30 ? 00:00:00 ueventd logd 13 1 0 08:31 ? 00:00:00 logd lmkd 14 1 0 08:31 ? 00:00:00 lmkd system 15 1 0 08:31 ? 00:00:01 servicemanager system 16 1 0 08:31 ? 00:00:00 hwservicemanager shell 17 1 0 08:31 ? 00:00:00 sh root 18 1 0 08:31 136:4 00:00:00 vold --blkid_context=u:r:blkid:s0 --blkid_untrusted_context=u:r:blkid_untrusted:s0 --fsck_context=u:r:fsck:s0 --fsck_un+ system 29 1 0 08:32 136:4 00:00:00 android.system.suspend@1.0-service nobody 30 1 0 08:32 136:4 00:00:00 android.hardware.keymaster@3.0-service tombstoned 43 1 0 08:33 136:4 00:00:00 tombstoned system 54 1 0 08:35 136:4 00:00:00 android.hidl.allocator@1.0-service audioserver 55 1 0 08:35 136:4 00:00:00 android.hardware.audio.service bluetooth 56 1 0 08:35 136:4 00:00:00 android.hardware.bluetooth@1.1-service.sim media 57 1 0 08:35 136:4 00:00:00 android.hardware.cas@1.2-service system 58 1 0 08:35 136:4 00:00:00 android.hardware.gatekeeper@1.0-service.software system 59 1 0 08:35 136:4 00:00:00 android.hardware.gnss@2.1-service system 60 1 0 08:35 136:4 00:00:00 android.hardware.graphics.allocator@2.0-service system 61 1 0 08:35 136:4 00:00:00 android.hardware.graphics.composer@2.1-service system 62 1 0 08:35 136:4 00:00:00 android.hardware.health@2.1-service system 66 1 0 08:35 136:4 00:00:00 android.hardware.power.stats@1.0-service.mock system 67 1 0 08:35 136:4 00:00:00 android.hardware.sensors@2.1-service.mock wifi 69 1 0 08:35 136:4 00:00:00 android.hardware.wifi@1.0-service nobody 70 1 0 08:35 136:4 00:00:00 android.hardware.power-service.example audioserver 71 1 0 08:35 136:4 00:00:00 audioserver credstore 73 1 0 08:35 136:4 00:00:00 credstore /data/misc/credstore gpu_service 74 1 0 08:35 136:4 00:00:00 gpuservice system 75 1 0 08:35 136:4 00:00:01 surfaceflinger shell 114 1 0 08:36 136:4 00:00:00 adbd --root_seclabel=u:r:su:s0 nobody 116 1 0 08:36 136:4 00:00:00 traced_probes nobody 117 1 0 08:36 136:4 00:00:00 traced root 120 1 0 08:36 136:4 00:00:01 zygote64 root 127 1 0 08:36 136:4 00:00:01 zygote cameraserver 128 1 0 08:36 136:4 00:00:00 cameraserver drm 129 1 0 08:36 136:4 00:00:00 drmserver incidentd 131 1 0 08:36 136:4 00:00:00 incidentd root 132 1 0 08:36 136:4 00:00:00 installd keystore 133 1 0 08:36 136:4 00:00:00 keystore /data/misc/keystore mediaex 134 1 0 08:36 136:4 00:00:00 media.extractor aextractor media 135 1 0 08:36 136:4 00:00:00 media.metrics diametrics media 136 1 0 08:36 136:4 00:00:00 mediaserver root 137 1 0 08:36 136:4 00:00:00 netd root 138 1 0 08:36 136:4 00:00:00 storaged wifi 139 1 0 08:36 136:4 00:00:00 wificond mediacodec 140 1 0 08:36 136:4 00:00:00 media.codec hw/android.hardware.media.omx@1.0-service radio 141 1 0 08:36 136:4 00:00:00 rild mediacodec 142 1 0 08:36 136:4 00:00:00 media.swcodec oid.media.swcodec/bin/mediaswcodec statsd 143 1 0 08:36 136:4 00:00:00 statsd mdnsr 204 1 0 08:36 136:4 00:00:00 mdnsd system 205 1 0 08:36 136:4 00:00:00 gatekeeperd /data/misc/gatekeeper system 207 120 3 08:37 136:4 00:00:10 system_server bluetooth 368 120 0 08:42 136:4 00:00:00 com.android.bluetooth u0_a108 390 120 1 08:42 136:4 00:00:03 com.android.systemui u0_a115 406 120 0 08:42 136:4 00:00:00 com.android.permissioncontroller webview_zyg+ 606 127 0 08:42 136:4 00:00:00 webview_zygote network_sta+ 624 120 0 08:43 136:4 00:00:00 com.android.networkstack.process secure_elem+ 653 120 0 08:43 136:4 00:00:00 com.android.se radio 681 120 0 08:43 136:4 00:00:00 com.android.phone u0_a109 742 120 2 08:43 136:4 00:00:04 com.android.launcher3 system 768 120 0 08:43 136:4 00:00:00 com.android.settings u0_a104 908 120 0 08:44 136:4 00:00:00 com.android.inputmethod.latin u0_a111 931 120 0 08:44 136:4 00:00:00 android.ext.services u0_a96 938 120 0 08:44 136:4 00:00:00 com.android.deskclock u0_a110 1021 120 0 08:45 136:4 00:00:00 com.android.smspush u0_a88 1116 120 0 08:45 136:4 00:00:00 com.android.printspooler u0_a64 1241 120 0 08:46 136:4 00:00:00 com.android.managedprovisioning u0_a61 1262 120 0 08:46 136:4 00:00:00 android.process.media u0_a67 1288 120 0 08:46 136:4 00:00:00 com.android.providers.calendar u0_a112 1349 120 0 08:46 136:4 00:00:00 com.android.providers.media.module u0_a93 1442 120 0 08:48 136:4 00:00:00 com.android.contacts system 1466 120 0 08:48 136:4 00:00:00 com.android.dynsystem u0_a97 1487 120 0 08:48 136:4 00:00:00 com.android.gallery3d u0_a91 1533 120 0 08:48 136:4 00:00:00 com.android.onetimeinitializer u0_a74 1560 120 0 08:48 136:4 00:00:00 com.android.packageinstaller system 1582 120 0 08:48 136:4 00:00:00 com.android.localtransport shell 1609 120 0 08:48 136:4 00:00:00 com.android.shell u0_a84 1632 120 0 08:48 136:4 00:00:00 com.android.traceur root 1691 0 3 13:48 pts/0 00:00:00 ps -ef
ping 8.8.8.8
and 10.0.0.2
(host ip)# nsenter -t `lxc-info redroid11 -p | awk '{print $2}'` -a ping 8.8.8.8
connect: Network is unreachable
# nsenter -t `lxc-info redroid11 -p | awk '{print $2}'` -a ping 10.169.0.2
connect: Network is unreachable
redroid11.log
lxc-start redroid11 20231229020830.654 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:997 - Set process title to [lxc monitor] /var/lib/lxc redroid11 lxc-start redroid11 20231229020830.654 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:858 - First child 44562 exited lxc-start redroid11 20231229020830.655 INFO lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor lxc-start redroid11 20231229020830.656 INFO start - start.c:lxc_init:884 - Container "redroid11" is initialized lxc-start redroid11 20231229020830.681 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1028 - The monitor process uses "lxc/redroid11/monitor" as cgroup lxc-start redroid11 20231229020830.682 DEBUG storage - storage/storage.c:get_storage_by_name:209 - Detected rootfs type "dir" lxc-start redroid11 20231229020830.682 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1136 - The container process uses "lxc/redroid11/payload" as inner and "lxc/redroid11/payload" as limit cgroup lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNS lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWPID lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWUTS lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWIPC lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNET lxc-start redroid11 20231229020830.683 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWCGROUP lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 21 and stashed path as mnt:/proc/44563/fd/21 lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 22 and stashed path as pid:/proc/44563/fd/22 lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 23 and stashed path as uts:/proc/44563/fd/23 lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 24 and stashed path as ipc:/proc/44563/fd/24 lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 25 and stashed path as net:/proc/44563/fd/25 lxc-start redroid11 20231229020830.683 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 26 and stashed path as cgroup:/proc/44563/fd/26 lxc-start redroid11 20231229020830.684 DEBUG conf - conf.c:lxc_mount_rootfs:1432 - Mounted rootfs "/var/lib/lxc/redroid11/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)" lxc-start redroid11 20231229020830.684 INFO conf - conf.c:setup_utsname:875 - Set hostname to "redroid11" lxc-start redroid11 20231229020830.684 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3948 - Network device "lo" has been setup lxc-start redroid11 20231229020830.684 INFO network - network.c:lxc_setup_network_in_child_namespaces:4005 - Finished setting up network devices with caller assigned names lxc-start redroid11 20231229020830.684 INFO conf - conf.c:mount_autodev:1215 - Preparing "/dev" lxc-start redroid11 20231229020830.684 INFO conf - conf.c:mount_autodev:1276 - Prepared "/dev" lxc-start redroid11 20231229020830.684 DEBUG conf - conf.c:lxc_mount_auto_mounts:735 - Invalid argument - Tried to ensure procfs is unmounted lxc-start redroid11 20231229020830.684 DEBUG conf - conf.c:lxc_mount_auto_mounts:758 - Invalid argument - Tried to ensure sysfs is unmounted lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:mount_entry:2412 - Remounting "/root/data-redroid11" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/data" to respect bind or remount options lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:mount_entry:2431 - Flags for "/root/data-redroid11" were 4096, required extra flags are 0 lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:mount_entry:2440 - Mountflags already were 4096, skipping remount lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:mount_entry:2475 - Mounted "/root/data-redroid11" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/data" with filesystem type "none" lxc-start redroid11 20231229020830.685 DEBUG cgfsng - cgroups/cgfsng.c:__cgroupfs_mount:1541 - Mounted cgroup filesystem cgroup2 onto 23((null)) lxc-start redroid11 20231229020830.685 INFO conf - conf.c:lxc_fill_autodev:1313 - Populating "/dev" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "full" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "null" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "random" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "tty" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "urandom" lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "zero" lxc-start redroid11 20231229020830.685 INFO conf - conf.c:lxc_fill_autodev:1401 - Populated "/dev" lxc-start redroid11 20231229020830.685 INFO conf - conf.c:lxc_transient_proc:3771 - Caller's PID is 1; /proc/self points to 1 lxc-start redroid11 20231229020830.685 DEBUG conf - conf.c:lxc_setup_devpts_child:1691 - No new devpts instance will be mounted since no pts devices are required lxc-start redroid11 20231229020830.685 INFO conf - conf.c:lxc_setup_ttys:1072 - Finished setting up 0 /dev/ttydevice(s) lxc-start redroid11 20231229020830.686 DEBUG conf - conf.c:setup_personality:1907 - Retaining original personality lxc-start redroid11 20231229020830.686 DEBUG conf - conf.c:capabilities_deny:3199 - Capabilities have been setup lxc-start redroid11 20231229020830.686 NOTICE conf - conf.c:lxc_setup:4464 - The container "redroid11" is set up lxc-start redroid11 20231229020830.686 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1186 - Set AppArmor label to "unconfined" lxc-start redroid11 20231229020830.686 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1231 - Changed AppArmor profile to unconfined lxc-start redroid11 20231229020830.686 DEBUG conf - conf.c:lxc_send_devpts_to_parent:1889 - No devpts file descriptor will be sent since no pts devices are requested lxc-start redroid11 20231229020830.686 DEBUG terminal - terminal.c:lxc_terminal_peer_default:695 - No such device - The process does not have a controlling terminal lxc-start redroid11 20231229020830.686 NOTICE utils - utils.c:lxc_drop_groups:1365 - Dropped supplimentary groups lxc-start redroid11 20231229020830.686 NOTICE start - start.c:start:2161 - Exec'ing "/init" lxc-start redroid11 20231229020830.688 NOTICE start - start.c:post_start:2172 - Started "/init" with pid "44564" lxc-start redroid11 20231229020837.351 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable cpuacct controller lxc-start redroid11 20231229020837.351 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable blkio controller lxc-start redroid11 20231229020838.839 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable cpuacct controller lxc-start redroid11 20231229020838.839 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable blkio controller lxc-start redroid11 20231229020839.590 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable cpuacct controller lxc-start redroid11 20231229020839.590 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable blkio controller lxc-start redroid11 20231229021241.636 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable cpuacct controller lxc-start redroid11 20231229021241.636 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable blkio controller
lxc (1:4.0.6-2+deb11u2 Debian:11.8/oldstable [amd64])
cat /var/lib/lxc/redroid11/config
lxc.net.0.type = empty
lxc.rootfs.path = dir:/var/lib/lxc/redroid11/rootfs
lxc.execute.cmd = '"/init" "androidboot.hardware=redroid" '
lxc.mount.auto = proc:mixed sys:mixed cgroup:mixed
lxc.environment =
lxc.uts.name = redroid11
lxc.init.uid = 0
lxc.init.gid = 0
lxc.init.cwd = /
### hacked
lxc.init.cmd = /init androidboot.hardware=redroid androidboot.redroid_gpu_mode=guest
lxc.apparmor.profile = unconfined
lxc.autodev = 1
lxc.autodev.tmpfs.size = 25000000
lxc.mount.entry = /root/data-redroid11 data none bind 0 0
is possible lxc.net.0.type = empty
cause this problem?
Check lxc-net
, and make sure container networkings are properly configured.
@dev-techmoe hello, can I ask which Proxmox version do you use ? Did the kernel modules come preinstalled or you had to add them ?
@dev-techmoe hello, can I ask which Proxmox version do you use ? Did the kernel modules come preinstalled or you had to add them ?
PVE Version: 7.2-3
uname -a
: Linux lu 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) x86_64 GNU/Linux
All kernel modules are built in this default kernel, I'm not adding anything to this kernel.
I'm busy working recently, and I will continue try to fix this LXC container running on PVE when I free.
- Try to launch ReDroid container
docker run -it --privileged --pull always -v /dev/binderfs/binder:/dev/binder -v /dev/binderfs/hwbinder:/dev/hwbinder -v /dev/binderfs/vndbinder:/dev/vndbinder -v ~/data8:/data -p 5555:5555 --rm redroid/redroid:8.1.0-latest
- I got container crashed immediately without any logs.
Ok, so it seems to start redroid in Docker IN LXC in Proxmox you just add lxc.apparmor.profile: unconfined
to the pve lxc conf.
However I now have the same issue as you with the network connect: Network is unreachable
curious if anyone got this working natively in proxmox using pct rather than the lxc methods linked?
curios if anyone got this working with gpu accel androidboot.redroid_gpu_mode=host
?
I added
lxc.cgroup2.devices.allow = c 226:0 rwm
lxc.cgroup2.devices.allow = c 226:128 rwm
lxc.mount.entry = /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
But the init process crashes every time. redroid11.log
shows
lxc-start redroid11 20240626030842.781 NOTICE start - ../src/lxc/start.c:post_start:2212 - Started "/init" with pid "39363"
lxc-start redroid11 20240626030842.797 DEBUG start - ../src/lxc/start.c:signal_handler:465 - Container init process 39363 exited
lxc-start redroid11 20240626030842.798 DEBUG start - ../src/lxc/start.c:__lxc_start:2147 - Hangup(1) - Container "redroid11" is rebooting
lxc-start redroid11 20240626030842.798 INFO error - ../src/lxc/error.c:lxc_error_set_and_log:34 - Child <39363> ended on signal Hangup(1)
Describe the bug I think to make redroid running in a lxc container would get a perfect memory usage than running in a VM, so I tried to do this.
Steps to reproduce:
/dev
of lxc container, but I found it's not in/dev
folder, so I choice to create and mount it manuallydmesg -T
just onlydocker logs redroid8
got nothing.make sure the required kernel modules present
grep binder /proc/filesystems
gotnodev binder
grep ashmem /proc/misc
got123 ashmem
collect debug logs
curl -fsSL https://raw.githubusercontent.com/remote-android/redroid-doc/master/debug.sh | sudo bash -s -- [CONTAINER]
omit CONTAINER if not exist any more.see info above, I have some privacy concern about this, please let me know if you need more information, sorry.
Screenshots If applicable, add screenshots to help explain your problem.