Open soundofspace opened 9 months ago
zhouziyang
commented
9 months ago docker run -d --rm \
--cap-add=ALL \
--security-opt=seccomp:unconfined \
--security-opt=apparmor:unconfined \
--device-cgroup-rule "c *:* rwm" \
-p 5555:5555 \
redroid/redroid:14.0.0-latest \
androidboot.redroid_gpu_mode=guest
## try adjust to your needs
soundofspace
commented
9 months ago Thanks for the response, adding -v /sys:/sys to that seems todo the trick. Any idea why? Or is there a way to make it output why it failed to launch?
docker run \
--cap-add=ALL \
--security-opt=seccomp:unconfined \
--security-opt=apparmor:unconfined \
--device-cgroup-rule "c *:* rwm" \
-v /sys:/sys \
-p 5555:5555 \
redroid/redroid:14.0.0-latest \
androidboot.redroid_gpu_mode=guest-v /sys:/sys should not be required.
My test environment:
ubuntu server 20.04docker 24.0.4soundofspace
commented
9 months ago Hmm maybe this is something Ubuntu 22.04 specific. This is also the image used by gcloud, so might be something more restrictive there.
I was able to narrow the volume mount to /sys/fs/cgroup
docker --version
Docker version 20.10.12, build 20.10.12-0ubuntu3Tested the same on Ubuntu 20.04.6 LTS without the volume mount and it does work there.
soundofspace
commented
9 months ago Testing on a fresh Ubuntu 22 vm even needs /sys/ mount /sys/fs/cgroup doesn't work here.
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammySome more info:
Ubuntu 20.04.6 LTS
5.15.0-1048-gcp/sys/fs/cgroup mount needed:
Ubuntu 22.04.3 LTS
5.15.0-1045-gkesys mount needed:
Ubuntu 22.04.3 LTS
6.2.0-1019-gcpTry switch back to cgroupv1
libprocessgroup: Failed to mount cgroup v2: Device or resource busy
...
libprocessgroup: Failed to make and chown /sys/fs/cgroup/uid_0: Read-only file systemIt's indeed caused by cgroup v2, thanks for the help!
How are you getting those logs? I'm experimenting with other ways to launch while still using v2 (which is sadly a requirement). There is lots of info online about launching systemd in docker with v2 which seems very promising, but haven't been able to make it work for this usecase, logs on what it fails would be a huge help.
zhouziyang
commented
9 months ago Try with podman ... --security-opt unmask=/sys/fs/cgroup ....
I'm not sure whether there is similar options for docker.
soundofspace
commented
9 months ago Will do some testing with podman, but how did you collect the logs posted in this comment. I tried using strace, but output of that is not really usable.
zhouziyang
commented
9 months ago Will do some testing with podman, but how did you collect the logs posted in this comment. I tried using strace, but output of that is not really usable.
dmesg
Take a look at https://github.com/remote-android/redroid-doc/blob/master/debug.sh , many debug instructions provider there.
soundofspace
commented
9 months ago Thanks for all the help, it's really appreciated!
I was able to make it run with this command (docker doesn't seem to support this currently)
podman run -it --cap-add=ALL --security-opt=seccomp=unconfined --security-opt=apparmor=unconfined -v /tmp/test:/sys/fs/cgroup --security-opt unmask=/sys/fs/cgroup --device-cgroup-rule 'c 238:* rwm' docker.io/redroid/redroid:14.0.0-latest shI added a volume mount for cgroup just to check if it wasn't unmasking the system cgroup folder. I think this is not needed, but I did it just to confirm that it works.
In my case binderfs had a major id of 238 so was also able to narrow that down too. Found by:
mkdir binder
mount -t binder binder binder
ls -al binder/
crw------- 1 root root 238, 9 Jan 31 13:08 binderHaven't experimented with narrowing capabilities yet.
How to translate all of this to Kubernetes is still a WIP, and currently doesn't seem feasible on first sight (without many hacks).
For other people reading this, without the proper device-cgroup-rule to pass in binderfs, dmesg will output no logs and docker/podman will crash silently.
zhouziyang
commented
9 months ago redroid will mount it's own cgroupfs actually.
Endgoal is to be able to run redroid in Kubernetes without privileged: true. To achieve that, I first tried running redroid docker without privileged, but without any success. Running it with privileged works fine.
Things I tried
Results in:
This only works for a second when mounting
sysbefore it crashes. When sys is not mounted starting the container fails without any output.I also tried
mount -t binder binder /dev/binderfs/on the host and then passing it to docker without any succes:It must be possible based on this issue. Only difference I see is how binder is mounted, and also mounting an additional video card, which is not needed?
I also tested that binderfs should technically work with the docker setup:
I sadly lack the knowledge on how binder works to make this work, or I'm missing some other crucial device/folder that should mounted. Any help would be extremely appreciated.
System: