search

remote-android / redroid-doc

redroid (Remote-Android) is a multi-arch, GPU enabled, Android in Cloud solution. Track issues / docs here
3.86k stars 282 forks source link

Trying to run redroid in Android #704

Open bobmorane83 opened 1 week ago

bobmorane83 commented 1 week ago

Describe the bug

My project is a bit unusual : I'am trying to run redroid container in android AOSP (BTW i'am newbee to Redroid) For now, I've Cuttlefish Android 14 running k3s fine (no Docker only Rancher stuff) (every kernel specific flags have been setted up)

As Android has already binder natively, do you think that redroid is able to use it ? So far after many tries the /init can't start. Missing debug logs to identify why ...

Android is able to create separated binder with binderfs, is it a way that can be done ? Here after my tests.

BTW Redroid is running fine on my k3s cluster on my Ubuntu 22.04 with kernel binder module.

Thanks for any help. V.

Here is my test : 

apiVersion: apps/v1
kind: StatefulSet 
  #kind: Deployment 
metadata:
  name: redroid
  labels:
    app: redroid
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redroid
  template:
    metadata:
      labels:
        app: redroid
    spec:
      containers:
      - name: redroid
       image: redroid/redroid:14.0.0_64only-latest 
#        command: ["/bin/sleep"]
#        args: ["3000"]
        readinessProbe:
          exec:
            command:
            - /system/bin/sh
            - -c
            - test "1" = "`/system/bin/getprop sys.boot_completed`"
          initialDelaySeconds: 5
        command: ["/init"]
        args: ["androidboot.hardware=redroid"]
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true 

        volumeMounts:
        - name: apex
          mountPath: /apex
        - name: linkerconfig
          mountPath: /linkerconfig
        - name: hwbinder
          mountPath: /dev/hwbinder
        - name: binder
          mountPath: /dev/binder
        - name: logcat
          mountPath: /dev/socket/logdw
        - name: binderfs
          mountPath: /dev/binderfs
        - name: properties
          mountPath: /dev/__properties__
        - name: data 
          mountPath: /data
        - name: sys
          mountPath: /sys
      volumes:
      - name: sys
        hostPath:
          path: /sys
      - name: apex
        hostPath:
          path: /apex
      - name: system
        hostPath:
          path: /system
      - name: linkerconfig
        hostPath:
          path: /linkerconfig
      - name: hwbinder
        hostPath:
          path: /dev/hwbinder2
      - name: binder
        hostPath:
          path: /dev/binder2
      - name: logcat
        hostPath:
          path: /dev/socket/logdw
      - name: binderfs
        hostPath:
          path: /dev/binderfs2
      - name: properties 
        hostPath:
          path: /dev/__properties__
      - name: proxy 
        hostPath:
          path: /dev/socket/dnsproxyd
      - name: data 
        emptyDir: {}
bobmorane83 commented 1 week ago

Adding some logs (logcat for Android host)

06-25 13:44:59.773   741   964 D NetdWrapper: NetdWrapper interface add, iface= veth9e661c3a
06-25 13:44:59.773   204   204 W auditd  : type=1700 audit(0.0:313164): dev=veth9e661c3a prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
06-25 13:44:59.773 19328 19328 W bridge  : type=1300 audit(0.0:313164): arch=c000003e syscall=44 success=yes exit=40 a0=9 a1=c000026360 a2=28 a3=0 items=0 ppid=19316 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 exe="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/cni" subj=u:r:su:s0 key=(null)
06-25 13:44:59.773   204   204 W auditd  : type=1327 audit(0.0:313164): proctitle="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/bridge"
06-25 13:44:59.773   204   204 W auditd  : type=1320 audit(0.0:313164): 

06-25 13:44:59.779     0     0 I cni0    : port 5(veth9e661c3a) entered blocking state
06-25 13:44:59.779     0     0 I cni0    : port 5(veth9e661c3a) entered disabled state
06-25 13:44:59.780     0     0 I         : device veth9e661c3a entered promiscuous mode
06-25 13:44:59.784     0     0 I IPv6    : ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
06-25 13:44:59.784     0     0 I IPv6    : ADDRCONF(NETDEV_CHANGE): veth9e661c3a: link becomes ready
06-25 13:44:59.784     0     0 I cni0    : port 5(veth9e661c3a) entered blocking state
06-25 13:44:59.784     0     0 I cni0    : port 5(veth9e661c3a) entered forwarding state

06-25 13:44:59.795     1     1 W libc    : Unable to set property "dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "dm-82": error code: 0x24
06-25 13:44:59.795     1     1 W libc    : Unable to set property "dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "vda17": error code: 0x24
06-25 13:44:59.795     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "vda": error code: 0x24
06-25 13:44:59.796     1     1 W libc    : Unable to set property "dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "dm-82": error code: 0x24
06-25 13:44:59.796     1     1 W libc    : Unable to set property "dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "vda17": error code: 0x24
06-25 13:44:59.796     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs" to "vda": error code: 0x24
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:44:59.800     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.c009f3f1a910e224ef921a21e7de4b52aa10e5bab6742ebb8bbf8c58e357f3e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed

06-25 13:44:59.805 19369 19369 I runc    : type=1400 audit(0.0:313165): avc:  denied  { bpf } for  capability=39  scontext=u:r:su:s0 tcontext=u:r:su:s0 tclass=capability2 permissive=1
06-25 13:44:59.814     0     0 W audit   : audit_lost=1242991 audit_rate_limit=5 audit_backlog_limit=64
06-25 13:44:59.814     0     0 E audit   : rate limit exceeded

06-25 13:45:01.564     1     1 W libc    : Unable to set property "dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "dm-82": error code: 0x24
06-25 13:45:01.564     1     1 W libc    : Unable to set property "dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "vda17": error code: 0x24
06-25 13:45:01.564     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "vda": error code: 0x24
06-25 13:45:01.565     1     1 W libc    : Unable to set property "dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "dm-82": error code: 0x24
06-25 13:45:01.565     1     1 W libc    : Unable to set property "dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "vda17": error code: 0x24
06-25 13:45:01.565     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs" to "vda": error code: 0x24
06-25 13:45:01.568     0     0 E init    : Unable to set property 'dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:01.569     0     0 E init    : Unable to set property 'dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:01.569     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:01.569     0     0 E init    : Unable to set property 'dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:01.569     0     0 E init    : Unable to set property 'dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:01.569     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.f53830f595585acae28c141db9b2d8a620d8322c7b51867bf44ab10e2cd9d415.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed

06-25 13:45:01.593 19403 19403 I runc    : type=1400 audit(0.0:313171): avc:  denied  { bpf } for  capability=39  scontext=u:r:su:s0 tcontext=u:r:su:s0 tclass=capability2 permissive=1
06-25 13:45:01.593 19403 19403 W runc    : type=1300 audit(0.0:313171): arch=c000003e syscall=321 success=yes exit=0 a0=10 a1=c00019de98 a2=20 a3=0 items=0 ppid=19359 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 exe="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/runc" subj=u:r:su:s0 key=(null)
06-25 13:45:01.593   204   204 W auditd  : type=1327 audit(0.0:313171): proctitle=72756E63002D2D726F6F74002F72756E2F636F6E7461696E6572642F72756E632F6B38732E696F002D2D6C6F67002F72756E2F6B33732F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6B38732E696F2F66353338333066353935353835616361653238633134316462
06-25 13:45:01.593   204   204 W auditd  : type=1320 audit(0.0:313171): 
06-25 13:45:01.593   204   204 W auditd  : type=1334 audit(0.0:313172): prog-id=153 op=LOAD
06-25 13:45:01.600     0     0 W audit   : audit_lost=1243008 audit_rate_limit=5 audit_backlog_limit=64
06-25 13:45:01.600     0     0 E audit   : rate limit exceeded

06-25 13:45:01.619     0     0 E init    : mkdir("/dev/pts", 0755) failed File exists
06-25 13:45:01.619     0     0 E init    : mount("sysfs", "/sys", "sysfs", 0, NULL) failed Device or resource busy
06-25 13:45:01.619     0     0 E init    : mknod("/dev/kmsg", S_IFCHR | 0600, makedev(1, 11)) failed File exists
06-25 13:45:01.619     0     0 E init    : mknod("/dev/kmsg_debug", S_IFCHR | 0622, makedev(1, 11)) failed File exists
06-25 13:45:01.619     0     0 E init    : mknod("/dev/random", S_IFCHR | 0666, makedev(1, 8)) failed File exists
06-25 13:45:01.619     0     0 E init    : mknod("/dev/urandom", S_IFCHR | 0666, makedev(1, 9)) failed File exists
06-25 13:45:01.619     0     0 E init    : mknod("/dev/ptmx", S_IFCHR | 0666, makedev(5, 2)) failed File exists
06-25 13:45:01.619     0     0 E init    : mknod("/dev/null", S_IFCHR | 0666, makedev(1, 3)) failed File exists
06-25 13:45:01.619     0     0 I init    : init first stage started!
06-25 13:45:01.619     0     0 I init    : Unable to open /lib/modules, skipping module loading.
06-25 13:45:01.620     0     0 I init    : Using vendor ramdisk copy of snapuserd /system/bin/snapuserd
06-25 13:45:01.620     0     0 I init    : hard linking /system/bin/snapuserd to /first_stage_ramdisk/system/bin/snapuserd succeeded
06-25 13:45:01.620     0     0 I init    : Switching root to '/first_stage_ramdisk'
06-25 13:45:01.620     0     0 F init    : Unable to move mount at '/var/run/secrets/kubernetes.io/serviceaccount' to '/first_stage_ramdisk/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory
06-25 13:45:01.621     0     0 E init    : InitFatalReboot: signal 6
06-25 13:45:01.624     0     0 E init    : #00 pc 000000000016eecb  /system/bin/init (android::init::InitFatalReboot(int)+251) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:01.624     0     0 E init    : #01 pc 00000000000edc50  /system/bin/init (android::init::InitAborter(char const*)+32) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:01.624     0     0 E init    : #02 pc 000000000001a8dd  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+61) (BuildId: c3f6769c408e5854d9df375be4d35b19)
06-25 13:45:01.624     0     0 E init    : #03 pc 0000000000019e02  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+354) (BuildId: c3f6769c408e5854d9df375be4d35b19)
06-25 13:45:01.624     0     0 E init    : #04 pc 000000000017fef1  /system/bin/init (android::init::SwitchRoot(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+993) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:01.624     0     0 E init    : #05 pc 0000000000121d5a  /system/bin/init (android::init::FirstStageMain(int, char**)+9066) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:01.624     0     0 E init    : #06 pc 00000000000529ff  /system/lib64/bootstrap/libc.so (__libc_init+95) (BuildId: 627fa002ea9ffdfba6f231ae3a8c8609)
06-25 13:45:01.624     0     0 I init    : Reboot ending, jumping to kernel

06-25 13:45:03.783     1     1 W libc    : Unable to set property "dev.mnt.dev.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "dm-82": error code: 0x24
06-25 13:45:03.783     1     1 W libc    : Unable to set property "dev.mnt.blk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "vda17": error code: 0x24
06-25 13:45:03.783     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "vda": error code: 0x24
06-25 13:45:03.784     1     1 W libc    : Unable to set property "dev.mnt.dev.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "dm-82": error code: 0x24
06-25 13:45:03.784     1     1 W libc    : Unable to set property "dev.mnt.blk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "vda17": error code: 0x24
06-25 13:45:03.784     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221" to "vda": error code: 0x24
06-25 13:45:03.787     0     0 E init    : Unable to set property 'dev.mnt.dev.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.788     0     0 E init    : Unable to set property 'dev.mnt.blk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.788     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.788     0     0 E init    : Unable to set property 'dev.mnt.dev.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.788     0     0 E init    : Unable to set property 'dev.mnt.blk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.788     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount3715131221' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.786     1     1 W libc    : Unable to set property "dev.mnt.dev.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "dm-82": error code: 0x24
06-25 13:45:03.786     1     1 W libc    : Unable to set property "dev.mnt.blk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "vda17": error code: 0x24
06-25 13:45:03.786     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "vda": error code: 0x24
06-25 13:45:03.787     1     1 W libc    : Unable to set property "dev.mnt.dev.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "dm-82": error code: 0x24
06-25 13:45:03.787     1     1 W libc    : Unable to set property "dev.mnt.blk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "vda17": error code: 0x24
06-25 13:45:03.787     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316" to "vda": error code: 0x24
06-25 13:45:03.790     0     0 E init    : Unable to set property 'dev.mnt.dev.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.791     0     0 E init    : Unable to set property 'dev.mnt.blk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.791     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.data.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.791     0     0 E init    : Unable to set property 'dev.mnt.dev.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.791     0     0 E init    : Unable to set property 'dev.mnt.blk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.791     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.var.lib.rancher.k3s.agent.containerd.tmpmounts.containerd-mount2975141316' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "dm-82": error code: 0x24
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "vda17": error code: 0x24
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "vda": error code: 0x24
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "dm-82": error code: 0x24
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "vda17": error code: 0x24
06-25 13:45:03.801     1     1 W libc    : Unable to set property "dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs" to "vda": error code: 0x24
06-25 13:45:03.805     0     0 E init    : Unable to set property 'dev.mnt.dev.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.805     0     0 E init    : Unable to set property 'dev.mnt.blk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.805     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.data.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.806     0     0 E init    : Unable to set property 'dev.mnt.dev.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.806     0     0 E init    : Unable to set property 'dev.mnt.blk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.806     0     0 E init    : Unable to set property 'dev.mnt.rootdisk.run.k3s.containerd.io.containerd.runtime.v2.task.k8s.io.2a72137e91f6951af5fd0e4f1276069e89bf2f3f6a89da945a49e11df14411e6.rootfs' from uid:0 gid:0 pid:1: __system_property_add failed
06-25 13:45:03.821 19464 19464 I runc    : type=1400 audit(0.0:313178): avc:  denied  { bpf } for  capability=39  scontext=u:r:su:s0 tcontext=u:r:su:s0 tclass=capability2 permissive=1
06-25 13:45:03.821 19464 19464 W runc    : type=1300 audit(0.0:313178): arch=c000003e syscall=321 success=yes exit=0 a0=10 a1=c000171e98 a2=20 a3=0 items=0 ppid=19359 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 exe="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/runc" subj=u:r:su:s0 key=(null)
06-25 13:45:03.821   204   204 W auditd  : type=1327 audit(0.0:313178): proctitle=72756E63002D2D726F6F74002F72756E2F636F6E7461696E6572642F72756E632F6B38732E696F002D2D6C6F67002F72756E2F6B33732F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6B38732E696F2F32613732313337653931663639353161663566643065346631
06-25 13:45:03.821   204   204 W auditd  : type=1320 audit(0.0:313178): 
06-25 13:45:03.821   204   204 W auditd  : type=1334 audit(0.0:313179): prog-id=156 op=LOAD
06-25 13:45:03.828     0     0 W audit   : audit_lost=1243022 audit_rate_limit=5 audit_backlog_limit=64
06-25 13:45:03.828     0     0 E audit   : rate limit exceeded
06-25 13:45:03.848     0     0 E init    : mkdir("/dev/pts", 0755) failed File exists
06-25 13:45:03.848     0     0 E init    : mount("sysfs", "/sys", "sysfs", 0, NULL) failed Device or resource busy
06-25 13:45:03.848     0     0 E init    : mknod("/dev/kmsg", S_IFCHR | 0600, makedev(1, 11)) failed File exists
06-25 13:45:03.848     0     0 E init    : mknod("/dev/kmsg_debug", S_IFCHR | 0622, makedev(1, 11)) failed File exists
06-25 13:45:03.848     0     0 E init    : mknod("/dev/random", S_IFCHR | 0666, makedev(1, 8)) failed File exists
06-25 13:45:03.848     0     0 E init    : mknod("/dev/urandom", S_IFCHR | 0666, makedev(1, 9)) failed File exists
06-25 13:45:03.848     0     0 E init    : mknod("/dev/ptmx", S_IFCHR | 0666, makedev(5, 2)) failed File exists
06-25 13:45:03.848     0     0 E init    : mknod("/dev/null", S_IFCHR | 0666, makedev(1, 3)) failed File exists
06-25 13:45:03.848     0     0 I init    : init first stage started!
06-25 13:45:03.848     0     0 I init    : Unable to open /lib/modules, skipping module loading.
06-25 13:45:03.849     0     0 I init    : Using vendor ramdisk copy of snapuserd /system/bin/snapuserd
06-25 13:45:03.849     0     0 I init    : hard linking /system/bin/snapuserd to /first_stage_ramdisk/system/bin/snapuserd succeeded
06-25 13:45:03.849     0     0 I init    : Switching root to '/first_stage_ramdisk'
06-25 13:45:03.849     0     0 F init    : Unable to move mount at '/var/run/secrets/kubernetes.io/serviceaccount' to '/first_stage_ramdisk/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory
06-25 13:45:03.849     0     0 E init    : InitFatalReboot: signal 6
06-25 13:45:03.852     0     0 E init    : #00 pc 000000000016eecb  /system/bin/init (android::init::InitFatalReboot(int)+251) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:03.852     0     0 E init    : #01 pc 00000000000edc50  /system/bin/init (android::init::InitAborter(char const*)+32) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:03.852     0     0 E init    : #02 pc 000000000001a8dd  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+61) (BuildId: c3f6769c408e5854d9df375be4d35b19)
06-25 13:45:03.852     0     0 E init    : #03 pc 0000000000019e02  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+354) (BuildId: c3f6769c408e5854d9df375be4d35b19)
06-25 13:45:03.853     0     0 E init    : #04 pc 000000000017fef1  /system/bin/init (android::init::SwitchRoot(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+993) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:03.853     0     0 E init    : #05 pc 0000000000121d5a  /system/bin/init (android::init::FirstStageMain(int, char**)+9066) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
06-25 13:45:03.853     0     0 E init    : #06 pc 00000000000529ff  /system/lib64/bootstrap/libc.so (__libc_init+95) (BuildId: 627fa002ea9ffdfba6f231ae3a8c8609)
06-25 13:45:03.853     0     0 I init    : Reboot ending, jumping to kernel

06-25 13:45:14.254     0     0 I init    : Untracked pid 19359 exited with status 0
06-25 13:45:14.254     0     0 I init    : Untracked pid 19359 did not have an associated service entry and will not be reaped
06-25 13:45:14.261   204   204 W auditd  : type=1334 audit(0.0:313185): prog-id=152 op=UNLOAD
06-25 13:45:14.261   204   204 W auditd  : type=1700 audit(0.0:313186): dev=veth9e661c3a prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
06-25 13:45:14.268     0     0 I cni0    : port 5(veth9e661c3a) entered disabled state
06-25 13:45:14.268     0     0 I         : device veth9e661c3a left promiscuous mode
06-25 13:45:14.269     0     0 I cni0    : port 5(veth9e661c3a) entered disabled state
06-25 13:45:14.261 19583 19583 W bridge  : type=1300 audit(0.0:313186): arch=c000003e syscall=44 success=yes exit=32 a0=b a1=c000104080 a2=20 a3=0 items=0 ppid=19578 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 exe="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/cni" subj=u:r:su:s0 key=(null)
06-25 13:45:14.261   204   204 W auditd  : type=1327 audit(0.0:313186): proctitle="/var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/bridge"
06-25 13:45:14.261   204   204 W auditd  : type=1320 audit(0.0:313186):
zhouziyang commented 1 week ago

You can run redroid inside Android OS (with docker installed). Some comments:

bobmorane83 commented 1 week ago

You can run redroid inside Android OS (with docker installed). Some comments:

  • disable selinux in android
  • only mount non-FBE protected directory as data partition
  • run official published redroid images
  • should not mount host system (dev, sys, binderfs etc.), just run redroid as normal

Tanks for the answer. selinux is disabled but :

# getprop ro.crypto.state
encrypted
# getprop ro.crypto.type
file

How to disable FBE on /data ?

BTW, I had to switch from F2FS to EXT4, because k3s do not support it ... Tx,

bobmorane83 commented 1 week ago

Disabling FBE :

# getprop ro.crypto.state
unsupported
docker run -itd --rm --privileged \
    --pull always \
    -v /data/redroid:/data \
    -p 5556:5555 \
    redroid/redroid:11.0.0-latest

and 

adb connect 192.168.98.3:5556
adb -s 192.168.98.3:5556 shell
redroid_x86_64:/ $

Also running in k3s

But this is the only version working : 12.0.0, 13.0.0, 14.0.0, and even 14.0.0_64only version failed BTW I'm running on a Cuttlefish A14

Any clues ?

Thanks, BR,

bobmorane83 commented 5 days ago

Here is the extract of crash log when running 14.0.0 :

[...]
07-02 15:32:38.079     0     0 I init    : Using vendor ramdisk copy of snapuserd /system/bin/snapuserd
07-02 15:32:38.079     0     0 I init    : hard linking /system/bin/snapuserd to /first_stage_ramdisk/system/bin/snapuserd succeeded
07-02 15:32:38.079     0     0 I init    : Switching root to '/first_stage_ramdisk'
07-02 15:32:38.079     0     0 F init    : Unable to move mount at '/var/run/secrets/kubernetes.io/serviceaccount' to '/first_stage_ramdisk/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory
07-02 15:32:38.079     0     0 E init    : InitFatalReboot: signal 6
07-02 15:32:38.082     0     0 E init    : #00 pc 000000000016eecb  /system/bin/init (android::init::InitFatalReboot(int)+251) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-02 15:32:38.082     0     0 E init    : #01 pc 00000000000edc50  /system/bin/init (android::init::InitAborter(char const*)+32) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-02 15:32:38.082     0     0 E init    : #02 pc 000000000001a8dd  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+61) (BuildId: c3f6769c408e5854d9df375be4d35b19)
07-02 15:32:38.082     0     0 E init    : #03 pc 0000000000019e02  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+354) (BuildId: c3f6769c408e5854d9df375be4d35b19)
07-02 15:32:38.082     0     0 E init    : #04 pc 000000000017fef1  /system/bin/init (android::init::SwitchRoot(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+993) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-02 15:32:38.082     0     0 E init    : #05 pc 0000000000121d5a  /system/bin/init (android::init::FirstStageMain(int, char**)+9066) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-02 15:32:38.082     0     0 E init    : #06 pc 00000000000529ff  /system/lib64/bootstrap/libc.so (__libc_init+95) (BuildId: 627fa002ea9ffdfba6f231ae3a8c8609)
07-02 15:32:38.082     0     0 I init    : Reboot ending, jumping to kernel
[...]

Any clue ?

BTW it's running fine on my Ubuntu.

bobmorane83 commented 4 days ago 
07-02 15:32:38.079     0     0 F init    : Unable to move mount at '/var/run/secrets/kubernetes.io/serviceaccount' to '/first_stage_ramdisk/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory

Why "SwitchRoot" fail on /var/run/secrets/kubernetes.io/serviceaccount ? The directory exist in the container. "first_stage_ramdisk" is created just before ... Some bad or missing option ?

void SwitchRoot(const std::string& new_root) {
    auto mounts = GetMounts(new_root);
    LOG(INFO) << "Switching root to '" << new_root << "'";
    for (const auto& mount_path : mounts) {
        auto new_mount_path = new_root + mount_path;
        mkdir(new_mount_path.c_str(), 0755);
        if (mount(mount_path.c_str(), new_mount_path.c_str(), nullptr, MS_MOVE, nullptr) != 0) {
            PLOG(FATAL) << "Unable to move mount at '" << mount_path << "' to "
                        << "'" << new_mount_path << "'";
        }
    }

This stage seems to not appear in 11.0.0 ...

Any help ?

bobmorane83 commented 4 days ago

Quite the same problem with Docker :

07-03 09:08:02.275     0     0 I init    : init first stage started!
07-03 09:08:02.275     0     0 I init    : Unable to open /lib/modules, skipping module loading.
07-03 09:08:02.275     0     0 I init    : Using vendor ramdisk copy of snapuserd /system/bin/snapuserd
07-03 09:08:02.275     0     0 I init    : hard linking /system/bin/snapuserd to /first_stage_ramdisk/system/bin/snapuserd succeeded
07-03 09:08:02.275     0     0 I init    : Switching root to '/first_stage_ramdisk'
07-03 09:08:02.275     0     0 F init    : Unable to move mount at '/etc/resolv.conf' to '/first_stage_ramdisk/etc/resolv.conf': No such file or directory
07-03 09:08:02.276     0     0 E init    : InitFatalReboot: signal 6
07-03 09:08:02.279     0     0 E init    : #00 pc 000000000016eecb  /system/bin/init (android::init::InitFatalReboot(int)+251) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-03 09:08:02.279     0     0 E init    : #01 pc 00000000000edc50  /system/bin/init (android::init::InitAborter(char const*)+32) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-03 09:08:02.279     0     0 E init    : #02 pc 000000000001a8dd  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+61) (BuildId: c3f6769c408e5854d9df375be4d35b19)
07-03 09:08:02.279     0     0 E init    : #03 pc 0000000000019e02  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+354) (BuildId: c3f6769c408e5854d9df375be4d35b19)
07-03 09:08:02.279     0     0 E init    : #04 pc 000000000017fef1  /system/bin/init (android::init::SwitchRoot(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+993) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-03 09:08:02.279     0     0 E init    : #05 pc 0000000000121d5a  /system/bin/init (android::init::FirstStageMain(int, char**)+9066) (BuildId: 72b3dfa78f6c71edc35cb9151c568ae1)
07-03 09:08:02.279     0     0 E init    : #06 pc 00000000000529ff  /system/lib64/bootstrap/libc.so (__libc_init+95) (BuildId: 627fa002ea9ffdfba6f231ae3a8c8609)
07-03 09:08:02.279     0     0 I init    : Reboot ending, jumping to kernel

and yes, "/etc/resolv.conf" do exist ...

bobmorane83 commented 4 days ago

Strange ...

# docker run -itd --rm --entrypoint /bin/sh --privileged -v /apex:/apex -v /data/redroid:/data -p 5556:5555 redroid/redroid:14.0.0-latest
# docker ps
CONTAINER ID   IMAGE                           COMMAND     CREATED         STATUS         PORTS                                       NAMES
1c40933b6e47   redroid/redroid:14.0.0-latest   "/bin/sh"   8 seconds ago   Up 2 seconds   0.0.0.0:5556->5555/tcp, :::5556->5555/tcp   dreamy_kapitsa

# docker exec -it 1c40933b6e47 /bin/sh

1c40933b6e47:/ # mkdir /first_stage_ramdisk                                                                            
130|1c40933b6e47:/ # mount -v --bind /etc/resolv.conf /first_stage_ramdisk/etc/resolv.conf 
try '/etc/resolv.conf' type 'ext3' on '/first_stage_ramdisk/etc/resolv.conf'
mount: '/etc/resolv.conf'->'/first_stage_ramdisk/etc/resolv.conf': No such file or directory

but ...

1c40933b6e47:/ # mount -t ext4 -v --bind /etc/resolv.conf /first_stage_ramdisk/etc/resolv.conf                         
try '/etc/resolv.conf' type 'ext4' on '/first_stage_ramdisk/etc/resolv.conf'
mount: '/etc/resolv.conf'->'/first_stage_ramdisk/etc/resolv.conf': No such file or directory

1c40933b6e47:/ # mount
[...]
/dev/block/vda17 on /etc/resolv.conf type ext4 (rw,seclabel,nosuid,nodev,noatime,errors=panic)
[...]
130|1c40933b6e47:/ # cd first_stage_ramdisk/                                                                           
1c40933b6e47:/first_stage_ramdisk # df -h .
Filesystem       Size Used Avail Use% Mounted on
/dev/block/vda17  77G 6.3G   71G   9% /etc/hosts

I'am quite lost there ...

zhouziyang commented 3 days ago

Try mask out /proc/bootconfig, /proc/device-tree (and possible other proc entries).

bobmorane83 commented 3 days ago

Thanks for the reply. Can't catch the point there ... you mean mask from the host ? What would be the propertie that cause the problem ? No "device-tree" directory ... only "device"

zhouziyang commented 3 days ago

The /proc/bootconfig (which introduced since Android 12) definitely impact redroid boot process. Just try mount an empty file docker run ... -v /dev/null:/proc/bootconfig ....

bobmorane83 commented 2 days ago

The /proc/bootconfig (which introduced since Android 12) definitely impact redroid boot process. Just try mount an empty file docker run ... -v /dev/null:/proc/bootconfig ....

Can't do that, ether for Docker and k3s ... (containerd/runc are the same)

RunContainerError: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/dev/null" to rootfs at "/proc/bootconfig": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b2cf197a1b1fc608b299f6ad6bf80e0e3d478157c4ecdfc044df6fb6ff4dd1ee/rootfs/proc/bootconfig" cannot be mounted because it is inside /proc: unknow