Root user can't have write access in Kata android container

[zhiminghufighting]

Description of problem

After launched an android container based on kata, root user can't write/create file even the file owner is root with "drwxr-x--x" access right;

Expected result

root user can own the access right of whom the file owner is root;

Actual result

1) enter android container: 31bc2e1d785a:/ # ls acct init.environ.rc property_contexts android-entry init.houdini.rc root bugreports init.logs.rc sbin cache init.multidroid.rc sdcard charger init.multidroid_cloud.rc seapp_contexts config init.multidroid_cloud.usb.rc selinux_version d init.rc sepolicy data init.usb.configfs.rc service_contexts default.prop init.usb.rc storage dev init.zygote32.rc sys etc init.zygote64_32.rc system file_contexts.bin ipc ueventd.multidroid_cloud.rc fstab.multidroid_cloud mnt ueventd.rc init oem vendor init.crashlogd.rc proc 31bc2e1d785a:/ #

2) list the own/group owner/other user access right by "ls -l" 31bc2e1d785a:/ # ls -l total 6136 drwxr-xr-x 1 root root 17 2019-05-29 13:42 acct -rwxr-xr-x 1 root root 1266408 2018-11-15 07:24 android-entry lrwxrwxrwx 1 root root 50 2018-11-15 07:24 bugreports -> /data/user_de/0/com.android.shell/files/bugreports drwxr-xr-x 2 root root 6 2018-11-15 07:24 cache lrwxrwxrwx 1 root root 13 2018-11-15 07:24 charger -> /sbin/healthd drwxr-xr-x 2 root root 6 2018-11-15 07:24 config lrwxrwxrwx 1 root root 17 2018-11-15 07:24 d -> /sys/kernel/debug drwxr-x--x 34 root root 4096 2019-05-29 09:13 data -rw-r--r-- 1 root root 1160 2018-11-15 07:24 default.prop drwxr-xr-x 15 root root 2980 2019-05-29 13:42 dev drwxr-xr-x 1 root root 66 2019-05-29 13:42 etc -rw-r--r-- 1 root root 77090 2018-11-15 07:24 file_contexts.bin -rw-r--r-- 1 root root 113 2018-11-15 07:24 fstab.multidroid_cloud -rwxr-xr-x 1 root root 1468488 2018-11-15 07:24 init -rw-r--r-- 1 root root 647 2018-11-15 07:24 init.crashlogd.rc -rw-r--r-- 1 root root 887 2018-11-15 07:24 init.environ.rc -rw-r--r-- 1 root root 541 2018-11-15 07:24 init.houdini.rc -rw-r--r-- 1 root root 1557 2018-11-15 07:24 init.logs.rc -rwxr-xr-x 1 root root 1506 2018-11-15 07:24 init.multidroid.rc -rwxr-xr-x 1 root root 71 2018-11-15 07:24 init.multidroid_cloud.rc -rw-r--r-- 1 root root 4106 2018-11-15 07:24 init.multidroid_cloud.usb.rc -rwxr-xr-x 1 root root 27158 2018-11-15 07:24 init.rc -rw-r--r-- 1 root root 9283 2018-11-15 07:24 init.usb.configfs.rc -rwxr-xr-x 1 root root 5715 2018-11-15 07:24 init.usb.rc -rw-r--r-- 1 root root 411 2018-11-15 07:24 init.zygote32.rc -rw-r--r-- 1 root root 684 2018-11-15 07:24 init.zygote64_32.rc drwxr-xr-x 3 root root 20 2019-05-29 09:12 ipc drwxr-xr-x 10 root system 220 2019-05-29 13:42 mnt drwxr-xr-x 2 root root 6 2018-11-15 07:24 oem dr-xr-xr-x 84 root root 0 2019-05-29 13:42 proc -rw-r--r-- 1 root root 4745 2018-11-15 07:24 property_contexts drwxr-xr-x 2 root root 6 2018-11-15 07:24 root drwxr-xr-x 2 root root 65 2018-11-15 07:24 sbin lrwxrwxrwx 1 root root 21 2018-11-15 07:24 sdcard -> /storage/self/primary -rw-r--r-- 1 root root 758 2018-11-15 07:24 seapp_contexts -rw-r--r-- 1 root root 71 2018-11-15 07:24 selinux_version -rw-r--r-- 1 root root 178142 2018-11-15 07:24 sepolicy -rw-r--r-- 1 root root 11162 2018-11-15 07:24 service_contexts drwxr-xr-x 3 root root 60 2019-05-29 13:42 storage dr-xr-xr-x 11 root root 0 2019-05-29 13:42 sys drwxr-xr-x 17 root root 224 2018-11-15 07:24 system -rw-r--r-- 1 root root 272 2018-11-15 07:24 ueventd.multidroid_cloud.rc -rwxr-xr-x 1 root root 4853 2018-11-15 07:24 ueventd.rc lrwxrwxrwx 1 root root 14 2018-11-15 07:24 vendor -> /system/vendor

3) try to modify a file and report the error "sh: can't create test.txt: Read-only file system" 31bc2e1d785a:/data # ls adb app-private drm misc_ce security user anr backup local misc_de ss user_de app bootchart lost+found ota system app-asec cache media ota_package system_ce app-ephemeral dalvik-cache mediadrm property system_de app-lib data misc resource-cache tombstones 31bc2e1d785a:/data # echo 111 > test.txt sh: can't create test.txt: Read-only file system 1|31bc2e1d785a:/data # 1|31bc2e1d785a:/data # ls -l total 24 drwx------ 2 root root 6 2019-05-29 09:13 adb drwxrwxr-x 2 root root 52 2019-05-29 09:13 anr drwxrwx--x 4 root root 63 2019-05-29 09:13 app drwx------ 2 root root 6 2019-05-29 09:13 app-asec drwxrwx--x 2 root root 6 2019-05-29 09:12 app-ephemeral drwxrwx--x 2 root root 6 2019-05-29 09:13 app-lib drwxrwx--x 2 root root 6 2019-05-29 09:13 app-private drwx------ 3 root root 62 2019-05-29 09:12 backup drwxr-xr-x 2 root root 6 2019-05-29 09:13 bootchart drwxrwx--- 5 root root 56 2019-05-29 09:13 cache drwxrwx--x 4 root root 31 2019-05-29 09:12 dalvik-cache drwxr-x--x 59 root root 4096 2019-05-29 09:13 data drwxrwx--- 2 root root 6 2019-05-29 09:13 drm drwxr-x--x 3 root root 17 2019-05-29 09:12 local drwxrwx--- 2 root root 6 2019-05-29 09:12 lost+found drwxrwx--- 4 root root 26 2019-05-29 09:13 media drwxrwx--- 2 root root 6 2019-05-29 09:13 mediadrm drwxrwx--x 30 root root 4096 2019-05-29 09:13 misc drwxrwx--x 3 root root 15 2019-05-29 09:13 misc_ce drwxrwx--x 3 root root 15 2019-05-29 09:13 misc_de drwxrwx--x 2 root root 6 2019-05-29 09:12 ota drwxrwx--- 2 root root 6 2019-05-29 09:13 ota_package drwx------ 2 root root 241 2019-05-29 09:13 property drwxrwx--x 2 root root 6 2019-05-29 09:13 resource-cache drwx--x--x 2 root root 6 2019-05-29 09:13 security drwx------ 2 root root 6 2019-05-29 09:13 ss drwxrwxr-x 14 root root 4096 2019-05-29 09:12 system drwxrwx--- 3 root root 15 2019-05-29 09:13 system_ce drwxrwx--- 3 root root 15 2019-05-29 09:12 system_de drwxrwxr-x 2 root root 206 2019-05-29 09:12 tombstones drwx--x--x 2 root root 6 2019-05-29 09:12 user drwx--x--x 3 root root 15 2019-05-29 09:12 user_de

By the way, if i run the same android container image based on runc or run other Linux container image based on kata, there is no such access issue. I am not sure if there is related with 9pfs shared ro or build container rootfs to change it into read-only mode. I will check and verify this issue by code debugging.

Kata version :1.7.0

[grahamwhaley]

@zhiminghufighting - seeing the output of mount inside the container might give some clues as well. thx.

[devimc]

@zhiminghufighting for security reasons, in kata containers the rootfs is mounted as a read-only filesystem, hence you can't modify it. Fortunately you can re-mount it running mount -o remount,rw / /

[zhiminghufighting]

@grahamwhaley yes, i also think it is related with 'mount' operations. @devimc i have questions about "kata containers rootfs is mounted as a read-only filesystem:

1. per my understanding, the 'mounted' actions is implemented by agent in kata guest VM, am i right?
2. so i need to do the 'mount -o remount,rm / /' in the Kata guest VM shell, am i right?
3. i see there are different properties for kataShared mount as below: A) in a centos container shell: Here the kataShared based on 9p is "rw". kataShared on / type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)

[root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9cb629eda3ce android:NAe000032 "/android-entry -c 0…" 4 hours ago Up 4 hours 0.0.0.0:5555->5555/tcp android0 2eaa35d9fd64 centos "/bin/bash" 4 hours ago Up 4 hours musing_mahavira 93a0d1fb64e0 centos "/bin/bash" 5 hours ago Up 5 hours hopeful_chandrasekhar 5ceff5ce0746 centos "/bin/bash" 24 hours ago Up 24 hours cocky_easley [root@localhost ~]# docker exec -ti 2eaa sh sh-4.2# sh-4.2# sh-4.2# sh-4.2# ls anaconda-post.log dev home lib64 mnt proc run srv tmp var bin etc lib media opt root sbin sys usr sh-4.2# mount kataShared on / type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) kataShared on /etc/resolv.conf type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio) kataShared on /etc/hostname type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio) kataShared on /etc/hosts type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666) proc on /proc/bus type proc (ro,relatime) proc on /proc/fs type proc (ro,relatime) proc on /proc/irq type proc (ro,relatime) proc on /proc/sys type proc (ro,relatime) tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755) tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755) tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755) tmpfs on /proc/scsi type tmpfs (ro,relatime) tmpfs on /sys/firmware type tmpfs (ro,relatime) sh-4.2#

B) in an android container shell: kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio) why it is "ro" instead of "rw" but they both are launched by the same command and kata settings?

[root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9cb629eda3ce android:NAe000032 "/android-entry -c 0…" 4 hours ago Up 4 hours 0.0.0.0:5555->5555/tcp android0 2eaa35d9fd64 centos "/bin/bash" 4 hours ago Up 4 hours musing_mahavira 93a0d1fb64e0 centos "/bin/bash" 5 hours ago Up 5 hours hopeful_chandrasekhar 5ceff5ce0746 centos "/bin/bash" 24 hours ago Up 24 hours cocky_easley [root@localhost ~]# docker exec -ti 9cb6 sh 9cb629eda3ce:/ # 9cb629eda3ce:/ # ls acct init.environ.rc property_contexts android-entry init.houdini.rc root bugreports init.logs.rc sbin cache init.multidroid.rc sdcard charger init.multidroid_cloud.rc seapp_contexts config init.multidroid_cloud.usb.rc selinux_version d init.rc sepolicy data init.usb.configfs.rc service_contexts default.prop init.usb.rc storage dev init.zygote32.rc sys etc init.zygote64_32.rc system file_contexts.bin ipc ueventd.multidroid_cloud.rc fstab.multidroid_cloud mnt ueventd.rc init oem vendor init.crashlogd.rc proc 9cb629eda3ce:/ # mount kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) kataShared on /data type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio) kataShared on /ipc type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio) kataShared on /etc/resolv.conf type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio) kataShared on /etc/hostname type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio) kataShared on /etc/hosts type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=755) devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime) tmpfs on /mnt type tmpfs (rw,relatime,mode=755,gid=1000) none on /dev/memcg type cgroup (rw,relatime,memory) none on /dev/cpuctl type cgroup (rw,relatime,cpu,cpuacct) none on /dev/cpuset type cgroup (rw,relatime,cpuset) tmpfs on /storage type tmpfs (rw,relatime,mode=755,gid=1000)

why they have different behavior with the same kata setting and same launching command?

[devimc]

@zhiminghufighting

per my understanding, the 'mounted' actions is implemented by agent in kata guest VM, am i right?

no, the guest kernel mounts the rootfs as a ro fs, you can check it in /proc/cmdline

so i need to do the 'mount -o remount,rm / /' in the Kata guest VM shell, am i right?

Yes

B) in an android container shell: kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio) why it is "ro" instead of "rw" but they both are launched by the same command and kata settings?

that's weird, are you using a custom kernel?

why they have different behavior with the same kata setting and same launching command?

good question ... :thinking:

[zhiminghufighting]

@devimc yes, i used customized kata guest kernel, but i just change the kernel config for some android depends on modular and never touch kernel cmdline.

[grahamwhaley]

@zhiminghufighting @devimc - you are trying to write to /data, yes - and from the mount output, we see:

kataShared on /data type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)

So, that /data' is anro9p mount. I guess the question is, how is/data` being requested/passed in to the container - on the command line as a volume etc. etc. ?

[zhiminghufighting]

@devimc @grahamwhaley i setup debug console in the way by add "agent.debug" in cmdline and enter the cmd(mount -o remount,rw / )to try to remount kata guest rootfs as "rw" and get errors as below:

[root@localhost multidroid_cloud-aic-NAe000032]# [root@localhost multidroid_cloud-aic-NAe000032]# [root@localhost multidroid_cloud-aic-NAe000032]# [root@localhost multidroid_cloud-aic-NAe000032]# sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}" [ 471.210563] init: couldn't write 2626 to /dev/cpuset/camera-daemon/tasks: No such file or directory [ 476.315887] init: couldn't write 2655 to /dev/cpuset/camera-daemon/tasks: No such file or directory [ 481.404290] init: couldn't write 2684 to /dev/cpuset/camera-daemon/tasks: No such file or directory bin dev home lib64 media opt root sbin sys usr boot etc lib lost+found mnt proc run srv tmp var bash-4.2# [ 487.306680] init: cannot execve('/system/bin/mediaserver'): Exec format error bash-4.2# mount -o remount,rw / mount: 'kataShared' not user mountable in fstab 1|cloud:/ $ [ 492.385424] init: cannot execve('/system/bin/mediaserver'): Exec format error [ 506.331266] init: couldn't write 2830 to /dev/cpuset/camera-daemon/tasks: No such file or directory [ 511.416619] init: couldn't write 2859 to /dev/cpuset/camera-daemon/tasks: No such file or directory

And kata guest filesystem is still read-only and the cmd doesn't work for this issue.

As the comparison , i try the same cmd in the kata guest shell of centos or ubuntu container and they can work.

[zhiminghufighting]

@devimc @grahamwhaley @jodh-intel i check all the possibile code in kata-runtime for bindmount & kernel cmd paramter and make sure there is no 'ro' can be set in host side bindmount. The cmd(mount -o remount,rw /) doesn't work in kata guest VM shell too.

Is there any other ways to enable 'write' right for root user in a container? Is there possibility that 'ro' property is set by agent in guest VM?

Thanks in advance!

[devimc]

@zhiminghufighting

mount -o remount,rw /
mount: 'kataShared' not user mountable in fstab

you forgot the last /, try again mount -o remount,rw / /

[zhiminghufighting]

@grahamwhaley /data is passed into container by '-v' for some dependent lib of android app. @devimc thank a lot! I try again the cmd and find new error as below.

bash-4.2# bash-4.2# mount -o remount,rw / / mount: '/' not in /proc/mounts

127|cloud:/ $ bash-4.2# [95713.216696] init: cannot execve('/system/bin/drmserver'): Exec format error [95713.218291] init: cannot execve('/system/bin/debuggerd'): Exec format error bash-4.2# cat /proc/mounts kataShared / 9p ro,dirsync,relatime,mmap,access=client,trans=virtio 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 kataShared /ipc 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0 kataShared /data 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0 kataShared /etc/resolv.conf 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0 kataShared /etc/hostname 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0 kataShared /etc/hosts 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0 shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0 devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 tmpfs /mnt tmpfs rw,relatime,mode=755,gid=1000 0 0 none /dev/memcg cgroup rw,relatime,memory 0 0 none /dev/cpuctl cgroup rw,relatime,cpu,cpuacct 0 0 none /dev/cpuset cgroup rw,relatime,cpuset 0 0 tmpfs /storage tmpfs rw,relatime,mode=755,gid=1000 0 0

It is weird, there is no /dev/root / in /proc/mounts file.

[jodh-intel]

@zhiminghufighting - That looks very odd - I think you are seeing a mixture of the Android init system and systemd there. The "exec format error" suggests a mismatch between binaries and the host architecture (either bitness or totally different architectures).

It would really help us to help you if you could explain how we can recreate this issue with publically available images.

[zhiminghufighting]

@jodh-intel yes, it is weird. i agree with your analysis it is high possibility of mixture of android init system and system there. And i wonder why 9p mount share point is "ro" instead of "rw".

I have this image and related kata kernel image to reproduce such issue. Do you need to access my environment or i copy the android contaienr image + kata guest image to any test bed? Then you can reproduce it easily.

[jodh-intel]

@zhiminghufighting - I think what would be best would be instructions on this issue explaining how to create the sort of environment you are using:

• details of custom kernel
• details of custom image
• details of docker image you are using.
• full Kata config (or just output of sudo kata-collect-data.sh, after you have reviewed it to make sure there is no sensitive / secret / personal information included in it).
• details of the command you are running.

I'm guessing it doesn't have to be exactly the same as the environment you are using (as that might have sensitive / customer information in it?) but I don't think anyone has tried running Kata with Android so the problems you are seeing are probably generic issues.

[zhiminghufighting]

@jodh-intel, there are many customization for this case.

1. kata guest kata is reconfigure to enable vga;
2. kata runtime is reconfigure to enable vga;
3. kata runtime to patch for floppy device filtering error for some specific host os;
4. kata guest luanching paramter is changed for GUI display;
5. i rebuild an andriod container image from a new base and no such "ro" issue in the new image and the suusgestion provided by @devimc run well. A. old andriod image + docker runc is OK without "ro" issue; B. new based andriod image + docker runc & kata is OK without "ro" issue; C. old andriod image + kata is with "ro" issue; So i wonder why the same image is Ok in runc but there is "ro" issue in kata? And this is the only one image being found such weird issue. I dig into it and guess there must be some potential nasty flaw in somewhere related with contianer image build or kata code.

Thanks you all for great support! i will keep to dig in and need to get more usefull info then to discuss with you guys.

[zhiminghufighting]

@devimc @jodh-intel @egernst @grahamwhaley I get help from PRC local android team and find the root cause of this issue: android container will remount rootfs as ro in this container and create some folder and assign there different access right according to different directories for security purpose. That beyond of the coverage of kata code and docker file. So i will close this issue and thank you all!