nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.78k stars 152 forks source link

Dockerd unable to run within sysbox containers in v0.6.1 pre-release #662

Closed rodnymolina closed 1 year ago

rodnymolina commented 1 year ago

In scenarios where recent versions of the docker-engine (>= 23.0.1) are installed within sysbox containers, the following error may be seen while attempting to run inner containers:

# docker run --rm -it hello-world
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default4070819680` failed with output: apparmor_parser: Unable to replace "docker-default".  Profile doesn't conform to protocol

error: exit status 185.

Issue was initially reported by @grzesuav -- thanks for that!

rodnymolina commented 1 year ago

The error above is observed while dockerd is attempting to load the docker-default apparmor profile during the initialization of the inner container.

Notice that apparmor profiles are usually not enforced in traditional docker-in-docker scenarios to prevent errors caused by attempting to write to read-only file-systems such as sysfs (i.e., /sys/kernel/security/apparmor).

https://github.com/moby/moby/blob/219f21bf07502b447095649b5a2764661737f164/vendor/github.com/containerd/containerd/pkg/apparmor/apparmor_linux.go#L34

// hostSupports returns true if apparmor is enabled for the host, if
// apparmor_parser is enabled, and if we are not running docker-in-docker.
//
// This is derived from libcontainer/apparmor.IsEnabled(), with the addition
// of checks for apparmor_parser to be present and docker-in-docker.
func hostSupports() bool {
    checkAppArmor.Do(func() {
        // see https://github.com/opencontainers/runc/blob/0d49470392206f40eaab3b2190a57fe7bb3df458/libcontainer/apparmor/apparmor_linux.go
        if _, err := os.Stat("/sys/kernel/security/apparmor"); err == nil && os.Getenv("container") == "" {
            if _, err = os.Stat("/sbin/apparmor_parser"); err == nil {
                buf, err := os.ReadFile("/sys/module/apparmor/parameters/enabled")
                appArmorSupported = err == nil && len(buf) > 1 && buf[0] == 'Y'
            }
        }
    })
    return appArmorSupported
}

This code is prone to error given that it relies on the presence of the container env-variable, which may not be defined in the execution context in which the inner dockerd operates. This env-variable is typically set by the container's high-level runtime operating in the host, but this is not always the case, e.g., docker doesn't do it -- see example below for cri-o's containers:

# echo $container
crio

Now, even in scenarios where this env-variable is present, chances are that it may not be taken into account if dockerd is launched through a process-manager such as systemd, which means that this container variable ends up being undefined and, in consequence, dockerd attempts to load the apparmor profile indicated above (i verified this by looking at strace dumps).

As a workaround for this problem we could attempt to adjust Docker's systemd file within each of the Sysbox's published images to set the container variable, but that wouldn't fix the problem for those users that install dockerd by themselves if they were to rely on generic sysbox's image such as ubuntu-jammy-systemd.

rodnymolina commented 1 year ago

By looking in more details to the error being reported during the inner container execution, as well as during dockerd's initialization, it seems clear that dockerd is attempting to load the docker-default apparmor profile during initialization, and as this is not achieved, dockerd attempts to do this during container creation too (see code here).

docker initialization error:

Mar 31 21:56:52 ubu-jammy-systemd systemd[1]: Starting Docker Application Container Engine...
Mar 31 21:56:52 ubu-jammy-systemd dockerd[11225]: time="2023-03-31T21:56:52.392130800Z" level=info msg="Starting up"
Mar 31 21:56:53 ubu-jammy-systemd dockerd[11225]: time="2023-03-31T21:56:53.457964045Z" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default4174630022` failed with output: apparmor_parser: Unable to replace \"docker-default\".  Profile doesn't conform to protocol\n\nerror: exit status 185"

docker-run error:

root@ubu-jammy-systemd:/# docker run --rm -it hello-world
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default3369832518` failed with output: apparmor_parser: Unable to replace "docker-default".  Profile doesn't conform to protocol
rodnymolina commented 1 year ago

I can think of two potential solutions for this issue:

1) Adjust Docker's systemd files to ensure that the container env-var is always present. Problem with this approach is that the container variable would need to be hardcoded to an explicit value that may not match the one being set by the high-level container runtime during execution. Also, the systemd file changes wouldn't be trivial (it may require an external shell script file). And more importantly, this wouldn't solve the problem for our users right now (an upgrade to a future docker release would be required).

2) Write new virtualization logic in sysbox-fs to allow the apparmor_parser load instruction to succeed, which is simply a write operation into one of the nodes in /sys/kernel/security/apparmor directory. These resources are non-namespaced anyways, so there's currently no possibility for a containerized process to write into them (i.e., they are displayed as nobody:nogroup within the sysbox container).

grzesuav commented 1 year ago

@rodnymolina does older docker help ? I tried with

root@grzegorz-sysbox:/# docker version
Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:34:13 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:32:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

but

root@grzegorz-sysbox:/# docker run --rm -it hello-world
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default2368293219` failed with output: apparmor_parser: Unable to replace "docker-default".  Profile doesn't conform to protocol

error: exit status 185.
rodnymolina commented 1 year ago

@grzesuav, I haven't identified the specific docker version where this issue was introduced. Based on empirical results, I assumed that it was after 23.0.1, but it may have been earlier too. I'll try to find out when this happened, but I can tell you that I'm not able to reproduce it with 20.10.17.

grzesuav commented 1 year ago

hmm interesting, I am trying that, however with different results -

root@grzegorz-sysbox-1:/# apt-get update && apt-get install -y \
    docker-ce=5:20.10.17~3-0~debian-bullseye \
    docker-ce-cli=5:20.10.17~3-0~debian-bullseye
Hit:1 https://download.docker.com/linux/debian bullseye InRelease
Hit:2 http://deb.debian.org/debian bullseye InRelease
Hit:3 http://deb.debian.org/debian-security bullseye-security InRelease
Hit:4 http://deb.debian.org/debian bullseye-updates InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  apparmor containerd.io dbus dbus-user-session dmsetup docker-ce-rootless-extras docker-scan-plugin iptables libapparmor1 libargon2-1 libcap2 libcryptsetup12 libdbus-1-3 libdevmapper1.02.1 libip4tc2 libip6tc2 libjansson4
  libjson-c5 libkmod2 libmnl0 libnetfilter-conntrack3 libnfnetlink0 libnftables1 libnftnl11 libnss-systemd libpam-systemd libslirp0 libsystemd0 libxtables12 nftables pigz slirp4netns systemd systemd-sysv systemd-timesyncd
Suggested packages:
  apparmor-profiles-extra apparmor-utils aufs-tools cgroupfs-mount | cgroup-lite firewalld kmod systemd-container policykit-1
The following NEW packages will be installed:
  apparmor containerd.io dbus dbus-user-session dmsetup docker-ce docker-ce-cli docker-ce-rootless-extras docker-scan-plugin iptables libapparmor1 libargon2-1 libcap2 libcryptsetup12 libdbus-1-3 libdevmapper1.02.1 libip4tc2
  libip6tc2 libjansson4 libjson-c5 libkmod2 libmnl0 libnetfilter-conntrack3 libnfnetlink0 libnftables1 libnftnl11 libnss-systemd libpam-systemd libslirp0 libxtables12 nftables pigz slirp4netns systemd systemd-sysv
  systemd-timesyncd
The following packages will be upgraded:
  libsystemd0
1 upgraded, 36 newly installed, 0 to remove and 80 not upgraded.
Need to get 111 MB of archives.
After this operation, 437 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 libsystemd0 amd64 247.3-7+deb11u1 [376 kB]
Get:2 https://download.docker.com/linux/debian bullseye/stable amd64 containerd.io amd64 1.6.20-1 [28.3 MB]
Get:3 http://deb.debian.org/debian bullseye/main amd64 libjansson4 amd64 2.13.1-1.1 [39.7 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 libmnl0 amd64 1.0.4-3 [12.5 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 libnftnl11 amd64 1.1.9-1 [63.7 kB]
Get:6 http://deb.debian.org/debian bullseye/main amd64 libxtables12 amd64 1.8.7-1 [45.1 kB]
Get:7 http://deb.debian.org/debian bullseye/main amd64 libnftables1 amd64 0.9.8-3.1+deb11u1 [261 kB]
Get:8 http://deb.debian.org/debian bullseye/main amd64 nftables amd64 0.9.8-3.1+deb11u1 [68.7 kB]
Get:9 http://deb.debian.org/debian bullseye/main amd64 libapparmor1 amd64 2.13.6-10 [99.3 kB]
Get:10 http://deb.debian.org/debian bullseye/main amd64 libcap2 amd64 1:2.44-1 [23.6 kB]
Get:11 http://deb.debian.org/debian bullseye/main amd64 libargon2-1 amd64 0~20171227-0.2 [19.6 kB]
Get:12 http://deb.debian.org/debian bullseye/main amd64 dmsetup amd64 2:1.02.175-2.1 [92.1 kB]
Get:13 http://deb.debian.org/debian bullseye/main amd64 libdevmapper1.02.1 amd64 2:1.02.175-2.1 [143 kB]
Get:14 http://deb.debian.org/debian bullseye/main amd64 libjson-c5 amd64 0.15-2 [42.8 kB]
Get:15 http://deb.debian.org/debian bullseye/main amd64 libcryptsetup12 amd64 2:2.3.7-1+deb11u1 [248 kB]
Get:16 http://deb.debian.org/debian bullseye/main amd64 libip4tc2 amd64 1.8.7-1 [34.6 kB]
Get:17 http://deb.debian.org/debian bullseye/main amd64 libkmod2 amd64 28-1 [55.6 kB]
Get:18 http://deb.debian.org/debian bullseye/main amd64 systemd amd64 247.3-7+deb11u1 [4501 kB]
Get:19 http://deb.debian.org/debian bullseye/main amd64 systemd-sysv amd64 247.3-7+deb11u1 [113 kB]
Get:20 http://deb.debian.org/debian bullseye/main amd64 libdbus-1-3 amd64 1.12.24-0+deb11u1 [222 kB]
Get:21 http://deb.debian.org/debian bullseye/main amd64 dbus amd64 1.12.24-0+deb11u1 [243 kB]
Get:22 http://deb.debian.org/debian bullseye/main amd64 pigz amd64 2.6-1 [64.0 kB]
Get:23 http://deb.debian.org/debian bullseye/main amd64 libnss-systemd amd64 247.3-7+deb11u1 [199 kB]
Get:24 http://deb.debian.org/debian bullseye/main amd64 libpam-systemd amd64 247.3-7+deb11u1 [283 kB]
Get:25 http://deb.debian.org/debian bullseye/main amd64 systemd-timesyncd amd64 247.3-7+deb11u1 [131 kB]
Get:26 http://deb.debian.org/debian bullseye/main amd64 apparmor amd64 2.13.6-10 [640 kB]
Get:27 http://deb.debian.org/debian bullseye/main amd64 dbus-user-session amd64 1.12.24-0+deb11u1 [99.7 kB]
Get:28 http://deb.debian.org/debian bullseye/main amd64 libip6tc2 amd64 1.8.7-1 [35.0 kB]
Get:29 http://deb.debian.org/debian bullseye/main amd64 libnfnetlink0 amd64 1.0.1-3+b1 [13.9 kB]
Get:30 http://deb.debian.org/debian bullseye/main amd64 libnetfilter-conntrack3 amd64 1.0.8-3 [40.6 kB]
Get:31 http://deb.debian.org/debian bullseye/main amd64 iptables amd64 1.8.7-1 [382 kB]
Get:32 http://deb.debian.org/debian bullseye/main amd64 libslirp0 amd64 4.4.0-1+deb11u2 [57.9 kB]
Get:33 http://deb.debian.org/debian bullseye/main amd64 slirp4netns amd64 1.0.1-2 [33.4 kB]
Get:34 https://download.docker.com/linux/debian bullseye/stable amd64 docker-ce-cli amd64 5:20.10.17~3-0~debian-bullseye [40.6 MB]
Get:35 https://download.docker.com/linux/debian bullseye/stable amd64 docker-ce amd64 5:20.10.17~3-0~debian-bullseye [21.0 MB]
Get:36 https://download.docker.com/linux/debian bullseye/stable amd64 docker-ce-rootless-extras amd64 5:23.0.2-1~debian.11~bullseye [8770 kB]
Get:37 https://download.docker.com/linux/debian bullseye/stable amd64 docker-scan-plugin amd64 0.23.0~debian-bullseye [3623 kB]
Fetched 111 MB in 1s (89.5 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 23421 files and directories currently installed.)
Preparing to unpack .../libsystemd0_247.3-7+deb11u1_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7+deb11u1) over (247.3-7) ...
Setting up libsystemd0:amd64 (247.3-7+deb11u1) ...
Selecting previously unselected package libjansson4:amd64.
(Reading database ... 23421 files and directories currently installed.)
Preparing to unpack .../00-libjansson4_2.13.1-1.1_amd64.deb ...
Unpacking libjansson4:amd64 (2.13.1-1.1) ...
Selecting previously unselected package libmnl0:amd64.
Preparing to unpack .../01-libmnl0_1.0.4-3_amd64.deb ...
Unpacking libmnl0:amd64 (1.0.4-3) ...
Selecting previously unselected package libnftnl11:amd64.
Preparing to unpack .../02-libnftnl11_1.1.9-1_amd64.deb ...
Unpacking libnftnl11:amd64 (1.1.9-1) ...
Selecting previously unselected package libxtables12:amd64.
Preparing to unpack .../03-libxtables12_1.8.7-1_amd64.deb ...
Unpacking libxtables12:amd64 (1.8.7-1) ...
Selecting previously unselected package libnftables1:amd64.
Preparing to unpack .../04-libnftables1_0.9.8-3.1+deb11u1_amd64.deb ...
Unpacking libnftables1:amd64 (0.9.8-3.1+deb11u1) ...
Selecting previously unselected package nftables.
Preparing to unpack .../05-nftables_0.9.8-3.1+deb11u1_amd64.deb ...
Unpacking nftables (0.9.8-3.1+deb11u1) ...
Selecting previously unselected package libapparmor1:amd64.
Preparing to unpack .../06-libapparmor1_2.13.6-10_amd64.deb ...
Unpacking libapparmor1:amd64 (2.13.6-10) ...
Selecting previously unselected package libcap2:amd64.
Preparing to unpack .../07-libcap2_1%3a2.44-1_amd64.deb ...
Unpacking libcap2:amd64 (1:2.44-1) ...
Selecting previously unselected package libargon2-1:amd64.
Preparing to unpack .../08-libargon2-1_0~20171227-0.2_amd64.deb ...
Unpacking libargon2-1:amd64 (0~20171227-0.2) ...
Selecting previously unselected package dmsetup.
Preparing to unpack .../09-dmsetup_2%3a1.02.175-2.1_amd64.deb ...
Unpacking dmsetup (2:1.02.175-2.1) ...
Selecting previously unselected package libdevmapper1.02.1:amd64.
Preparing to unpack .../10-libdevmapper1.02.1_2%3a1.02.175-2.1_amd64.deb ...
Unpacking libdevmapper1.02.1:amd64 (2:1.02.175-2.1) ...
Selecting previously unselected package libjson-c5:amd64.
Preparing to unpack .../11-libjson-c5_0.15-2_amd64.deb ...
Unpacking libjson-c5:amd64 (0.15-2) ...
Selecting previously unselected package libcryptsetup12:amd64.
Preparing to unpack .../12-libcryptsetup12_2%3a2.3.7-1+deb11u1_amd64.deb ...
Unpacking libcryptsetup12:amd64 (2:2.3.7-1+deb11u1) ...
Selecting previously unselected package libip4tc2:amd64.
Preparing to unpack .../13-libip4tc2_1.8.7-1_amd64.deb ...
Unpacking libip4tc2:amd64 (1.8.7-1) ...
Selecting previously unselected package libkmod2:amd64.
Preparing to unpack .../14-libkmod2_28-1_amd64.deb ...
Unpacking libkmod2:amd64 (28-1) ...
Selecting previously unselected package systemd.
Preparing to unpack .../15-systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd (247.3-7+deb11u1) ...
Setting up libapparmor1:amd64 (2.13.6-10) ...
Setting up libcap2:amd64 (1:2.44-1) ...
Setting up libargon2-1:amd64 (0~20171227-0.2) ...
Setting up libjson-c5:amd64 (0.15-2) ...
Setting up libip4tc2:amd64 (1.8.7-1) ...
Setting up libkmod2:amd64 (28-1) ...
Setting up libdevmapper1.02.1:amd64 (2:1.02.175-2.1) ...
Setting up libcryptsetup12:amd64 (2:2.3.7-1+deb11u1) ...
Setting up systemd (247.3-7+deb11u1) ...
Created symlink /etc/systemd/system/getty.target.wants/getty@tty1.service → /lib/systemd/system/getty@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/remote-fs.target → /lib/systemd/system/remote-fs.target.
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service → /lib/systemd/system/systemd-pstore.service.
Initializing machine ID from random generator.
Setting up dmsetup (2:1.02.175-2.1) ...
Selecting previously unselected package systemd-sysv.
(Reading database ... 24336 files and directories currently installed.)
Preparing to unpack .../00-systemd-sysv_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd-sysv (247.3-7+deb11u1) ...
Selecting previously unselected package libdbus-1-3:amd64.
Preparing to unpack .../01-libdbus-1-3_1.12.24-0+deb11u1_amd64.deb ...
Unpacking libdbus-1-3:amd64 (1.12.24-0+deb11u1) ...
Selecting previously unselected package dbus.
Preparing to unpack .../02-dbus_1.12.24-0+deb11u1_amd64.deb ...
Unpacking dbus (1.12.24-0+deb11u1) ...
Selecting previously unselected package pigz.
Preparing to unpack .../03-pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) ...
Selecting previously unselected package libnss-systemd:amd64.
Preparing to unpack .../04-libnss-systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking libnss-systemd:amd64 (247.3-7+deb11u1) ...
Selecting previously unselected package libpam-systemd:amd64.
Preparing to unpack .../05-libpam-systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking libpam-systemd:amd64 (247.3-7+deb11u1) ...
Selecting previously unselected package systemd-timesyncd.
Preparing to unpack .../06-systemd-timesyncd_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd-timesyncd (247.3-7+deb11u1) ...
Selecting previously unselected package apparmor.
Preparing to unpack .../07-apparmor_2.13.6-10_amd64.deb ...
Unpacking apparmor (2.13.6-10) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../08-containerd.io_1.6.20-1_amd64.deb ...
Unpacking containerd.io (1.6.20-1) ...
Selecting previously unselected package dbus-user-session.
Preparing to unpack .../09-dbus-user-session_1.12.24-0+deb11u1_amd64.deb ...
Unpacking dbus-user-session (1.12.24-0+deb11u1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../10-docker-ce-cli_5%3a20.10.17~3-0~debian-bullseye_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.17~3-0~debian-bullseye) ...
Selecting previously unselected package libip6tc2:amd64.
Preparing to unpack .../11-libip6tc2_1.8.7-1_amd64.deb ...
Unpacking libip6tc2:amd64 (1.8.7-1) ...
Selecting previously unselected package libnfnetlink0:amd64.
Preparing to unpack .../12-libnfnetlink0_1.0.1-3+b1_amd64.deb ...
Unpacking libnfnetlink0:amd64 (1.0.1-3+b1) ...
Selecting previously unselected package libnetfilter-conntrack3:amd64.
Preparing to unpack .../13-libnetfilter-conntrack3_1.0.8-3_amd64.deb ...
Unpacking libnetfilter-conntrack3:amd64 (1.0.8-3) ...
Selecting previously unselected package iptables.
Preparing to unpack .../14-iptables_1.8.7-1_amd64.deb ...
Unpacking iptables (1.8.7-1) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../15-docker-ce_5%3a20.10.17~3-0~debian-bullseye_amd64.deb ...
Unpacking docker-ce (5:20.10.17~3-0~debian-bullseye) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../16-docker-ce-rootless-extras_5%3a23.0.2-1~debian.11~bullseye_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:23.0.2-1~debian.11~bullseye) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../17-docker-scan-plugin_0.23.0~debian-bullseye_amd64.deb ...
Unpacking docker-scan-plugin (0.23.0~debian-bullseye) ...
Selecting previously unselected package libslirp0:amd64.
Preparing to unpack .../18-libslirp0_4.4.0-1+deb11u2_amd64.deb ...
Unpacking libslirp0:amd64 (4.4.0-1+deb11u2) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../19-slirp4netns_1.0.1-2_amd64.deb ...
Unpacking slirp4netns (1.0.1-2) ...
Setting up systemd-sysv (247.3-7+deb11u1) ...
Setting up libip6tc2:amd64 (1.8.7-1) ...
Setting up docker-scan-plugin (0.23.0~debian-bullseye) ...
Setting up libnss-systemd:amd64 (247.3-7+deb11u1) ...
First installation detected...
Checking NSS setup...
Setting up libjansson4:amd64 (2.13.1-1.1) ...
Setting up apparmor (2.13.6-10) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Created symlink /etc/systemd/system/sysinit.target.wants/apparmor.service → /lib/systemd/system/apparmor.service.
Setting up libdbus-1-3:amd64 (1.12.24-0+deb11u1) ...
Setting up dbus (1.12.24-0+deb11u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libmnl0:amd64 (1.0.4-3) ...
Setting up systemd-timesyncd (247.3-7+deb11u1) ...
Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /lib/systemd/system/systemd-timesyncd.service.
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.
Setting up containerd.io (1.6.20-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up libxtables12:amd64 (1.8.7-1) ...
Setting up docker-ce-cli (5:20.10.17~3-0~debian-bullseye) ...
Setting up libslirp0:amd64 (4.4.0-1+deb11u2) ...
Setting up pigz (2.6-1) ...
Setting up libnfnetlink0:amd64 (1.0.1-3+b1) ...
Setting up libpam-systemd:amd64 (247.3-7+deb11u1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Setting up slirp4netns (1.0.1-2) ...
Setting up libnftnl11:amd64 (1.1.9-1) ...
Setting up dbus-user-session (1.12.24-0+deb11u1) ...
Setting up libnetfilter-conntrack3:amd64 (1.0.8-3) ...
Setting up docker-ce-rootless-extras (5:23.0.2-1~debian.11~bullseye) ...
Setting up libnftables1:amd64 (0.9.8-3.1+deb11u1) ...
Setting up nftables (0.9.8-3.1+deb11u1) ...
Setting up iptables (1.8.7-1) ...
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in auto mode
update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in auto mode
update-alternatives: using /usr/sbin/iptables-nft to provide /usr/sbin/iptables (iptables) in auto mode
update-alternatives: using /usr/sbin/ip6tables-nft to provide /usr/sbin/ip6tables (ip6tables) in auto mode
update-alternatives: using /usr/sbin/arptables-nft to provide /usr/sbin/arptables (arptables) in auto mode
update-alternatives: using /usr/sbin/ebtables-nft to provide /usr/sbin/ebtables (ebtables) in auto mode
Setting up docker-ce (5:20.10.17~3-0~debian-bullseye) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.31-13+deb11u3) ...
root@grzegorz-sysbox-1:/# dockerd > /dev/null 2>&1 &
[1] 1994
root@grzegorz-sysbox-1:/#  docker run --rm -it  hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:ffb13da98453e0f04d33a6eee5bb8e46ee50d08ebe17735fc0779d0349e889e9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default711670642` failed with output: apparmor_parser: Unable to replace "docker-default".  Profile doesn't conform to protocol

error: exit status 185.
root@grzegorz-sysbox-1:/# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:17 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
grzesuav commented 1 year ago

@rodnymolina is there a plan to handle it along with kubernetes 1.25 support or as a standalone bug ?

rodnymolina commented 1 year ago

Problem was not caused by a specific docker release as I originally thought (I've adjusted this issue's tittle accordingly), but by the latest Sysbox's pre-release that we have been recently testing.

In Sysbox v0.5.2 (and previous releases) we were exporting slightly smaller sections of the /sys file-system within sysbox containers. For example, we were not exporting anything within the /sys/kernel/security branch:

root@ubu-jammy-systemd:/# ls -lrt /sys/kernel/security
total 0

As part of recent changes in sysbox-fs code, we started to expose /sys/kernel/security, as well as other nodes within the /sys/kernel hierarchy:

root@ubu-jammy-systemd:/# ls -lrt /sys/kernel/security
total 0
-r--r--r-- 1 nobody nogroup 0 Mar 31 15:13 lsm
-rw-r--r-- 1 nobody nogroup 0 Mar 31 15:13 lockdown
drwxr-xr-x 3 nobody nogroup 0 Mar 31 15:13 apparmor
drwxr-xr-x 4 nobody nogroup 0 Mar 31 15:13 integrity
drwxr-xr-x 2 nobody nogroup 0 Mar 31 15:13 ima
-rw-rw---- 1 nobody nogroup 0 Mar 31 15:13 evm

In retrospective, this was not a good idea since these nodes are not namespace'd, and as such, cannot be modified within a sysbox container. Moreover, by exposing these system-wide nodes within sysbox containers, we are sharing information that could be used by an attacker to extract security-related information to optimize hacking toolkits (e.g., loaded lsm modules, apparmor profiles in service, etc.).

As a result of these new nodes being exposed, dockerd is able to determine the presence of apparmor in the system (see code below), and in consequence attempts to load the docker-default profile mentioned above, which resulted in the error being observed.

func hostSupports() bool {
    checkAppArmor.Do(func() {
        // see https://github.com/opencontainers/runc/blob/0d49470392206f40eaab3b2190a57fe7bb3df458/libcontainer/apparmor/apparmor_linux.go
        if _, err := os.Stat("/sys/kernel/security/apparmor"); err == nil && os.Getenv("container") == "" {
            if _, err = os.Stat("/sbin/apparmor_parser"); err == nil {
                buf, err := os.ReadFile("/sys/module/apparmor/parameters/enabled")
                appArmorSupported = err == nil && len(buf) > 1 && buf[0] == 'Y'
            }
        }
    })
    return appArmorSupported
}

A potential fix for this issue is to simply hide all nodes in the /sys/kernel/security hierarchy.

rodnymolina commented 1 year ago

@rodnymolina is there a plan to handle it along with kubernetes 1.25 support or as a standalone bug ?

@grzesuav, yes, will generate a new pre-release image for you to try (which includes support for k8s 1.24-1.26). Will ping you when done.

grzesuav commented 1 year ago

@rodnymolina thanks!

Can I ask form where you build your sources ? As I have currently installed prerel version in my cluster and security team is not happy not knowing what changes are inside

rodnymolina commented 1 year ago

@grzesuav, forgot to answer your question above. We build both of our Sysbox installers (deb pkgs and sysbox-deploy-k8s daemonset) from a separate git repo that has been kept private till now. We will be exposing this repo publicly within the next few days.

rodnymolina commented 1 year ago

Issue has been fixed in Sysbox's v0.6.1 release. Closing now.