Provide AppArmor profile for running inside docker container

[michaelzangl]

Description

When running buildah inside a docker container (Gitlab CI with docker runner, default setup), I had to disable apparmor for this docker container. I would like to just set a more loose profile. I could not find any information on the profile required for buildah (for seccomp, I could find this information)

Background: The docker runner is a privileged docker container, so no issues there. It starts unprivileged containers that then run each build step. In this container, building a docker image is only possible if one does not use the default apparmor profile for that container.

Steps to reproduce the issue: (Can probably be done on plain docker without gitlab, but this is what I used)

1. Install docker, gitlab CI, the docker runner
2. Start the docker runner once => it will create it's default config.toml
3. Modify the config.toml (it is mounted into the docker runner container and defines the parameters for starting a new CI task)
   • I want to use the seccomp profile that podman provides
   • I have to disable apparmor (← this is what I want to avoid)
   • So basically I added this line to the [runners.docker] section.
   • security_opt = ['apparmor:unconfined', 'seccomp:{ "defaultAction": "SCMP_ACT_ERRNO", "archMap": [ { "architecture": "SCMP_ARCH_X86_64", "subArchitectures": [ "SCMP_ARCH_X86", "SCMP_ARCH_X32" ] }, { "architecture": "SCMP_ARCH_AARCH64", "subArchitectures": [ "SCMP_ARCH_ARM" ] }, { "architecture": "SCMP_ARCH_MIPS64", "subArchitectures": [ "SCMP_ARCH_MIPS", "SCMP_ARCH_MIPS64N32" ] }, { "architecture": "SCMP_ARCH_MIPS64N32", "subArchitectures": [ "SCMP_ARCH_MIPS", "SCMP_ARCH_MIPS64" ] }, { "architecture": "SCMP_ARCH_MIPSEL64", "subArchitectures": [ "SCMP_ARCH_MIPSEL", "SCMP_ARCH_MIPSEL64N32" ] }, { "architecture": "SCMP_ARCH_MIPSEL64N32", "subArchitectures": [ "SCMP_ARCH_MIPSEL", "SCMP_ARCH_MIPSEL64" ] }, { "architecture": "SCMP_ARCH_S390X", "subArchitectures": [ "SCMP_ARCH_S390" ] } ], "syscalls": [ { "names": [ "_llseek", "_newselect", "accept", "accept4", "access", "adjtimex", "alarm", "bind", "brk", "capget", "capset", "chdir", "chmod", "chown", "chown32", "clock_adjtime", "clock_adjtime64", "clock_getres", "clock_getres_time64", "clock_gettime", "clock_gettime64", "clock_nanosleep", "clock_nanosleep_time64", "clone", "clone3", "close", "close_range", "connect", "copy_file_range", "creat", "dup", "dup2", "dup3", "epoll_create", "epoll_create1", "epoll_ctl", "epoll_ctl_old", "epoll_pwait", "epoll_wait", "epoll_wait_old", "eventfd", "eventfd2", "execve", "execveat", "exit", "exit_group", "faccessat", "faccessat2", "fadvise64", "fadvise64_64", "fallocate", "fanotify_mark", "fchdir", "fchmod", "fchmodat", "fchown", "fchown32", "fchownat", "fcntl", "fcntl64", "fdatasync", "fgetxattr", "flistxattr", "flock", "fork", "fremovexattr", "fsetxattr", "fstat", "fstat64", "fstatat64", "fstatfs", "fstatfs64", "fsync", "ftruncate", "ftruncate64", "futex", "futimesat", "get_robust_list", "get_thread_area", "getcpu", "getcwd", "getdents", "getdents64", "getegid", "getegid32", "geteuid", "geteuid32", "getgid", "getgid32", "getgroups", "getgroups32", "getitimer", "getpeername", "getpgid", "getpgrp", "getpid", "getppid", "getpriority", "getrandom", "getresgid", "getresgid32", "getresuid", "getresuid32", "getrlimit", "getrusage", "getsid", "getsockname", "getsockopt", "gettid", "gettimeofday", "getuid", "getuid32", "getxattr", "inotify_add_watch", "inotify_init", "inotify_init1", "inotify_rm_watch", "io_cancel", "io_destroy", "io_getevents", "io_setup", "io_submit", "ioctl", "ioprio_get", "ioprio_set", "ipc", "keyctl", "kill", "lchown", "lchown32", "lgetxattr", "link", "linkat", "listen", "listxattr", "llistxattr", "lremovexattr", "lseek", "lsetxattr", "lstat", "lstat64", "madvise", "memfd_create", "mincore", "mkdir", "mkdirat", "mknod", "mknodat", "mlock", "mlock2", "mlockall", "mmap", "mmap2", "mount", "mprotect", "mq_getsetattr", "mq_notify", "mq_open", "mq_timedreceive", "mq_timedsend", "mq_unlink", "mremap", "msgctl", "msgget", "msgrcv", "msgsnd", "msync", "munlock", "munlockall", "munmap", "name_to_handle_at", "nanosleep", "newfstatat", "open", "openat", "openat2", "pause", "pidfd_getfd", "pidfd_open", "pidfd_send_signal", "pipe", "pipe2", "pivot_root", "poll", "ppoll", "ppoll_time64", "prctl", "pread64", "preadv", "preadv2", "prlimit64", "pselect6", "pselect6_time64", "pwrite64", "pwritev", "pwritev2", "read", "readahead", "readlink", "readlinkat", "readv", "reboot", "recv", "recvfrom", "recvmmsg", "recvmsg", "remap_file_pages", "removexattr", "rename", "renameat", "renameat2", "restart_syscall", "rmdir", "rt_sigaction", "rt_sigpending", "rt_sigprocmask", "rt_sigqueueinfo", "rt_sigreturn", "rt_sigsuspend", "rt_sigtimedwait", "rt_tgsigqueueinfo", "sched_get_priority_max", "sched_get_priority_min", "sched_getaffinity", "sched_getattr", "sched_getparam", "sched_getscheduler", "sched_rr_get_interval", "sched_setaffinity", "sched_setattr", "sched_setparam", "sched_setscheduler", "sched_yield", "seccomp", "select", "semctl", "semget", "semop", "semtimedop", "send", "sendfile", "sendfile64", "sendmmsg", "sendmsg", "sendto", "set_robust_list", "set_thread_area", "set_tid_address", "setfsgid", "setfsgid32", "setfsuid", "setfsuid32", "setgid", "setgid32", "setgroups", "setgroups32", "setitimer", "setpgid", "setpriority", "setregid", "setregid32", "setresgid", "setresgid32", "setresuid", "setresuid32", "setreuid", "setreuid32", "setrlimit", "setsid", "setsockopt", "setuid", "setuid32", "setxattr", "shmat", "shmctl", "shmdt", "shmget", "shutdown", "sigaltstack", "signalfd", "signalfd4", "sigreturn", "socket", "socketcall", "socketpair", "splice", "stat", "stat64", "statfs", "statfs64", "statx", "symlink", "symlinkat", "sync", "sync_file_range", "syncfs", "sysinfo", "syslog", "tee", "tgkill", "time", "timer_create", "timer_delete", "timer_getoverrun", "timer_gettime", "timer_gettime64", "timer_settime", "timerfd_create", "timerfd_gettime", "timerfd_gettime64", "timerfd_settime", "timerfd_settime64", "times", "tkill", "truncate", "truncate64", "ugetrlimit", "umask", "umount", "umount2", "uname", "unlink", "unlinkat", "unshare", "utime", "utimensat", "utimensat_time64", "utimes", "vfork", "vmsplice", "wait4", "waitid", "waitpid", "write", "writev" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 0, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 8, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 131072, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 131080, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 4294967295, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": {} }, { "names": [ "sync_file_range2" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "arches": [ "ppc64le" ] }, "excludes": {} }, { "names": [ "arm_fadvise64_64", "arm_sync_file_range", "sync_file_range2", "breakpoint", "cacheflush", "set_tls" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "arches": [ "arm", "arm64" ] }, "excludes": {} }, { "names": [ "arch_prctl" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "arches": [ "amd64", "x32" ] }, "excludes": {} }, { "names": [ "modify_ldt" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "arches": [ "amd64", "x32", "x86" ] }, "excludes": {} }, { "names": [ "s390_pci_mmio_read", "s390_pci_mmio_write", "s390_runtime_instr" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "arches": [ "s390", "s390x" ] }, "excludes": {} }, { "names": [ "open_by_handle_at" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_DAC_READ_SEARCH" ] }, "excludes": {} }, { "names": [ "bpf", "clone", "fanotify_init", "lookup_dcookie", "mount", "name_to_handle_at", "perf_event_open", "quotactl", "setdomainname", "sethostname", "setns", "umount", "umount2", "unshare" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_ADMIN" ] }, "excludes": {} }, { "names": [ "clone" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 2080505856, "valueTwo": 0, "op": "SCMP_CMP_MASKED_EQ" } ], "comment": "", "includes": {}, "excludes": { "caps": [ "CAP_SYS_ADMIN" ], "arches": [ "s390", "s390x" ] } }, { "names": [ "clone" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 1, "value": 2080505856, "valueTwo": 0, "op": "SCMP_CMP_MASKED_EQ" } ], "comment": "s390 parameter ordering for clone is different", "includes": { "arches": [ "s390", "s390x" ] }, "excludes": { "caps": [ "CAP_SYS_ADMIN" ] } }, { "names": [ "reboot" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_BOOT" ] }, "excludes": {} }, { "names": [ "chroot" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_CHROOT" ] }, "excludes": {} }, { "names": [ "delete_module", "init_module", "finit_module", "query_module" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_MODULE" ] }, "excludes": {} }, { "names": [ "get_mempolicy", "mbind", "name_to_handle_at", "set_mempolicy" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_NICE" ] }, "excludes": {} }, { "names": [ "acct" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_PACCT" ] }, "excludes": {} }, { "names": [ "kcmp", "process_vm_readv", "process_vm_writev", "ptrace" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_PTRACE" ] }, "excludes": {} }, { "names": [ "iopl", "ioperm" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_RAWIO" ] }, "excludes": {} }, { "names": [ "settimeofday", "stime", "clock_settime", "clock_settime64" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_TIME" ] }, "excludes": {} }, { "names": [ "vhangup" ], "action": "SCMP_ACT_ALLOW", "args": [], "comment": "", "includes": { "caps": [ "CAP_SYS_TTY_CONFIG" ] }, "excludes": {} }, { "names": [ "socket" ], "action": "SCMP_ACT_ERRNO", "args": [ { "index": 0, "value": 16, "valueTwo": 0, "op": "SCMP_CMP_EQ" }, { "index": 2, "value": 9, "valueTwo": 0, "op": "SCMP_CMP_EQ" } ], "comment": "", "includes": {}, "excludes": { "caps": [ "CAP_AUDIT_WRITE" ] }, "errnoRet": 22 }, { "names": [ "socket" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 2, "value": 9, "valueTwo": 0, "op": "SCMP_CMP_NE" } ], "comment": "", "includes": {}, "excludes": { "caps": [ "CAP_AUDIT_WRITE" ] } }, { "names": [ "socket" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 16, "valueTwo": 0, "op": "SCMP_CMP_NE" } ], "comment": "", "includes": {}, "excludes": { "caps": [ "CAP_AUDIT_WRITE" ] } }, { "names": [ "socket" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 2, "value": 9, "valueTwo": 0, "op": "SCMP_CMP_NE" } ], "comment": "", "includes": {}, "excludes": { "caps": [ "CAP_AUDIT_WRITE" ] } }, { "names": [ "socket" ], "action": "SCMP_ACT_ALLOW", "args": null, "comment": "", "includes": { "caps": [ "CAP_AUDIT_WRITE" ] }, "excludes": {} } ] }']
4. Create a new Project with the Dockerfile and this .gitlab-ci.yml

   
   image: quay.io/buildah/stable

build: stage: build script:

• export _BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot
• export STORAGE_DRIVER=vfs
• buildah --version
• echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
• buildah -t image1 bud .
• buildah push image1 "docker://$CI_REGISTRY_IMAGE"

Describe the results you received:

When running with default AppArmor profile:

level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"

When running with apparmor:unconfined: No error, but no apparmor active

Describe the results you expected:

Find a apparmor profile that I can use instead of disabeling apparmor completely.

Output of buildah version (container):

(using latest buildah container of 2020-02-25)

buildah version 1.18.0 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)

Output of uname -a (host):

Linux .... 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of dockerd --version (host):

Docker version 18.09.1, build 4c52b90

[rhatdan]

Are you running buildah bud --isolation=chroot?

This does less risky stuff, and is justified since you are running buildah within a container.

quay.io/buildah/stable has an image that we encourage users to use when running buildah inside of a container.

[rhatdan]

I know little about what this is breaking in apparmor. I believe there are some seccomp issues to get this to work as well, if your docker daemon is enforcing seccomp.

Works well with podman or using the /usr/share/containers/seccomp.json file that ships with buildah.

[michaelzangl]

@rhatdan Tanks for the comments. Yes, I use --isolation=chroot and I have adjusted the seccomp profile to the one I included above.

The only thing missing is an apparmor profile. For now, it runs without apparmor - that works, but restricting a bit more would not hurt.

[rhatdan]

Can't you just run the docker container with --security-opt apparmor:unconfined for the buildah container? BTW What is the apparmor error when you attempt this? Finally if you run the same buildah container within podman, does it work with apparmor enabled?

[michaelzangl]

@rhatdan Yes, I am running it with apparmor:unconfined to work around this.

The Idea behind apparmor is to define a set of things an application may do. Although there are some standard profiles, it is best to define a profile that is specific to the application and as restrictive as possible, but should still allow all functionality to be used.

[rhatdan]

Yes I am familiar with Apparmor, having worked on SELinux for 20 years. :^) My question is what blows up if you don't run with the apparmor:unconfined flag? Are there messages in the /var/log/audit/audit.log indicating what is denied.

With SELinux we lock container into its sandbox. Apparmor attempts to control what is going on inside of the sandbox, But the goal of both is to prevent privilege escalation.

[HubbeKing]

I suspect the issue here is the same as here: https://flavio.castelli.me/2020/09/16/build-multi-architecture-container-images-using-kubernetes/

In other words, docker is likely using an AppArmor profile that denies mount, much like podman does.

I'm also running into this same error trying to use buildah in the drone.io kubernetes runner, where disabling apparmor is rather less trivial.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@saschagrunert Any chance you could look at this? Basically allow us to provide apparmor policy outside of the builtin?

[saschagrunert]

@saschagrunert Any chance you could look at this? Basically allow us to provide apparmor policy outside of the builtin?

To me it looks like that we can provide an AppArmor profile for running buildah inside of a container. The distribution would have to package that profile and ship it, whereas admins would have to configure their runtimes to consume that profile.

Do we want to maintain such a profile within this repository? I'm not sure about this.

[michaelzangl]

@saschagrunert I would see it as part of the documentation.

[saschagrunert]

@saschagrunert I would see it as part of the documentation.

Sounds good to me, are you planning to contribute those docs? :)

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we received no more data, I am going to close this issue, reopen if more work is required.

[michaelzangl]

To anyone using this config to run gitlab + buildah: If you receive the error:

runtime/cgo: pthread_create failed: Operation not permitted

This is caused by a change in recent glibc. You need to add "clone3" to the list of allowed system calls - I updated the config above to what I am using now.

[rhatdan]

What version of Buildah and containers-common are you seeing this with?

[michaelzangl]

I use the docker image quay.io/buildah/stable

$ buildah --version
buildah version 1.23.1 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)

[rhatdan]

We need someone with apparmor experience to contribute patchs/documentation to fix this. As an SELinux guy, I am have no idea how to fix. :^)

[michaelzangl]

Just don't.

It needs to be fixed by anyone running buildah in a container. Those are the container restrictions for the buildah container. Podman provides the profile somewhere in their docs, they should be adjusted.