podman w runc keycreate invalid argument

[smijolovic]

podman 2.1.1 (built with selinux buildtag) runc 1.0.0-rc92 (built with selinux buildtag)

Running into the: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: failed to set /proc/self/attr/keycreate on procfs: write /proc/self/attr/keycreate: invalid argument

error...which I believe was once addressed in the past but seems to be present in Ubuntu (running 20.04.1 LTS with GA 5.4.0-64-generic #72-Ubuntu SMP kernel - which should have the kernel patch on keycreate from 5.3 upstream linux kernel).

STEP 5: RUN apt-get update && apt-get dist-upgrade -y DEBU[0007] RUN imagebuilder.Run{Shell:true, Args:[]string{"apt-get update && apt-get dist-upgrade -y"}}, docker.Config{Hostname:"", Domainname:"", User:"", Memory:0, MemorySwap:0, MemoryReservation:0, KernelMemory:0, CPUShares:0, CPUSet:"", PortSpecs:[]string(nil), ExposedPorts:map[docker.Port]struct {}{}, PublishService:"", StopSignal:"", StopTimeout:0, Env:[]string{"DEBIAN_FRONTEND=noninteractive", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "", "DEBIAN_FRONTEND=noninteractive"}, Cmd:[]string{"bash"}, Shell:[]string{}, Healthcheck:(*docker.HealthConfig)(nil), DNS:[]string(nil), Image:"", Volumes:map[string]struct {}{}, VolumeDriver:"", WorkingDir:"", MacAddress:"", Entrypoint:[]string{}, SecurityOpts:[]string(nil), OnBuild:[]string{}, Mounts:[]docker.Mount(nil), Labels:map[string]string{"io.buildah.version":"1.19.2"}, AttachStdin:false, AttachStdout:false, AttachStderr:false, ArgsEscaped:false, Tty:false, OpenStdin:false, StdinOnce:false, NetworkDisabled:false, VolumesFrom:""} DEBU[0007] using "/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356" to hold bundle data DEBU[0007] Resources: &buildah.CommonBuildOptions{AddHost:[]string{}, CgroupParent:"", CPUPeriod:0x0, CPUQuota:0, CPUShares:0x0, CPUSetCPUs:"", CPUSetMems:"", HTTPProxy:true, Memory:0, DNSSearch:[]string(nil), DNSServers:[]string(nil), DNSOptions:[]string(nil), MemorySwap:0, LabelOpts:[]string(nil), OmitTimestamp:false, SeccompProfilePath:"", ApparmorProfile:"", ShmSize:"65536k", Ulimit:[]string{}, Volumes:[]string{}} DEBU[0007] stdio is a terminal, defaulting to using a terminal DEBU[0007] ensuring working directory "/home/chaasmadmin/.local/share/containers/storage/overlay/8d72ecbc0511c49fdcde78e7f105710cab5337325d86f5226d7f0afb5c448a08/merged" exists DEBU[0000] bind mounted "/home/chaasmadmin/.local/share/containers/storage/overlay/8d72ecbc0511c49fdcde78e7f105710cab5337325d86f5226d7f0afb5c448a08/merged" to "/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/mnt/rootfs" DEBU[0000] config = {"ociVersion":"1.0.2-dev","process":{"terminal":true,"user":{"uid":0,"gid":0},"args":["/bin/sh","-c","apt-get update \u0026\u0026 apt-get dist-upgrade -y"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","TERM=xterm","DEBIAN_FRONTEND=noninteractive","HOSTNAME=44935a23928e"],"cwd":"/","capabilities":{"bounding":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"effective":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"inheritable":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"permitted":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"ambient":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"]},"rlimits":[{"type":"RLIMIT_NOFILE","hard":1024,"soft":1024}],"selinuxLabel":"system_u:system_r:container_t:s0:c75,c162"},"root":{"path":"/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/mnt/rootfs"},"hostname":"44935a23928e","mounts":[{"destination":"/run/.containerenv","type":"bind","source":"/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/run/.containerenv","options":["rbind"]},{"destination":"/etc/hosts","type":"bind","source":"/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/hosts","options":["rbind"]},{"destination":"/etc/resolv.conf","type":"bind","source":"/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/resolv.conf","options":["rbind"]},{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","strictatime","mode=755","size=65536k"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["nosuid","noexec","nodev","mode=1777","size=65536k"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/sys","type":"bind","source":"/sys","options":["rbind","nosuid","noexec","nodev","ro"]}],"linux":{"resources":{"devices":[{"allow":false,"access":"rwm"}]},"namespaces":[{"type":"pid"},{"type":"ipc"},{"type":"uts"},{"type":"mount"}],"seccomp":{"defaultAction":"SCMP_ACT_ERRNO","architectures":["SCMP_ARCH_X86_64","SCMP_ARCH_X86","SCMP_ARCH_X32"],"syscalls":[{"names":["_llseek","_newselect","accept","accept4","access","adjtimex","alarm","bind","brk","capget","capset","chdir","chmod","chown","chown32","clock_adjtime","clock_getres","clock_gettime","clock_nanosleep","clone","close","connect","copy_file_range","creat","dup","dup2","dup3","epoll_create","epoll_create1","epoll_ctl","epoll_ctl_old","epoll_pwait","epoll_wait","epoll_wait_old","eventfd","eventfd2","execve","execveat","exit","exit_group","faccessat","faccessat2","fadvise64","fadvise64_64","fallocate","fanotify_mark","fchdir","fchmod","fchmodat","fchmodat2","fchown","fchown32","fchownat","fcntl","fcntl64","fdatasync","fgetxattr","flistxattr","flock","fork","fremovexattr","fsetxattr","fstat","fstat64","fstatat64","fstatfs","fstatfs64","fsync","ftruncate","ftruncate64","futex","futimesat","get_robust_list","get_thread_area","getcpu","getcwd","getdents","getdents64","getegid","getegid32","geteuid","geteuid32","getgid","getgid32","getgroups","getgroups32","getitimer","getpeername","getpgid","getpgrp","getpid","getppid","getpriority","getrandom","getresgid","getresgid32","getresuid","getresuid32","getrlimit","getrusage","getsid","getsockname","getsockopt","gettid","gettimeofday","getuid","getuid32","getxattr","inotify_add_watch","inotify_init","inotify_init1","inotify_rm_watch","io_cancel","io_destroy","io_getevents","io_setup","io_submit","ioctl","ioprio_get","ioprio_set","ipc","kill","lchown","lchown32","lgetxattr","link","linkat","listen","listxattr","llistxattr","lremovexattr","lseek","lsetxattr","lstat","lstat64","madvise","memfd_create","mincore","mkdir","mkdirat","mknod","mknodat","mlock","mlock2","mlockall","mmap","mmap2","mount","mprotect","mq_getsetattr","mq_notify","mq_open","mq_timedreceive","mq_timedsend","mq_unlink","mremap","msgctl","msgget","msgrcv","msgsnd","msync","munlock","munlockall","munmap","name_to_handle_at","nanosleep","newfstatat","open","openat","openat2","pause","pipe","pipe2","pivot_root","poll","ppoll","prctl","pread64","preadv","preadv2","prlimit64","pselect6","pwrite64","pwritev","pwritev2","read","readahead","readlink","readlinkat","readv","reboot","recv","recvfrom","recvmmsg","recvmsg","remap_file_pages","removexattr","rename","renameat","renameat2","restart_syscall","rmdir","rt_sigaction","rt_sigpending","rt_sigprocmask","rt_sigqueueinfo","rt_sigreturn","rt_sigsuspend","rt_sigtimedwait","rt_tgsigqueueinfo","sched_get_priority_max","sched_get_priority_min","sched_getaffinity","sched_getattr","sched_getparam","sched_getscheduler","sched_rr_get_interval","sched_setaffinity","sched_setattr","sched_setparam","sched_setscheduler","sched_yield","seccomp","select","semctl","semget","semop","semtimedop","send","sendfile","sendfile64","sendmmsg","sendmsg","sendto","set_robust_list","set_thread_area","set_tid_address","setfsgid","setfsgid32","setfsuid","setfsuid32","setgid","setgid32","setgroups","setgroups32","setitimer","setpgid","setpriority","setregid","setregid32","setresgid","setresgid32","setresuid","setresuid32","setreuid","setreuid32","setrlimit","setsid","setsockopt","setuid","setuid32","setxattr","shmat","shmctl","shmdt","shmget","shutdown","sigaltstack","signalfd","signalfd4","sigreturn","socketcall","socketpair","splice","stat","stat64","statfs","statfs64","statx","symlink","symlinkat","sync","sync_file_range","syncfs","sysinfo","syslog","tee","tgkill","time","timer_create","timer_delete","timer_getoverrun","timer_gettime","timer_settime","timerfd_create","timerfd_gettime","timerfd_settime","times","tkill","truncate","truncate64","ugetrlimit","umask","umount","umount2","uname","unlink","unlinkat","unshare","utime","utimensat","utimes","vfork","vmsplice","wait4","waitid","waitpid","write","writev"],"action":"SCMP_ACT_ALLOW"},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":0,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":8,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131072,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131080,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":4294967295,"op":"SCMP_CMP_EQ"}]},{"names":["arch_prctl"],"action":"SCMP_ACT_ALLOW"},{"names":["modify_ldt"],"action":"SCMP_ACT_ALLOW"},{"names":["clone"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":2080505856,"op":"SCMP_CMP_MASKED_EQ"}]},{"names":["chroot"],"action":"SCMP_ACT_ALLOW"},{"names":["socket"],"action":"SCMP_ACT_ERRNO","errnoRet":22,"args":[{"index":0,"value":16,"op":"SCMP_CMP_EQ"},{"index":2,"value":9,"op":"SCMP_CMP_EQ"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":2,"value":9,"op":"SCMP_CMP_NE"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":16,"op":"SCMP_CMP_NE"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":2,"value":9,"op":"SCMP_CMP_NE"}]}]},"maskedPaths":["/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/fs/selinux","/sys/dev"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"],"mountLabel":"system_u:object_r:container_file_t:s0:c75,c162"}} DEBU[0000] Running ["/usr/sbin/runc" "create" "--bundle" "/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356" "--pid-file" "/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/pid" "--console-socket" "/home/chaasmadmin/nimbus8/kube19/kubernetes-chaasm-1.19.7/tmp/buildah742050356/console.sock" "buildah-buildah742050356"] WARN[0000] signal: killed
ERRO[0000] container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: failed to set /proc/self/attr/keycreate on procfs: write /proc/self/attr/keycreate: invalid argument error running container: error creating container for [/bin/sh -c apt-get update && apt-get dist-upgrade -y]: : exit status 1 DEBU[0007] error building at step {Env:[DEBIAN_FRONTEND=noninteractive PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DEBIAN_FRONTEND=noninteractive] Command:run Args:[apt-get update && apt-get dist-upgrade -y] Flags:[] Attrs:map[] Message:RUN apt-get update && apt-get dist-upgrade -y Original:RUN apt-get update && apt-get dist-upgrade -y}: error while running runtime: exit status 1 Error: error building at STEP "RUN apt-get update && apt-get dist-upgrade -y": error while running runtime: exit status 1

Any ideas on what might be causing runc to fail? I have not seen this issue in CentOS 8.

[rhatdan]

Looks like the kernel you are using does not have the /proc/self/attr/keycreate selinux key in the kernel. Is this an older kernel on RHEL8 then on Centos8?

[smijolovic]

5.4.0-64-generic #72-Ubuntu from Ubuntu 20.04.1 LTS. The kernel in CentOS8 was the 4.18.0 kernel

I assume you are referring to this?: https://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux.git/commit/?id=464c258aa45b09f16aa0f05847ed8895873262d9

[rhatdan]

Yup

[smijolovic]

Thank you for the pointer. Will bring this up with canonical this week. Did a strace to look for more evidence....

[pid 120566] openat(AT_FDCWD, "/proc/self/attr/keycreate", O_WRONLY|O_CLOEXEC) = 6 [pid 120566] epoll_ctl(7, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2235174584,u64=140520680197816}}) = -1 EPERM (Operation not permitted) [pid 120566] epoll_ctl(7, EPOLL_CTL_DEL, 6, 0xc0000e0dfc) = -1 EPERM (Operation not permitted) [pid 120566] fstatfs(6, {f_type=PROC_SUPER_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0 [pid 120566] write(6, "system_u:system_r:container_t:s0"..., 42) = -1 EINVAL (Invalid argument) [pid 120566] close(6) = 0

[rhatdan]

Since this is an issue with Ubuntu, I am going to close this issue.