buildah throws lots of errors lsetfilecon Operation not supported

[llegolas]

Description

OS fedora 31; selinux enforcing/targeted; using the below script with buildah in rootles mode:

cat fedora-31-base.sh
#!/usr/bin/env bash
# Create a container
container=$(buildah from scratch)

# Mount the container filesystem
mountpoint=$(buildah mount $container)

# Install a basic filesystem and minimal set of packages
dnf install --installroot $mountpoint  --releasever 31 glibc-minimal-langpack bash coreutils omping iproute --setopt install_weak_deps=false -y
dnf clean all --installroot $mountpoint

buildah config  --cmd /bin/bash $container
buildah config --label name=fedora-31-base $container

# Save the container to an image
buildah commit $container fedora-31-base

# Cleanup
buildah unmount $container
buildah rm $container

$ buildah unshare bash fedora-31-base.sh
.......
.......
error: lsetfilecon: (/usr/share/man/man8/rtpr.8.gz;5dce3f3b, system_u:object_r:man_t:s0) Operation not supported
error: lsetfilecon: (/usr/share/man/man8/rtstat.8.gz;5dce3f3b, system_u:object_r:man_t:s0) Operation not supported
.....................................
...................................

Steps to reproduce the issue:

1. as normal user run buildah unshare bash fedora-31-base.sh

Describe the results you received: lots of lsetfilecon errors. the resulting image runs OK with podman on the same as far as I can tell.

Describe the results you expected: no errors

Output of rpm -q buildah or apt list buildah:

buildah-1.11.4-2.fc31.x86_64

Output of buildah version:

Version:         1.11.4
Go Version:      go1.13.1
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.0.0
Git Commit:      
Built:           Thu Jan  1 02:00:00 1970
OS/Arch:         linux/amd64

*Output of `cat /etc/release`:**

Fedora release 31 (Thirty One)

Output of uname -a:

Linux lylith.localdomain 5.3.11-300.fc31.x86_64 #1 SMP Tue Nov 12 19:08:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

cat /etc/containers/storage.conf |grep -v ^#
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
additionalimagestores = [
]
size = ""
mount_program = "/usr/bin/fuse-overlayfs"
override_kernel_check = "true"
mountopt = "nodev,metacopy=on"

[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"

[TomSweeneyRedHat]

@llegolas can you try your script with a slight modification? Please try remove bash from the last line: buildah unshare bash fedora-31-base.sh, so it would look like buildah unshare fedora-31-base.sh

@giuseppe any other thoughts?

[llegolas]

@TomSweeneyRedHat 'buildah unshare bash...' is not part of the script. This is how I invoke it. If i remove 'bash' from the command and call the script like

$  buildah unshare fedora-31-base.sh
ERRO exec: "fedora-31-base.sh": executable file not found in $PATH 
ERRO (unable to determine exit status)            
ERRO exit status 1                         

with bash like

$  buildah unshare bash fedora-31-base.sh
...
...
error: lsetfilecon: (/usr/share/crypto-policies/FIPS/libreswan.txt;5dd060dc, system_u:object_r:usr_t:s0) Operation not supported
error: lsetfilecon: (/usr/share/crypto-policies/FIPS/libssh.config;5dd060dc, system_u:object_r:usr_t:s0) Operation not supported
error: lsetfilecon: (/usr/share/crypto-policies/FIPS/libssh.txt;5dd060dc, system_u:object_r:usr_t:s0) Operation not supported
...

as reported

Ahh I see what you mean. Actually the whole heredoc part of the report is broken and the line you talk about indeed is not part of the script (and never was). I correct the initial post.

[TomSweeneyRedHat]

@llegolas thanks for the follow up, this makes a bit more sense now.

[llegolas]

Could it be problem with fuse-overlayfs? I think I managed to reproduce it w/o buildah unvoled at all

$ ls
lower  merged  upper  workdir
$ fuse-overlayfs -o debug,lowerdir=lower/,upperdir=upper/,workdir=workdir/,rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other merged/
$ chcon system_u:object_r:httpd_config_t:s0 merged/file
chcon: failed to change context of 'merged/file' to ‘system_u:object_r:httpd_config_t:s0’: Operation not supported
$

[rhatdan]

No this is not a file system issue. In the case of fuse-overlay, it is creating a fuse file system. Fuse file systems do not accept XATTRS so when you attempt to chcon on it, it gets an unsupported error since chcon can not write the XATTR.

Usually in containers we attempt to disable SELinux code, so that it does not attempt to set labels. But in the case of buildah mount and dnf install above, it is running on the host (Not in a container) and the dnf command sees that SELinux is enabled so it is attempting to lay down the SELinux labels.

Since fuse does not support them, you get the error.

I am wondering if their is a flag to dnf/rpm to tell it not to do selinux labeling.

[llegolas]

That's easy: just added --setopt=tsflags=nocontexts to the dnf line like that (removed the docs too):

dnf install --installroot $mountpoint  --releasever 31 --disablerepo=* --enablerepo=fedora glibc-minimal-langpack bash coreutils omping iproute --setopt=install_weak_deps=false --setopt=tsflags=nocontexts,nodocs -y

There are no errors now. It would be good if this gets documented somewhere.

I'm totally not sure why it worked back in the times of fedora 29. It could have been the storage driver but i can't remember for my live what the setup was before the reinstall.

[rhatdan]

I think this would work fine in Rootfull mode, but would not work in rootless. Since on Rootful, you are allowed to lay down labels on Overlayfs, but in rootless mode, you are using fuse-overlay, which would not allow labels.

[llegolas]

I just did some tests so to recap. The above script produces errors in both rootful and rootless mode with only difference being that in rootful it is executed w/o 'buildah unshare'. the Storage.conf is basically the same for root and ordinary user just some UID/GID remapping and different containers/storage location and is using fuse-overlayfs.

Modifying the script slightly. Adding to the 'dnf install' line

..... --setopt=tsflags=nocontexts...

fixes the behaviour as now no security contexts are tried to apply at all.

[giuseppe]

fuse-overfayfs support xattrs, but it looks like it doesn't even get the request for setting the xattrs.

Probably the request is blocked by SELinux before reaching the file system

[llegolas]

@giuseppe @rhatdan any ideas how can I try to repro? For instance I have no selinux alerts in the logs. Setting selinux mode from enforcing to permissive (setenforce 0) has no affect. I'll try disabling selinux altogether next.

[llegolas]

On centos 8 the above script runs (rootless and rootfull) w/o the lsetfilecon errors using yum instead for dnf of course. storage.config the same as on f31.

[llegolas]

I managed to narrow it a bit down. the script from above works fine on vanilla f29 and f30 (both updated to the latest as of today). It breaks with f31 booted with both f30 and f31 kernels kernel-core-5.3.11-100.fc29.x86_64 and kernel-core-5.3.11-300.fc31.x86_64 respectively.

Seem like problem is somewhere in userspace.

[rhatdan]

I would figure something changed in dnf/yum that is causing this. The labels will not be allowed to be applied, because the mount point has a label assigned to it. SELinux does not allow a label on a file system that that already has a label assigned via the mount context options.

merged type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c338,c1006",lowerdir=/var/lib/containers/storage/overlay/880de1b82361c780df9920d3b70546c4d868659661fc7f2ad2e026fe08d1b415/empty,upperdir=/var/lib/containers/storage/overlay/880de1b82361c780df9920d3b70546c4d868659661fc7f2ad2e026fe08d1b415/diff,workdir=/var/lib/containers/storage/overlay/880de1b82361c780df9920d3b70546c4d868659661fc7f2ad2e026fe08d1b415/work)

[rhatdan]

If this works in F30 and F31, then dnf/rpm was not doing the setfilecon in those situations. Everything ends up working, since the setfilecon is not a failure mode. Adding the --setopt=tsflags=nocontexts turns off the noice. I do not know who on the dnf team to ask, if something changed.

[rhatdan]

@llegolas is this still an issue?

[llegolas]

@rhatdan yes, it is still like that but as you said it has to do something with dnf/rpm. Maybe I can close this one and/or reference it in DNF github page? The no nocontexts trick works for me, yet what bugs me on is the lack of documentation around the change of the behaviour unless it is unintentional.

[rhatdan]

I believe this is fixed by always masking over /sys/fs/selinux, Reopen if I am mistaken.