Handling both read-only and writable volumes on NFS fails

[jskov-jyskebank-dk]

Description

Podman fails build when mounting volumes from folders on NFS with different options (writable and read-only).

This is when running in a rooless container on OpenShift (4.4.20).

Steps to reproduce the issue:

1. Create containerfile:

   $ cat > /tmp/Containerfile.bug <<EOF
   FROM alpine
   RUN env|sort
   EOF

2. Prepare scratch folders on a NFS drive, here /opt/persistence:

   sh-5.0$ mkdir -p /opt/persistence/ro /opt/persistence/rw

3. Try to build with volumes mounted as rw and ro (:Z,rw and :Z,ro):

   sh-5.0$ podman build --no-cache -v /opt/persistence/rw:/rw:Z -v /opt/persistence/ro:/ro:Z,ro -f /tmp/Containerfile.bug

Describe the results you received:

STEP 1: FROM alpine
STEP 2: RUN env|sort
error running subprocess: error remounting "/var/tmp/buildah174514365/mnt/rootfs/ro" in mount namespace with expected flags: permission denied
Error: error building at STEP "RUN env|sort": exit status 1

/opt/persistence in my OpenShift container is:

sh-5.0$ mount|grep persistence
192.168.0.129:/xxx_nfs4 on /opt/persistence type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.138.65.74,local_lock=none,addr=192.168.0.129)

Describe the results you expected:

It works fine when using folders on /tmp:

sh-5.0$ mkdir -p /tmp/rw /tmp/ro
sh-5.0$ podman build --no-cache -v /tmp/rw:/rw:Z -v /tmp/ro:/ro:Z,ro -f /tmp/Containerfile.bug
STEP 1: FROM alpine
STEP 2: RUN env|sort
HOME=/root
HOSTNAME=5916256815c7
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
TERM=xterm
STEP 3: COMMIT
--> 4e2d3192c4f
4e2d3192c4fabdbc0f764da70fb8b0db8bca5e880f0313f177cb848fbe201b1b

/tmp is mounted like this:

sh-5.0$ mount|grep "/ "
overlay on / type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c2,c28",lowerdir=/var/lib/containers/storage/overlay/l/JZ73GJS4T2ELO5WEGKPI6OXCDM:/var/lib/containers/storage/overlay/l/4LJBTSLFMULWX2DC7QTYQBDX3D:/var/lib/containers/storage/overlay/l/S5VJJD3RLBVBE6DP5CNAEQNWHM:/var/lib/containers/storage/overlay/l/5NDQMBFSVPNZBD27KQP562WZZO:/var/lib/containers/storage/overlay/l/46R4MZK7KE3LWDLKQZFORGFLP2:/var/lib/containers/storage/overlay/l/VDNSCQSSVK4EEA4YVFZHAGJVUE:/var/lib/containers/storage/overlay/l/YDXRKFEVVI7G4KQW4UOLMFT5KK:/var/lib/containers/storage/overlay/l/XUVJNZVI7HWKQXXUZUCFOFH6K6:/var/lib/containers/storage/overlay/l/UJDHUK7WVVNANRN2FFJY7PMD55:/var/lib/containers/storage/overlay/l/CEYFE5M7Y7AAY5SFEKJSUQKOGV:/var/lib/containers/storage/overlay/l/2VOU6TPIZ3Q6X63LLIIHSNHU6F:/var/lib/containers/storage/overlay/l/HERUR24LDI7MQFAK4HVFZ2EXLL:/var/lib/containers/storage/overlay/l/SCIQ7DONOXXT7MIP4FDAAQYCS6:/var/lib/containers/storage/overlay/l/KHI7ZXAJDWLFAEZTB7YTCKVT3B:/var/lib/containers/storage/overlay/l/JC3YKFRD6QW26FDGYAB3HSSRD6,upperdir=/var/lib/containers/storage/overlay/236b2f5269f07488c03f1d5950abf1baa55c6667f79b1ca3eba0209bd88f9760/diff,workdir=/var/lib/containers/storage/overlay/236b2f5269f07488c03f1d5950abf1baa55c6667f79b1ca3eba0209bd88f9760/work)

Output of rpm -q buildah or apt list buildah:

buildah-1.15.1-1.fc32.x86_64

Output of buildah version:

Version:         1.15.1
Go Version:      go1.14.6
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.5.1
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Built:        Tue Sep  1 19:26:51 2020
OS/Arch:      linux/amd64

*Output of `cat /etc/release`:**

Fedora release 32 (Thirty Two)
NAME=Fedora
VERSION="32 (Container Image)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
Fedora release 32 (Thirty Two)
Fedora release 32 (Thirty Two)

Output of uname -a:

Linux builder-service-bdcf6cdbf-9dxmm 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = false

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Size is used to set a maximum size of the container image.
# size = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[TomSweeneyRedHat]

@giuseppe or @rhatdan thoughts?

[rhatdan]

Did you see any AVC messages ausearch -m avc -ts recent

[jskov-jyskebank-dk]

Uh, @rhatdan, I run it in a container on an OpenShift platform. I do not have shell access on the host.

I do not assume you meant executed in the container? FWIW that fails (maybe I need to start some process?)

# ausearch -m avc -ts recent
Error opening /var/log/audit/audit.log (No such file or directory)

[rhatdan]

The AVC's should be in the journal if the audit.log does not exists.

journalctl -b | grep -i avc

[jskov-jyskebank-dk]

There is no audit.log file and no journal:

sh-5.0$ journalctl -b | grep -i avc
No journal files were found.

[rhatdan]

Could you try that with sudo?

[jskov-jyskebank-dk]

Same.

But as expected, surely? I run in a F32 container on OpenShift which just runs a python web server (as in https://github.com/containers/buildah/blob/master/docs/tutorials/05-openshift-rootless-bud.md). This happens in the Pod's terminal.

sh-5.0# journalctl -b | grep -i avc
No journal files were found.

sh-5.0# ls -lrt /var/log
total 492
drwxr-xr-x. 2 root root    235 May 15 06:48 anaconda
-rw-r--r--. 1 root root   1040 May 31 05:05 README
-rw-rw-r--. 1 root utmp      0 Jul 15 12:01 wtmp
-rw-------. 1 root root      0 Jul 15 12:01 tallylog
drwx------. 2 root root      6 Jul 15 12:01 private
-rw-rw----. 1 root utmp      0 Jul 15 12:01 btmp
-rw-r--r--. 1 root root  29885 Sep 22 05:52 dnf.librepo.log
-rw-r--r--. 1 root root   6473 Sep 22 05:53 dnf.rpm.log
-rw-r--r--. 1 root root    561 Sep 22 05:53 hawkey.log
-rw-r--r--. 1 root root  62030 Sep 22 05:53 dnf.log
-rw-rw-r--. 1 root utmp 292292 Sep 22 05:54 lastlog

[rhatdan]

This seems a little weird. The machine you running this on does not support journal?

sh-5.0# journalctl -b | grep -i avc
No journal files were found.

[jskov-jyskebank-dk]

It is a F32 image running on OpenShift. So it runs nothing but a dumb web server (to keep the Pod alive) and my terminal.

The host - the OpenShift platform - is not something I have access to.

I can try to reach out to those who run the platform...

[rhatdan]

@jskovjyskebankdk Did you ever get any further with this? I don't believe this is a buildah issue per-se but something to do with the environment that it is running in.

[jskov-jyskebank-dk]

Sorry.

I thought so, but must have forgotten, because I can find no mails about it.

I have just reproduced with Podman:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

on OCP 4.6.13

I will send that mail to the admins.

[jskov-jyskebank-dk]

Data from the admins. Hope this tells you something @rhatdan

# ausearch -m avc -ts recent
----
time->Thu Feb 11 07:17:33 2021
type=AVC msg=audit(1613024253.291:12420): avc:  denied  { remount } for  pid=2064839 comm="5" scontext=system_u:system_r:container_t:s0:c2,c28 tcontext=system_u:object_r:nfs_t:s0 tclass=filesystem permissive=0

# cat /etc/redhat-release
Red Hat Enterprise Linux CoreOS release 4.6
# uname -a
Linux XXX 4.18.0-193.40.1.el8_2.x86_64 #1 SMP Wed Jan 6 10:54:57 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

[rhatdan]

Well that should be allowed by SELinux, Fixed in upstream.

https://github.com/containers/container-selinux/releases/tag/v2.158.0

Should end up being in RHEL8.4 release.

[jskov-jyskebank-dk]

Awesome! Cheers!