search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.37k stars 781 forks source link

rootless "podman build" fails when run under an agent systemd service (Jenkins/Azure Devops/TFS) #1872

Closed scorpionlion closed 4 years ago

scorpionlion commented 5 years ago

Description rootless "podman build" fails when run under an agent systemd service (Jenkins/Azure Devops/TFS)

Steps to reproduce the issue:

  1. Run commands

    sudo setsebool -P container_manage_cgroup on
sysctl user.max_user_namespaces=28633
sudo echo [user]:100000:65536 >>/etc/subuid
sudo echo [user]:100000:65536 >>/etc/subgid
yum install podman slirp4netns -y

  2. /home/[user]/docker/DockerFile

    FROM registry.redhat.io/rhel7/rhel:7.6
USER 0
RUN yum install telnet -y

  3. /usr/bin/runpodman.sh

    #!/usr/bin/bash
podman  build --no-cache -f "/home/[user]/docker/Dockerfile" "/home/[user]/docker/"

  4. /usr/lib/systemd/system/podmanbuild.service The service example is isolated as a simple service, but in real the service is the agent service 

    
[Unit]
Description=Podman call from service

[Service] ExecStart=/usr/bin/runpodman.sh User=[user]

[Install] WantedBy=default.target


4. _Run commands_

sudo chmod +x /usr/bin/runpodman.sh sudo systemctl daemon-reload sudo systemctl start podmanbuild.service sudo journalctl -u podmanbuild.service


**Describe the results you received:**
I get the below error.

systemd[1]: Started Podman call from service. runpodman.sh[128699]: STEP 1: FROM registry.redhat.io/rhel7/rhel:7.6 runpodman.sh[128699]: STEP 2: USER 0 runpodman.sh[128699]: 45fbc4c388592a6c56bbb9da14dc29b1bc9f1a39182f3f091a9292b1a69ea1be runpodman.sh[128699]: STEP 3: RUN yum install telnet -y runpodman.sh[128699]: standard_init_linux.go:211: exec user process caused "permission denied" runpodman.sh[128699]: Error: error building at STEP "RUN yum install telnet -y": error while running runtime: exit status 1 systemd[1]: podmantest.service: main process exited, code=exited, status=125/n/a systemd[1]: Unit podmantest.service entered failed state. systemd[1]: podmantest.service failed.

**Describe the results you expected:**
telnet installed inside the image, and image built successfully. 
1. If I log in as the same [user] interactively in console and run /usr/bin/runpodman.sh  then I successfully manage to create the  image
2. If as part of service (ExecStart) I call the command present in runpodman.sh directly then also the image gets built successfully.
3. when setenforfce is disabled ( 0 ), then also the image gets built successfully.

Note: It fails on any command which requires exec permissions not just yum install.

**Output of `rpm -q buildah` or `apt list buildah`:**

buildah-1.9.0-2.el7.x86_64


**Output of `buildah version`:**

Version: 1.9.0 Go Version: go1.10.8 Image Spec: 1.0.0 Runtime Spec: 1.0.0 CNI Spec: 0.4.0 libcni Version: Git Commit: Built: Thu Jan 1 00:00:00 1970 OS/Arch: linux/amd64


**Output of `podman version` if reporting a `podman build` issue:**

Version: 1.4.4 RemoteAPI Version: 1 Go Version: go1.10.3 OS/Arch: linux/amd64


**Output of `cat /etc/*release`:**

NAME="Red Hat Enterprise Linux Server" VERSION="7.7 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.7" PRETTY_NAME="OpenShift Enterprise" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.7" Red Hat Enterprise Linux Server release 7.7 (Maipo) Red Hat Enterprise Linux Server release 7.7 (Maipo)


**Output of `uname -a`:**

Linux DVDOBLDLAZ01 3.10.0-1062.1.1.el7.x86_64 #1 SMP Tue Aug 13 18:39:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


**Output of `cat /etc/containers/storage.conf`:**

[storage.options]

Storage options to be passed to underlying storage drivers

AdditionalImageStores is used to pass paths to additional Read/Only image stores

Must be comma separated list.

additionalimagestores = [ ]

Size is used to set a maximum size of the container image. Only supported by

certain container storage drivers.

size = ""

OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version

override_kernel_check = "true"

Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of

a container, to UIDs/GIDs as they should appear outside of the container, and

the length of the range of UIDs/GIDs. Additional mapped sets can be listed

and will be heeded by libraries, but there are limits to the number of

mappings which the kernel will allow when you later attempt to run a

container.

#

remap-uids = 0:1668442479:65536

remap-gids = 0:1668442479:65536

Remap-User/Group is a name which can be used to look up one or more UID/GID

ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting

with an in-container ID of 0 and the a host-level ID taken from the lowest

range that matches the specified name, and using the length of that range.

Additional ranges are then assigned, using the ranges which specify the

lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,

until all of the entries have been used for maps.

#

remap-user = "storage"

remap-group = "storage"

[storage.options.thinpool]

Storage Options for thinpool

autoextend_percent determines the amount by which pool needs to be

grown. This is specified in terms of % of pool size. So a value of 20 means

that when threshold is hit, pool will be grown by 20% of existing

pool size.

autoextend_percent = "20"

autoextend_threshold determines the pool extension threshold in terms

of percentage of pool size. For example, if threshold is 60, that means when

pool is 60% full, threshold has been hit.

autoextend_threshold = "80"

basesize specifies the size to use when creating the base device, which

limits the size of images and containers.

basesize = "10G"

blocksize specifies a custom blocksize to use for the thin pool.

blocksize="64k"

directlvm_device specifies a custom block storage device to use for the

thin pool. Required if you setup devicemapper

directlvm_device = ""

directlvm_device_force wipes device even if device already has a filesystem

directlvm_device_force = "True"

fs specifies the filesystem type to use for the base device.

fs="xfs"

log_level sets the log level of devicemapper.

0: LogLevelSuppress 0 (Default)

2: LogLevelFatal

3: LogLevelErr

4: LogLevelWarn

5: LogLevelNotice

6: LogLevelInfo

7: LogLevelDebug

log_level = "7"

min_free_space specifies the min free space percent in a thin pool require for

new device creation to succeed. Valid values are from 0% - 99%.

Value 0% disables

min_free_space = "10%"

mkfsarg specifies extra mkfs arguments to be used when creating the base

device.

mkfsarg = ""

mountopt specifies extra mount options used when mounting the thin devices.

mountopt = ""

use_deferred_removal Marking device for deferred removal

use_deferred_removal = "True"

use_deferred_deletion Marking device for deferred deletion

use_deferred_deletion = "True"

xfs_nospace_max_retries specifies the maximum number of retries XFS should

attempt to complete IO when ENOSPC (no space) error is returned by

underlying storage device.

xfs_nospace_max_retries = "0"

rhatdan commented 5 years ago

@giuseppe PTAL

giuseppe commented 5 years ago

@scorpionlion, I think it could be related to SELinux. Could you check if it is blocking the process?

Does it work if you temporarily use "setenforce 0"?

scorpionlion commented 5 years ago

yes, It does work when setenforce is disabled. Sorry I forgot to add this.

The observation I have is when podman build gets called through a parent process which happens to be a child process of systemd service then It fails, perhaps the delegation of permissions not passed to podman build.

giuseppe commented 5 years ago

it works on Fedora 30, so I guess the selinux policy requires some tweaking on RHEL 7.7.

What do you see with ausearch -m avc -ts recent?

scorpionlion commented 5 years ago 

sudo ausearch -m avc -ts recent
----
time->Mon Sep 30 11:12:44 2019
type=PROCTITLE msg=audit(1569841964.772:74301): proctitle=72756E6300696E6974
type=PATH msg=audit(1569841964.772:74301): item=0 name="/bin/sh" inode=405865944 dev=08:02 mode=0100755 ouid=332003120 ogid=332000513 rdev=00:00 obj=system_u:object_r:container_file_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1569841964.772:74301):  cwd="/"
type=SYSCALL msg=audit(1569841964.772:74301): arch=c000003e syscall=59 success=no exit=-13 a0=c0000e45f8 a1=c0000eaf00 a2=c000179a10 a3=0 items=1 ppid=15083 pid=15101 auid=4294967295 uid=332003120 gid=332000513 euid=332003120 suid=332003120 fsuid=332003120 egid=332000513 sgid=332000513 fsgid=332000513 tty=(none) ses=4294967295 comm="runc:[2:INIT]" exe="/" subj=system_u:system_r:unconfined_service_t:s0 key="auoms"
type=AVC msg=audit(1569841964.772:74301): avc:  denied  { transition } for  pid=15101 comm="runc:[2:INIT]" path="/usr/bin/bash" dev="sda2" ino=405865944 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c185,c622 tclass=process permissive=0
scorpionlion commented 5 years ago

Also want to check does podman work on RHEL 7.6?

giuseppe commented 5 years ago

it works but has some limitations, as it is not possible to use fuse-overlayfs

scorpionlion commented 5 years ago

Thanks for confirming. Please advise the way forward for this issue.

rhatdan commented 5 years ago

For now add the rules via grep container_t /var/log/audit/audit.log | audit2allow -M mycontainer semodule -i mycontainer.pp

That should fix selinux until the selinux-policy package gets updated.

scorpionlion commented 5 years ago

Thanks for this and apologies for my limited knowledge of se linux rules - in the above command do I need to change mycontainer to something else? I need a solution which allows building of all images not specific to any container.

scorpionlion commented 5 years ago

With help from aglenn@redhat.com created below policies for SELinux and this has resolved the issue.

Please advise when the new selinux package will be available.

created podmanselinux.te

module podmanselinux 1.0;

require {
        type unconfined_service_t;
        type container_t;
        class process transition;
        class process { sigchld transition };
}

allow unconfined_service_t container_t:process transition;
allow container_t unconfined_service_t:process sigchld;

created module from type enforcement

checkmodule -M -m -o podmanselinux.mod podmanselinux.te

created policy file from module

semodule_package -o podmanselinux.pp -m podmanselinux.mod

loaded/applied policy

sudo semodule -i podmanselinux.pp

rhatdan commented 5 years ago

LGTM Does that fix your issue.

scorpionlion commented 5 years ago

yes It does thanks

rhatdan commented 4 years ago

Fixed in container-selinux.