search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.37k stars 781 forks source link

running a container based on buildah image on docker host #1802

Closed fins2 closed 3 years ago

fins2 commented 5 years ago

Description

Unable to start a container with docker from an image build with buildah.

Steps to reproduce the issue:

  1. create a conainter image with buildah and the --format docker option
  2. start a container with docker based on the image

Describe the results you received:

docker run -it --rm docker-ansible-from-buildah /bin/bash
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory".

The container start based on the same image with podman run.

Describe the results you expected:

Output of rpm -q buildah or apt list buildah:

buildah-1.9.0-1.el7.centos.x86_64

Output of buildah version:

Version:         1.9.0
Go Version:      go1.10.3
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:         1.9.0
Go Version:      go1.10.3
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64
root@test:~# podman version
Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.10.3
OS/Arch:            linux/amd64

Output of docker version :

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-102.git7f2769b.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      7f2769b/1.13.1
 Built:           Mon Aug  5 15:09:42 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-102.git7f2769b.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      7f2769b/1.13.1
 Built:           Mon Aug  5 15:09:42 2019
 OS/Arch:         linux/amd64
 Experimental:    false

*Output of `cat /etc/release`:**

CentOS Linux release 7.6.1810 (Core) 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core) 
CentOS Linux release 7.6.1810 (Core)

Output of uname -a:

Linux test 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"
rhatdan commented 5 years ago

Do you have an example Dockerfile you used to build the container image with?

fins2 commented 5 years ago

I use a bash script to build my container image. You can find the script bellow:

#!/bin/bash

#
# Generate minimal container image (~57MB) from RHEL7/OL7/CentOS7 repo using buildah (https://github.com/projectatomic/buildah)
#

set -ex

# start new container from scratch
newcontainer=$(buildah from scratch)
scratchmnt=$(buildah mount ${newcontainer})

# install the packages
yum install --nogpgcheck --installroot ${scratchmnt} rpm bash coreutils ansible python2-pyvmomi python-requests shadow-utils --releasever 7 --setopt=tsflags=nodocs --setopt=override_install_langs=en_US.utf8 -y

# Clean up yum cache
if [ -d "${scratchmnt}" ]; then
  rm -rf "${scratchmnt}"/var/cache/yum
fi

#create user and group
buildah run ${newcontainer} groupadd -g 10002 testgroup
buildah run ${newcontainer} useradd -u 10109 -g testgroup testuser

# configure container label, entrypoint, default user and default working directory
buildah config --label name=el7-ansible ${newcontainer}
buildah config --cmd /bin/bash ${newcontainer}
buildah config --user testuser ${newcontainer}
buildah config --workingdir /etc/ansible ${newcontainer}

# commit the image
buildah unmount ${newcontainer}
buildah commit --format docker ${newcontainer} el7-ansible

#some cleanup
buildah rm ${newcontainer}
rhatdan commented 5 years ago

@TomSweeneyRedHat PTAL

vguaglione commented 4 years ago

I'm running into the same problem with the image in the quay repo:

docker run --privileged quay.io/buildah/stable -v mywkdir:/tmp buildah bud -f /tmp/Dockerfile

Error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"-v\": executable file not found in $PATH": unknown.

rhatdan commented 4 years ago

Does the container work with podman?
This looks like the docker parser is getting confused? I just tried to run the buildah container locally and I had no issues

# docker run -ti quay.io/buildah/stable sh
# id
uid=0(root) gid=0(root) groups=0(root)
# exit
rhatdan commented 4 years ago

Actually try docker run --privileged -v mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile

vguaglione commented 4 years ago

Ok, I can run the container locally (your first question) no problem. The command above:

docker run --privileged -v mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile

Gets me closer. Volume error is gone but the problem is the contents of the /tmp folder when mapped:

Mywkdir on the host is this: ls -la mywkdir total 16 drwxr-xr-x 4 vincentguaglione staff 128 Dec 3 15:04 . drwxr-xr-x 10 vincentguaglione staff 320 Dec 3 15:04 .. -rw-r--r-- 1 vincentguaglione staff 1443 Dec 3 15:04 Dockerfile -rw-r--r-- 1 vincentguaglione staff 261 Dec 3 15:04 entrypoint.sh

The error I receive is: error reading info about "/tmp/Dockerfile": stat /tmp/Dockerfile: no such file or directory

So I run this command and list the directory: docker run --privileged -v mywkdir:/tmp quay.io/buildah/stable sh sh-5.0# ls -la /tmp total 12 drwxrwxrwt 2 root root 4096 Dec 3 19:46 . drwxr-xr-x 1 root root 4096 Dec 3 20:02 .. -rwx------ 1 root root 1379 Oct 28 05:48 ks-script-h2x0hr

You can see that the volume mapping either is not working, or, there's something about the container that's causing it to overlay the mapping with some specific file. At least this is what appears to be happening. This is confusing as I've never seen this type of behavior before.

????

rhatdan commented 4 years ago

Try

docker run --privileged -v ${PWD}/mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile or docker run --privileged -v ./mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile

Not sure if the second one works with Docker.

If you just specify -v mywkdir:/tmp Docker will generate an empty volume with this name and use that as the mount point not the path that you expected.

Of course I would prefer to get rid of Docker altogether and just use podman

pdoman run --privileged -v ./mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile

vguaglione commented 4 years ago

Yes sir, that fixed it. -v ${PWD}/mywkdir is what I needed to do to get to recognize the mapping. Thanks for the help.

Unfortunately, I'm now hitting a problem building the image from the dockerfile. Appears to be a problem with an apt-get update, similar to what I've seen with Kaniko container builds:

RUN apt-get update -y && apt-get install -y php7.0-mbstring php-mysqli libnss-wrapper process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1 error building at STEP "RUN apt-get update -y && apt-get install -y php7.0-mbstring php-mysqli libnss-wrapper": exit status 1

I can go ahead and create a separate issue for this problem, unless you can point me in the direction of a fix that's already in place?

rhatdan commented 4 years ago

Could you give me the actual Dockerfile and actual Podman/Docker command you are executing?

vguaglione commented 4 years ago

No problem.

Docker command:

docker run --privileged -v ${PWD}/mywkdir:/tmp quay.io/buildah/stable buildah bud -f /tmp/Dockerfile

Dockerfile:

FROM bitnami/php-fpm:latest EXPOSE 9000 WORKDIR /app

RUN apt-get update -y \ && apt-get install -y php7.0-mbstring php-mysqli libnss-wrapper

Create variables for app and users

ENV APP_ROOT=/app \ USER_NAME=default \ USER_UID=10001 ENV APP_HOME=${APP_ROOT}/src PATH=$PATH:${APP_ROOT}/bin

Create app directory

RUN mkdir -p ${APP_HOME} ${APP_ROOT}/etc

move nss wrapper script in

COPY entrypoint.sh ${APP_ROOT}/bin/

change permissions on app root directories and create new user

RUN chmod -R ug+x ${APP_ROOT}/bin ${APP_ROOT}/etc && sync && \ useradd -l -u ${USER_UID} -r -g 0 -d ${APP_ROOT} -s /sbin/nologin -c "${USER_NAME} user" ${USER_NAME} && \ chown -R ${USER_UID}:0 ${APP_ROOT} && \ chmod -R g=u ${APP_ROOT}

change permissions on php directories

RUN chmod g+rwx -R /opt/bitnami/php /opt/bitnami/php/logs RUN chgrp -R root /opt/bitnami

Containers should NOT run as root as a good practice

USER 10001

start in app root

WORKDIR ${APP_ROOT}

modify app user information in etc password

RUN sed "s@${USER_NAME}:x:${USER_UID}:0@${USER_NAME}:x:\${USER_ID}:\${GROUP_ID}@g" /etc/passwd > ${APP_ROOT}/etc/passwd.template

ENTRYPOINT [ "entrypoint.sh" ]

CMD [ "php-fpm", "-F", "--pid", "/opt/bitnami/php/tmp/php-fpm.pid", "-y", "/opt/bitnami/php/etc/php-fpm.conf" ]

rhatdan commented 4 years ago

@giuseppe I think something might be going on with fuse-overlay on this one?

giuseppe commented 4 years ago

@giuseppe I think something might be going on with fuse-overlay on this one?

I don't see anything related to rootless in this issue. Was the image built with fuse-overlayfs (I doubt that would work on CentOS 7.6)?

vguaglione commented 4 years ago

We are fairly new to some of this alternate docker container build technologies, however, we have seen very similar problems building some images via kaniko, and these also manifest during apt-get updates/installs. Not sure if this helps but I thought to pass it on. Thank you for looking into this for us.

rhatdan commented 4 years ago

Buildah is using fuse-overlay for building inside of a container.

rhatdan commented 4 years ago

@fins2 Is this still an issue?

ketank-new commented 4 years ago

(Edited to touch up formatting issues) I am experiencing a issue building a dockerfile using buildah

i am using base image ubi8/buildah my dockerfile includes steps to addgroup and users on group add i encounter a issue saying

groupadd: /etc/group.241: lock file already used [build-docker-file] groupadd: cannot lock /etc/group; try again later. [build-docker-file] subprocess exited with status 10 [build-docker-file] subprocess exited with status 10

following are my env details

any help will be appreciated i also referred to this https://github.com/containers/buildah/issues/2239

i tried using add '--storage-driver=overlay' -> still the same behaviour also tried using FROM : registry.access.redhat.com/ubi8/ubi:8.2-> same behaviour

Below is the sample dockerfile i am trying to build

------------------------------------------------------------------------------
FROM registry.access.redhat.com/ubi8/ubi:8.2

ENV CICD_GROUP=cicd
ENV CICD_ID=1500
ENV BUILD_USER=builder
ENV BUILD_ID=1084
ENV IBM_POWERAI_LICENSE_ACCEPT=yes

RUN export ARCH="$(uname -m)" && \
    yum repolist && yum install -y \
    sudo \
    bzip2 \
    git \
    patch && \
    # Create CICD Group
    rm -rf /etc/*.lock && \
    groupadd --gid ${CICD_ID} ${CICD_GROUP}  && \
    # Adduser Builder
    useradd -b /home --create-home --gid ${CICD_GROUP} --groups wheel \
    --uid ${BUILD_ID} --comment "User for Building" ${BUILD_USER}
-----------------------------------------------------------------------------------------
TomSweeneyRedHat commented 4 years ago

@ketank-new Couple quick things. Can you include your buildah bud command that you used? Also, as a test, could you replace FROM registry.access.redhat.com/ubi8/ubi:8.2 in your Dockerfile with 'FROM quay.io/buildah/stable:latest` and try that? I've a suspicion some dependency is missing from the UBI8 image.

@fatherlinux FYI

ketank-new commented 4 years ago

@TomSweeneyRedHat : I tried using quay.io/buildah/stable:latestin my dockerfile .. i get a exec error i am executing this on a Power machine i think the image suggested by you is for amd64

my buildah command used is below, this is done using a tekton conatiner command: ['buildah', 'bud','--format=oci', '--tls-verify=true', '--layers', '-f', 'Dockerfile', '-t', 'kk:try']

I have also noted , spinning up a ppc64le container over base image registry.redhat.io/rhel8/buildah and trying to build a dockerfile i notice the below issue on doing a buildah bud -f dockerfile (container wass spinned in privilage=true and /var/lib/containers were mounted)

error mounting container "348e52048b351d57ecbf511d62b6f7c4e77b0df97988927aa767d03a6b969212": error mounting build container "348e52048b351d57ecbf511d62b6f7c4e77b0df97988927aa767d03a6b969212": failed to canonicalise path for "/var/lib/containers/storage/overlay/b9a7c3cb883564f20aa053c7aa85a2e9c68d2f2c72618c90e1bd1c120b41833b/merged": lstat /var/lib/containers/storage/overlay/b9a7c3cb883564f20aa053c7aa85a2e9c68d2f2c72618c90e1bd1c120b41833b/merged: invalid argument

TomSweeneyRedHat commented 4 years ago

@ketank-new thanks for the follow up. I've not tried running that image on a power machine, and didn't catch that in your op. I'm not too surprised it wasn't happy there. @fatherlinux any thoughts on the ubi image on a power machine?

TomSweeneyRedHat commented 4 years ago

We just got a Bugzilla reported too: https://bugzilla.redhat.com/show_bug.cgi?id=1873224, I believe it's the same beast, but the BZ is against Docker. I think it should instead be against UBI8

rhatdan commented 4 years ago

@fins2 Is this still an issue? Can I close this? I am not sure this is a buildah problem

fins2 commented 4 years ago

Yes It's still an issue.

rhatdan commented 4 years ago

@jnovy @fatherlinux who do we point this at?

jnovy commented 3 years ago

@TomSweeneyRedHat the bug 1873224 is irrelevant to this issue as somebody was using s390x Z12 machine which lacks required instruction set.

rhatdan commented 3 years ago

Not sure what we should do with this?

giuseppe commented 3 years ago

Not sure what we should do with this?

I don't think we can fix such kind of issues on CentOS 7 anymore. We expect users to move to RHEL/CentOS 8 at this point

rhatdan commented 3 years ago

Yes, I am going to close.