Buildah bud writes stdout to stderr, which causes pipeline builds to fail.

[erbzn]

Description

I am trying to use Buildah in Azure DevOps Pipeline with the example repository phippyandfriends (https://github.com/Azure/phippyandfriends). First step is to build phippy image. This image build fails with "buildah bud", as it writes some standard output to standard error, e.g. Copying blob sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c. I am using a self-hosting agent built from the buildah image "quay.io/buildah/stable:latest" and configured to connect to Azure DevOps agent pool with the following Dockerfile (start.sh connects to agent pool):

FROM quay.io/buildah/stable:latest

RUN yum install -y \
        git \
        iputils \
        libicu \
        libunwind \
        nmap-ncat \
        unzip \
        jq

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

Within this Container I want to build the following image:

FROM composer:1 AS build-env
COPY . /app
RUN cd /app && composer install

FROM php:7.3.12-apache-stretch
ENV PORT 3333
EXPOSE 3333
COPY --from=build-env /app /var/www/html
RUN usermod -u 1000 www-data; \
    a2enmod rewrite; \
    chown -R www-data:www-data /var/www/html

I've tested with 2 versions of buildah (1.14.9, 1.16.0-dev), everytime it is writing parts to stderr but the build does not fail. Pushing and running built image works as normal afterwards.

Steps to reproduce the issue:

1. Run buildah image (tested with docker): docker run -it --net=host --security-opt label=disable --security-opt seccomp=unconfined --device /dev/fuse:rw quay.io/buildah/stable:latest /bin/bash
2. Inside container install git and clone phippyandfriends: git clone https://github.com/Azure/phippyandfriends.git
3. Build image and write stdout and stderr to files: buildah bud phippy/. > stdout.txt 2> stderr.txt

Describe the results you received:

[root@docker-desktop s]# cat stdout.txt
STEP 1: FROM composer:1 AS build-env
STEP 2: COPY . /app
STEP 3: RUN cd /app && composer install
STEP 4: FROM php:7.3.12-apache-stretch
STEP 5: ENV PORT 3333
STEP 6: EXPOSE 3333
STEP 7: COPY --from=build-env /app /var/www/html
STEP 8: RUN usermod -u 1000 www-data;     a2enmod rewrite;     chown -R www-data:www-data /var/www/html
Enabling module rewrite.
To activate the new configuration, you need to run:
  service apache2 restart
STEP 9: COMMIT
--> 0375aa93a3d
0375aa93a3da0243df08baabf55ffbb084d64897c70f68e103c4f63b277e8f8d

[root@docker-desktop s]# cat stderr.txt
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Generating autoload files
Getting image source signatures
Copying blob sha256:99b5261d397c75b1de2c91f0c89f4e6f287248669b72de6cb143e82ea67ad056
Copying blob sha256:4f64650bebe7fc61d223ca9d0ec930f768f497b9d44b0e6bcaef8662ff745b9a
Copying blob sha256:8b29d1731799f58be01226085909c431b91744ce73df896ac6b46f09588a3db6
Copying blob sha256:9a2750ba31fdf3568bbfd56722a6e44fd4bc5ec928ec9963fd83dffd98cfb1db
Copying blob sha256:ae7c653abd6771cb43439282ee80eb74587e9329f7eb7d4fe0c71e3bd157af9c
Copying blob sha256:15ce240390f5b203cdcf6c8af7b0b6665cb413596506eb60ad7915d31abcb46f
Copying blob sha256:1bf609e05f27a02b3c5ddb368f87515f66dd5e736bbf3c269127dd96a17da24e
Copying blob sha256:dbb207a9407eb84540b44e00017652018f5cff853e3c22e52c6ccce4b38d6100
Copying blob sha256:0d494fbe0ebfdf853eff44b8f7e9cc4b32026fa1e9f5ca8dafbfbd498e80844f
Copying blob sha256:eff25448236dbb341ee281d2bcc9a3fcc8f768289f3892b5f17ca97a1c0572de
Copying blob sha256:df2423fff0ff4df6302a6efe0ad1544995307ed8e445e3998645cfb0b5e5026e
Copying blob sha256:a85820fe580ed5832eb5def67903789f2e03faeea121843b8ae52851f79c9322
Copying blob sha256:7764a46f2f2b5af0d842f10876895931fcca003767b9b76703eaad05447fc627
Copying blob sha256:6199319a6c96abc38a5f03d2b9449f42a229c587ab93980a91a2c6b393b9e9d0
Copying config sha256:0375aa93a3da0243df08baabf55ffbb084d64897c70f68e103c4f63b277e8f8d
Writing manifest to image destination
Storing signatures

Describe the results you expected: Only stdout is writing log, as image build is successful.

Output of rpm -q buildah or apt list buildah:

[root@docker-desktop s]# rpm -q buildah
buildah-1.14.9-1.fc31.x86_64

Output of buildah version: Tested with 2 versions:

[root@docker-desktop /]# buildah version
Version:         1.14.9
Go Version:      go1.13.10
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.4.3
Git Commit:
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

[root@docker-desktop s]# buildah version
Version:         1.16.0-dev
Go Version:      go1.13.11
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  v0.7.2-0.20190904153231-83439463f784
image Version:   5.5.1
Git Commit:      9fd14e65
Built:           Wed Jun 24 12:33:53 2020
OS/Arch:         linux/amd64

*Output of `cat /etc/release`:**

[root@docker-desktop /]# cat /etc/*release
Fedora release 31 (Thirty One)
NAME=Fedora
VERSION="31 (Container Image)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
Fedora release 31 (Thirty One)
Fedora release 31 (Thirty One)

Output of uname -a:

[root@docker-desktop /]# uname -a
Linux docker-desktop 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[root@docker-desktop /]# cat /etc/containers/storage.conf
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
"/var/lib/shared",
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = false

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,fsync=0"

# Size is used to set a maximum size of the container image.
# size = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[rhatdan]

Have you tried with the --quiet option?

I belive Buidlah and Docker build always write progress information to stderr and only write the image id to stdout.

[erbzn]

Thank you for the fast answer.

If I run Docker build for the mentioned Dockerfile nothing is written to stderr. If I run Buildah with the --quiet option still some lines are written to stderr:

[root@docker-desktop phippyandfriends]# buildah bud --quiet phippy/. > stdout.txt 2> stderr.txt
[root@docker-desktop phippyandfriends]# cat stdout.txt
ec18b5036cfc029125e425d3c4526bc38cd6efcbd9fac451f2e5456f9e562dc4
[root@docker-desktop phippyandfriends]# cat stderr.txt
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Generating autoload files

And as there is the configuration option failOnStderr: true in Azure DevOps Pipeline the successful build is marked as failure.

[rhatdan]

Ok that is a bug.

[umohnani8]

Tested the fix in #2942 out and @erbzn looks like there is an issue in the composer install step in the phippy container file, that step writes to stderr instead of stdout. Verified with various other containerfiles and we always write to stdout now with the fix.

[umohnani8]

@erbzn you can use the --quiet flag to suppress any output to stderr. A mentioned in https://github.com/containers/buildah/issues/2423#issuecomment-768358187, there is an issue with the process running in your container that writes to stderr. Closing as this is not a bug.

[kanadaj]

@umohnani8 Does that mean Buildah is suppressing the output of the docker container's stderr? Since I see no errors in the buildah output other than printing the steps as stderr.