Buildah commands failing with no such file or directory

[nomadme]

Description

Steps to reproduce the issue: Any buildah commands failing with No such file or directory, no matter what command I use

1. buildah images
2. buildah bud -f Dockerfile -t foo:latest .

Describe the results you received: buildah images

[root@3ebf0faaa664 buildah-test]# buildah images
Error while loading images: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)       

buildah bud -f Dockerfile -t foo:latest .

Error while loading bud: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)    

Describe the results you expected: Image is built

Output of rpm -q buildah or apt list buildah:

buildah-1.22.3-2.module+el8.5.0+12582+56d94c81.x86_64

Output of buildah version:

Version:         1.22.4-dev
Go Version:      go1.16.7
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  v0.8.1
image Version:   5.15.2
Git Commit:      
Built:           Mon Sep 13 15:13:43 2021
OS/Arch:         linux/amd64

*Output of `cat /etc/release`:**

NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"
Red Hat Enterprise Linux release 8.5 (Ootpa)
Red Hat Enterprise Linux release 8.5 (Ootpa)

Output of uname -a:

Linux 3ebf0faaa664 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[flouthoc]

Hi @nomadme thanks for creating the issue.

Are you running buildah inside a container ? I think its failing while performing unshare but first I'd suggest updating buildah to latest version.

and could you share more details about your environment like if its a container and logs with buildah --log-level=debug <your command>

I am suspecting missing privileges on your working environment but your release is showing misleading error message. So please share details with your attempt from a latest version.

[nomadme]

Thank you for getting back on this @flouthoc.

Yes, it is running inside a container built from: https://catalog.redhat.com/software/containers/ubi8/openjdk-11/5dd6a4b45a13461646f677f4?tag=1.10-10.1638383051

[root@2c088bb099bf tmp]# buildah --log-level=debug info
DEBU[0000] running [/usr/bin/buildah-in-a-user-namespace --log-level=debug info] with environment [_=/usr/bin/buildah BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
} LESSOPEN=||/usr/bin/lesspipe.sh %s JBOSS_CONTAINER_JAVA_PROXY_MODULE=/opt/jboss/container/java/proxy LD_PRELOAD=libnss_wrapper.so CHROME_BIN=/bin/chrome JBOSS_CONTAINER_UTIL_LOGGING_MODULE=/opt/jboss/container/util/logging/ PATH=/home/jboss/.local/bin:/home/jboss/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/s2i JBOSS_CONTAINER_MAVEN_S2I_MODULE=/opt/jboss/container/maven/s2i MAVEN_VERSION=3.6 S2I_SOURCE_DEPLOYMENTS_FILTER=*.jar SHLVL=1 JAVA_VENDOR=openjdk JOLOKIA_VERSION=1.6.2 NSS_WRAPPER_GROUP=/etc/group JBOSS_CONTAINER_PROMETHEUS_MODULE=/opt/jboss/container/prometheus TERM=xterm AB_JOLOKIA_AUTH_OPENSHIFT=true JBOSS_IMAGE_VERSION=1.10 JAVA_DATA_DIR=/deployments/data JBOSS_CONTAINER_JAVA_JVM_MODULE=/opt/jboss/container/java/jvm HOME=/home/jboss PWD=/tmp JAVA_VERSION=11 AB_JOLOKIA_HTTPS=true AB_PROMETHEUS_JMX_EXPORTER_CONFIG=/opt/jboss/container/prometheus/etc/jmx-exporter-config.yaml NSS_WRAPPER_PASSWD=/home/jboss/passwd container=oci JBOSS_CONTAINER_JAVA_S2I_MODULE=/opt/jboss/container/java/s2i which_declare=declare -f JBOSS_CONTAINER_JOLOKIA_MODULE=/opt/jboss/container/jolokia JBOSS_CONTAINER_OPENJDK_JDK_MODULE=/opt/jboss/container/openjdk/jdk JBOSS_CONTAINER_S2I_CORE_MODULE=/opt/jboss/container/s2i/core/ JBOSS_CONTAINER_MAVEN_36_MODULE=/opt/jboss/container/maven/36/ JAVA_HOME=/usr/lib/jvm/java-11 HOSTNAME=2c088bb099bf AB_JOLOKIA_PASSWORD_RANDOM=true LANG=C.utf8 JBOSS_CONTAINER_JAVA_RUN_MODULE=/opt/jboss/container/java/run JBOSS_CONTAINER_MAVEN_DEFAULT_MODULE=/opt/jboss/container/maven/default/ JBOSS_IMAGE_NAME=ubi8/openjdk-11 TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}] 
qemu: unknown option 'log-level=debug'
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)  

Here is my local docker info: docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.1)
  scan: Docker Scan (Docker Inc., v0.14.0)

Server:
 Containers: 4
  Running: 3
  Paused: 0
  Stopped: 1
 Images: 7
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 7
 Total Memory: 9.718GiB
 Name: docker-desktop
 ID: GHQF:7XPX:HPMX:UONT:A37R:2M7V:VCUZ:Z3MW:3GTK:T22G:SK2I:Q3RN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I'm running on M1Pro Chip, if that clues in something as well.

[nomadme]

I also ran a container from https://quay.io/repository/buildah/stable?tab=info that is also getting same error as above container I'm running.

Output of: cat /etc/*release

[root@b359094d70dc /]# cat /etc/*release
Fedora release 35 (Thirty Five)
NAME="Fedora Linux"
VERSION="35 (Container Image)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
Fedora release 35 (Thirty Five)
Fedora release 35 (Thirty Five)

Output of: buildah --log-level=debug info

[root@b359094d70dc /]# buildah --log-level debug info
DEBU[0000] running [/usr/bin/buildah-in-a-user-namespace --log-level debug info] with environment [_=/usr/bin/buildah PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/  SHLVL=1 TERM=xterm BUILDAH_ISOLATION=chroot FGC=f35 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36: LANG=C.UTF-8 HOME=/root container=oci PWD=/ DISTTAG=f35container HOSTNAME=b359094d70dc TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}] 
qemu: unknown option 'log-level'
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)  

[clydet]

Hello @flouthoc , Not sure whether or not I'm running into a similar issue.

$ docker run -it quay.io/buildah/stable:latest /bin/sh

sh-5.1# buildah --log-level debug info
DEBU[0000] running [buildah-in-a-user-namespace --log-level debug info] with environment [HOSTNAME=141826c3abd4 DISTTAG=f35container PWD=/ container=oci HOME=/root FGC=f35 BUILDAH_ISOLATION=chroot TERM=xterm SHLVL=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/buildah TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}] 
Error during unshare(CLONE_NEWUSER): Operation not permitted
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)

Does anything from that output indicate a known issue?

[rhatdan]

Try docker run --security-opt seccomp=unconfined -it quay.io/buildah/stable:latest /bin/sh

Or better yet, use Podman. :^)

Docker default seccomp.json does not allow the unshare Syscall.

[nomadme]

Thanks @rhatdan

When I ran it as suggested, I still got the same error.

docker run --security-opt seccomp=unconfined --platform linux/amd64 -it quay.io/buildah/stable:latest /bin/sh                                               130 ↵ gg@GG-MBP-M1Pro
sh-5.1# buildah
qemu: no user program specified
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)            
sh-5.1# buildah info
Error while loading info: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)

here is the log:

sh-5.1# buildah --log-level debug info
DEBU[0000] running [/usr/bin/buildah-in-a-user-namespace --log-level debug info] with environment [_=/usr/bin/buildah PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHLVL=1 TERM=xterm BUILDAH_ISOLATION=chroot FGC=f35 HOME=/root container=oci PWD=/ DISTTAG=f35container HOSTNAME=7bc9798701e5 TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}] 
qemu: unknown option 'log-level'
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)            
sh-5.1# 

[flouthoc]

Hi @nomadme and @clydet ,

Sorry for late reply. I tried playing with docker and quay.io/buildah/stable:latest inside docker.

I think inside container buildah uses fuse-overlay rather than host kernel overlay so we would also need to mount /dev/fuse when running buildah inside docker.

Could you guys please try with following command. Following works for me.

sudo docker run -it --device /dev/fuse:rw --security-opt seccomp=unconfined --security-opt apparmor=unconfined quay.io/buildah/stable:latest /bin/sh

Output from my terminal

flouthoc@flouthoc-pc:~$ sudo docker run -it --device /dev/fuse:rw --security-opt seccomp=unconfined --security-opt apparmor=unconfined quay.io/buildah/stable:latest /bin/shsh-5.1# buildah images
REPOSITORY   TAG   IMAGE ID   CREATED   SIZE
sh-5.1# vi Dockerfile
sh-5.1# buildah build -t test . 
STEP 1/2: FROM alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 59bf1c3509f3 done  
Copying config c059bfaa84 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo hello
hello
COMMIT test
Getting image source signatures
Copying blob 8d3ac3489996 skipped: already exists  
Copying blob 274a6abcd2da done  
Copying config 00566efaba done  

[clydet]

@flouthoc, You're a stud. That works for me locally. We're looking to use buildah in containers running in AWS' ECS via Fargate. According to AWS docs regarding devices:

Note If you are using tasks that use the Fargate launch type, the devices parameter is not supported.

Any idea how we might work around an environment that doesn't allow us access to host devices?

[nomadme]

Thanks @flouthoc

For me, it still doesn't work. I'm on M1Pro chip.

sudo docker run -it --device /dev/fuse:rw --security-opt seccomp=unconfined --security-opt apparmor=unconfined quay.io/buildah/stable:latest /bin/sh                       1 ↵ gg@GG-MBP-M1Pro
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
sh-5.1# buildah images
Error while loading images: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)            
sh-5.1# vi Dockerfile
sh-5.1# buildah build -t test .
Error while loading build: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)

Here is the other try with M1 chip specific run.

sudo docker run -it --device /dev/fuse:rw --security-opt seccomp=unconfined --security-opt apparmor=unconfined --platform linux/amd64 quay.io/buildah/stable:latest /bin/sh
sh-5.1# buildah images
Error while loading images: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)            
sh-5.1# vi
sh-5.1# vi Dockerfile
sh-5.1# buildah build -t test .
Error while loading build: No such file or directory
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)

[clydet]

@flouthoc, I Googled around a bit and saw this, which states:

Podman can use native overlay file system with the Linux kernel versions 5.13. Up until now, we have been using fuse-overlayfs. The kernel gained rootless support in the 5.11 kernel, but a bug prevented SELinux use with the file system; this bug was fixed in 5.13.

It looks like Fedora is backporting the fix into its 5.12 kernels, so users should be able to use it once they get access to the kernel.

I haven't found a fedora container with a kernel later than 5.10.47-linuxkit. Guessing for our use case we'll need to move away from Fargate as AWS docs seem to indicate support for host device exposure when favoring EC2 based services. 🤞

Do you think that there's any use in searching about for any RHEL derivative containers with a 5.13 kernel? If so please point me in the right direction.

[flouthoc]

@clydet I think that would need a host kernel change. Afaik fargate locks you with a kernel so I am not sure if you could change it. Could you confirm if host kernel can be changed on ECS with fargate from AWS support. I think not.

I'd like to recommend using ECS with EC2 that should easily allow you use --device or change the kernel via using custom AMI whatever you like.

@nomadme could you please try with --platform linux/arm64/v8 or --platform linux/arm i think docker desktop is doing something with emulation not sure.

[rhatdan]

You can probably use the "vfs" driver.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[ibyrktr]

I ran into same issue when using AWS ECS with Fargate and it uses vfs as default storage layer and if I use flag --storage-driver vfs still the same result.

buildah --storage-driver vfs bud -t myimage:1.0 -f . Error during unshare(CLONE_NEWUSER): Operation not permitted time="2022-03-11T17:40:04Z" level=error msg="error parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax" time="2022-03-11T17:40:04Z" level=error msg="(unable to determine exit status)"

AWS Fargate version we use is 1.4.0 and it seems to be using containerd as container runtime. Use of EC2 instead of Fargate is an option but AWS folks seems to have managed it with kaniko: https://aws.amazon.com/blogs/containers/building-container-images-on-amazon-ecs-on-aws-fargate/ and I was wondering if anyone managed to get around this with buildah.

[rhatdan]

This looks like you are being blocked by either SECCOMP or no CAP_SET_UID

[javierlga]

This works on my M1:

docker run -it -e _BUILDAH_STARTED_IN_USERNS="" -e BUILDAH_ISOLATION=chroot \
--security-opt seccomp=unconfined --security-opt label:disabled \
quay.io/buildah/stable:latest /bin/sh

However it's necessary to use vfs as the storage driver, e.g. : buildah bud --storage-driver vfs . For overlay, it's necessary to mount the device, e.g.:

docker run -it --device /dev/fuse:rw -e _BUILDAH_STARTED_IN_USERNS="" -e BUILDAH_ISOLATION=chroot \
--security-opt seccomp=unconfined --security-opt label:disabled \
quay.io/buildah/stable:latest /bin/sh

[javierlga]

Interesting how everyone is using app armor on a RHEL based docker image instead of label:disabled for SELinux

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.