search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.28k stars 766 forks source link

overlay not working on ubuntu: Error processing tar file(exit status 1): operation not permitted #1745

Closed ianmiell closed 5 years ago

ianmiell commented 5 years ago

Description

When trying to use overlay, I get a permission error regarding permissions.

It defaults to vfs, which works (but uses far too much disk space).

I assume it's some kind of mapping error, but I haven't found a way to resolve.

Steps to reproduce the issue:

  1. rm -rf ~/.local/shared/containers
  2. podman build --storage-driver overlay .

Describe the results you received:

imiell@basquiat:~/tmp  🐧  podman build --storage-driver overlay .
STEP 1: FROM docker.io/ubuntu:xenial
Getting image source signatures
Copying blob edf72af6d627 done
Copying blob 3e4f86211d23 done
Copying blob 8d3eac894db4 done
Copying blob f7277927d38a done
Copying config 5e13f8dd4c done
Writing manifest to image destination
Storing signatures
Error: error creating build container: Error committing the finished image: error adding layer with blob "sha256:8d3eac894db4dc4154377ad28643dfe6625ff0e54bcfa63e0d04921f1a8ef7f8": Error processing tar file(exit status 1): operation not permitted

Describe the results you expected:

Simple docker image is built.

Output of rpm -q buildah or apt list buildah:

imiell@basquiat:~/tmp  🐧  apt list buildah
Listing... Done
buildah/bionic,now 1.9.0-1~dev~ubuntu18.04~ppa17 amd64 [installed]

Output of buildah version:

imiell@basquiat:~/tmp  🐧  buildah version
Version:         1.9.0-dev
Go Version:      go1.10.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Fri May 31 00:39:16 2019
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

imiell@basquiat:~/tmp  🐧  podman version
Version:            1.4.3
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

*Output of `cat /etc/release`:**

imiell@basquiat:~/tmp  🐧  cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output of uname -a:

imiell@basquiat:~/tmp  🐧  uname -a
Linux basquiat 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

imiell@basquiat:~/tmp  🐧  cat /etc/containers/storage.conf 
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary read-write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional read-only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers (currently overlay, zfs, vfs, btrfs)
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"```

It works fine as root.

Strace gives me more output than I can parse, and results in a different error:

Error: error running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted

ERRO[0000] error writing "0 1000 1\n" to /proc/10449/gid_map: write /proc/10449/gid_map: operation not permitted ERRO[0000] (unable to determine exit status) 


For reference:

imiell@basquiat:/etc 🐧 cat subuid imiell:100000:65536 jira:165536:65536 nexus:231072:65536 strongswan:296608:65536

imiell@basquiat:/etc 🐧 cat subgid imiell:100000:65536 jira:165536:65536 nexus:231072:65536 strongswan:296608:65536


In my kernel config:

CONFIG_USER_NS=y

ianmiell commented 5 years ago

Looks like: https://github.com/containers/buildah/issues/1709

TomSweeneyRedHat commented 5 years ago

@ianmiell thanks for the issue. FWIW I edited the description slightly to get the back ticks in the right place for happier formatting. I didn't change any of the other text.

Do you by chance have a Dockerfile that you can share that you used in your test?

ianmiell commented 5 years ago

I’m out and about right now but it was a trivial one, something like

FROM docker.io/ubuntu:xenial RUN ls CMD bash

On Mon, 29 Jul 2019 at 15:24, Tom Sweeney notifications@github.com wrote:

@ianmiell https://github.com/ianmiell thanks for the issue. FWIW I edited the description slightly to get the back ticks in the right place for happier formatting. I didn't change any of the other text.

Do you by chance have a Dockerfile that you can share that you used in your test?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTIPYLMBQRY5FGPSK3TQB34QXA5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3A33NQ#issuecomment-516013494, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVNZTNYABHQTAYY2SSGLCTQB34QXANCNFSM4IHNKLWQ .

giuseppe commented 5 years ago

Looks like: #1709

it is indeed a duplicate of #1720

The issue is that an unprivileged user has no privileges for using mknod so we cannot (easily) create whiteout files when extracting an image.

Our suggestion is to use fuse-overlayfs on Ubuntu as well, you'd need to install fuse-overlayfs and add mount_program = "/usr/bin/fuse-overlayfs under [storage.options] in your ~/.config/containers/storage.conf file.

@lsm5 can we add a dependency to fuse-overlayfs?

We probably need a more descriptive error message since the issue was already reported twice.

ianmiell commented 5 years ago

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 buildah --debug bud .

DEBU[0000] running [buildah-in-a-user-namespace --debug bud .] with environment [SHELL=/bin/bash HISTCONTROL=erasedups:ignorespace HISTSIZE=1000000 HOSTNAME=basquiat LANGUAGE=en_GB:en JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 HISTTIMEFORMAT=%d/%m/%y %T EDITOR=vim GOBIN=/space/go/bin PWD=/space/git/alertonchange LOGNAME=imiell QT_QPA_PLATFORMTHEME=appmenu-qt5 XDG_SESSION_TYPE=tty GEM_ROOT=/opt/chefdk/embedded/lib/ruby/gems/2.5.0 TZ=GB CDPATH=.:/space/git:/space/git/work HOME=/home/imiell LANG=en_GB.UTF-8 HISTFILE=/home/imiell/.bash_history LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36: SSH_CONNECTION=86.137.120.162 49837 192.168.1.124 22 GOROOT=/space/go GEM_PATH=/home/imiell/.chefdk/gem/ruby/2.5.0:/opt/chefdk/embedded/lib/ruby/gems/2.5.0 GEM_HOME=/home/imiell/.chefdk/gem/ruby/2.5.0 LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s LIBVIRT_DEFAULT_URI=qemu:///system USER=imiell VISUAL=vim SHLVL=1 PAGER=less XDG_SESSION_ID=12103 XDG_RUNTIME_DIR=/run/user/1000 SSH_CLIENT=86.137.120.162 49837 22 ANSIBLE_HOST_KEY_CHECKING=False XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop PATH=/Users/imiell/anaconda3/bin:/usr/local/bin:/usr/local/opt/openssl/bin:/home/imiell/.krew/bin:/opt/chefdk/bin:/home/imiell/.chefdk/gem/ruby/2.5.0/bin:/opt/chefdk/embedded/bin:/home/imiell/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/bin:~/.dotfiles/bin::/opt/chefdk/gitbin/space/git/work/bin:/space/git/work/bin/archive:/space/git/work/bin/asciidoctor:/space/git/work/bin/git:/space/git/work/bin/.gitignore:/space/git/work/bin/home:/space/git/work/bin/jenkins:/space/git/work/bin/man:/space/git/work/bin/media:/space/git/work/bin/music:/space/git/work/bin/notes:/space/git/work/bin/openstack:/space/git/work/bin/state_street:/space/git/work/bin/utils:/space/git/work/bin/vms:/space/git/home/bin: HISTFILESIZE=1000000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MAIL=/var/mail/imiell SSHTTY=/dev/pts/0 GOPATH=/home/imiell/go =/usr/bin/buildah OLDPWD=/home/imiell _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:1000 Size:1} {ContainerID:1 HostID:100000 Size:65536}], and GID map [{ContainerID:0 HostID:1000 Size:1} {ContainerID:1 HostID:100000 Size:65536}]

DEBU[0000] [graphdriver] trying provided driver "vfs"

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 cat !$

cat /etc/containers/storage.conf

storage.conf is the configuration file for all tools

that share the containers/storage libraries

See man 5 containers-storage.conf for more information

The "container storage" table contains all of the server options.

[storage]

Default Storage Driver

driver = "overlay"

mount_program = "/usr/bin/fuse-overlayfs"

Temporary storage location

runroot = "/var/run/containers/storage"

Primary read-write location of container storage

graphroot = "/var/lib/containers/storage"

[storage.options]

AdditionalImageStores is used to pass paths to additional read-only image

stores

Must be comma separated list.

additionalimagestores = [

]

Size is used to set a maximum size of the container image. Only

supported by

certain container storage drivers (currently overlay, zfs, vfs, btrfs)

size = ""

OverrideKernelCheck tells the driver to ignore kernel checks based on

kernel version

override_kernel_check = "true"

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 ls -l /usr/bin/fuse-overlayfs

-rwxr-xr-x 1 root root 63920 Jan 15 2019

/usr/bin/fuse-overlayfs

I've confirmed with strace that that's the config file being read.

On Tue, Jul 30, 2019 at 2:41 PM Giuseppe Scrivano notifications@github.com wrote:

Looks like: #1709 https://github.com/containers/buildah/issues/1709

it is indeed a duplicate of #1720 https://github.com/containers/buildah/pull/1720

The issue is that an unprivileged user has no privileges for using mknod so we cannot (easily) create whiteout files when extracting an image.

Our suggestion is to use fuse-overlayfs on Ubuntu as well, you'd need to install fuse-overlayfs and add mount_program = "/usr/bin/fuse-overlayfs under [storage.options] in your ~/.config/containers/storage.conf file.

@lsm5 https://github.com/lsm5 can we add a dependency to fuse-overlayfs?

We probably need a more descriptive error message since the issue was already reported twice.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTO4UE4HRLMUFSBW6NDQCBAG3A5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3D77RA#issuecomment-516423620, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVNZTPQ52QW3ZQDT7IARDLQCBAG3ANCNFSM4IHNKLWQ .

giuseppe commented 5 years ago

/etc/containers/storage.conf is the configuration for root containers.

When you are running rootless, Podman/Buildah will use the configuration files under $HOME/.config/containers.

An issue I see in the configuration file you are using is that mount_program = "/usr/bin/fuse-overlayfs" is not in the correct place. It must be under the [storage.options] TOML section.

Another solution is to delete the configuration files and let podman recreate them It will automatically configure the storage to use fuse-overlayfs when it is available: rm -rf $HOME/.config/containers && podman system migrate

ianmiell commented 5 years ago

If $HOME/.config/containers/storage.conf does not exist, then strace reports that it looks at /etc/containers/storage.conf. Maybe it acts differently under strace (I notice I get a different error message, presumably due to some interaction between strace and the syscalls (?).

In any case, the last suggestion rm -rf... worked a treat. Thanks.

giuseppe commented 5 years ago

unfortunately strace breaks creating the user namespace, as setuid binaries (such as /usr/bin/newuidmap and /usr/bin/newgidmap) cannot be traced.

/etc/containers/storage.conf is used only to cherry pick some settings, like runtime paths, but the whole configuration file cannot be used as most of the settings won't work when running without root privileges

ianmiell commented 5 years ago

Ah, thanks - that's interesting. Explains why it was complaining to me about new*idmap.

On Wed, Jul 31, 2019 at 1:58 PM Giuseppe Scrivano notifications@github.com wrote:

unfortunately strace breaks creating the user namespace, as setuid binaries (such as /usr/bin/newuidmap and /usr/bin/newgidmap) cannot be traced.

/etc/containers/storage.conf is used only to cherry pick some settings, like runtime paths, but the whole configuration file cannot be used as most of the settings won't work when running without root privileges

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTLSWIKMSGJRZJKXLHLQCGEADA5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HFBRA#issuecomment-516837572, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVNZTNTLJKUEFTAFFUNX2DQCGEADANCNFSM4IHNKLWQ .

tennox commented 4 years ago

I like that you wrote a troubleshooting section, but do you have plans to detect / use fuse-overlay as default on ubuntu? I ran into this too, and took me now ~2 hours to debug and find this.

rhatdan commented 4 years ago

I guess the question is does the Podman package require the fuse-overlay package. On Fedora we have this as a Recommends, which means it is installed by default and Podman will use it.