Can't commit images to docker if from alpine base

[mklvr]

When I try to use buildah to build a container that starts from an alpine base, I get an error (see below)

Steps to reproduce the issue:

1. sudo buildah from --name test docker://alpine:latest
2. sudo buildah commit test docker-daemon:mklvr/test:latest

Describe the results you received:

ERRO[0000] error unmounting /var/lib/containers/storage/overlay/cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215/merged: no such file or directory 
error committing container "test" to "docker-daemon:mklvr/test:latest": error copying layers and metadata: Error initializing source containers-storage:test: error extracting layer "cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215": error creating overlay mount to /var/lib/containers/storage/overlay/cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215/merged: no such file or directory

Describe the results you expected: I expected the image to be commit to the local docker daemon without error.

Output of rpm -q buildah or apt list buildah: I'm running Fedora Atomic Workstation 28

buildah-1.0-1.git1ab80bc.fc28.x86_64

Output of buildah version:

Version:       1.0
Go Version:    go1.10.1
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    1ab80bc
Built:         Mon May  7 09:01:02 2018
OS/Arch:       linux/amd64

*Output of `cat /etc/release`:**

Fedora release 28 (Twenty Eight)

Output of uname -a:

Linux koenigii.mklvr.io 4.16.11-300.fc28.x86_64 #1 SMP Tue May 22 18:29:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

[TomSweeneyRedHat]

Hi @mklvr,

It worked for me on F28:

# buildah from --name test docker://alpine:latest
Getting image source signatures
Copying blob sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28
 1.97 MiB / 1.97 MiB [======================================================] 0s
Copying config sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353
 1.48 KiB / 1.48 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
test

# buildah commit test docker-daemon:mklvr/test:latest
Getting image source signatures
Copying blob sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28
 1.97 MiB / 1.97 MiB [======================================================] 0s
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
 32 B / 32 B [==============================================================] 0s
Copying config sha256:4e68576ad9f3e8e4e265463c20908753ed578f51032de285f6054b10cf6eb8fb
 721 B / 721 B [============================================================] 0s
Writing manifest to image destination
Storing signatures

# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64
 Go version:      go1.10
 Git commit:      c301b04-unsupported
 Built:           Wed Mar 28 13:53:29 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64
 Go version:      go1.10
 Git commit:      c301b04-unsupported
 Built:           Wed Mar 28 13:53:29 2018
 OS/Arch:         linux/amd64
 Experimental:    false

Are you using the same version of Docker? @nalind any thoughts?

[mklvr]

Perhaps it's related to running Fedora Atomic Workstation. If we can get someone running Fedora Atomic Workstation 28 to give it a try, that would probably help. Otherwise, if nobody else can reproduce this, we can probably write this off as my problem unless anyone has some other suggestions.

[rhatdan]

Can you mount the ctr?

[TomSweeneyRedHat]

@mklvr OH shoot, I missed the Atomic Host bit. I'll try spinning one up early next week.

[rhatdan]

This works for me also, but on regular Fedora 28. @cgwalters Could you try this?

[cgwalters]

Works for me on both FAH28 and FAW/Silverblue 28.20180524.0. I think you'll have to get more debugging info, my go-to is strace, but someone more familiar with buildah might have better suggestions.

[mklvr]

@rhatdan I can mount the container. @cgwalters Thanks for giving that a try.

strace mklvr_buildah_strace_20180527.txt

rpm-ostree status

State: idle; auto updates enabled (check; no runs since boot)
Deployments:
● ostree://fedora-28-ah:fedora/28/x86_64/workstation
                   Version: 28.20180527.0 (2018-05-27 19:21:56)
                BaseCommit: 0bd9f83f4b0b849f7ee91ccf88a2abb10bee8ce0ecb7938bb71a5b327944ab2a
              GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
           LayeredPackages: NetworkManager-iodine-gnome NetworkManager-l2tp-gnome NetworkManager-libreswan-gnome abrt-desktop abrt-java-connector ansible arandr arc-theme atril atril-caja atril-thumbnailer
                            autoconf blueberry caja caja-actions caja-image-converter caja-open-terminal caja-sendto caja-wallpaper caja-xattr-tags calibre claws-mail ctags dconf-editor dnfdragora-gui
                            dnfdragora-updater emacs engrampa eom exaile fd-find filezilla firewall-config flatpak-builder fros-recordmydesktop git gnome-epub-thumbnailer gnome-tweak-tool gnote gparted
                            gtk2-engines gucharmap hexchat htop hugo i3 i3lock i3status icecat icedtea-web initial-setup-gui java-1.8.0-openjdk keepassxc kubernetes-kubeadm levien-inconsolata-fonts.noarch
                            libmatekbd libmatemixer libmateweather libvirt-daemon-qemu lightdm lightdm-gtk lightdm-gtk-greeter-settings marco mariadb mate-applets mate-backgrounds mate-calc mate-control-center
                            mate-desktop mate-dictionary mate-disk-usage-analyzer mate-icon-theme mate-media mate-menu mate-menus mate-menus-preferences-category-menu mate-notification-daemon mate-panel
                            mate-polkit mate-power-manager mate-screensaver mate-screenshot mate-search-tool mate-session-manager mate-settings-daemon mate-system-log mate-system-monitor mate-terminal mate-themes
                            mate-user-guide mozo neovim network-manager-applet nitrogen nmap numix-gtk-theme numix-icon-theme openssl-devel p7zip p7zip-plugins pandoc parole pass pcsc-lite perlbrew perltidy pluma
                            python-virtualenv python2-docker-py rakudo ratpoison readline-devel remmina sbcl screen scrot seahorse seahorse-caja setroubleshoot snapd stow strace sway system-config-language
                            system-config-printer system-config-users terminator texinfo the_silver_searcher thunderbird tmux transmission-gtk vim vim-enhanced virt-manager wireshark xfburn xmonad xmonad-basic
                            xmonad-config xmonad-core xmonad-log-applet-mate zsh

[mklvr]

I force removed all my images and containers and tried to build from the alpine image again. Everything appears to be working normally. I have no idea what the problem was, but it apparently had something to do with the existing images/containers. I'll re-open this if I am ever able to reproduce the problem. Thanks all.

[rhatdan]

@mklvr It could have been a previous version of buildah, if you have been playing with this for a while. We have had some storage bugs. But definitely want to know about this if it happens again.