search

chainguard-dev / apko

Build OCI images from APK packages directly without Dockerfile
https://apko.dev
Apache License 2.0
1.18k stars 118 forks source link

Error: failed to build layer image #452

Closed leongross closed 1 year ago

leongross commented 1 year ago

Running the example from the documentation fails when using docker.

Running the build using docker fails:

$ docker run -v "$PWD":/work cgr.dev/chainguard/apko build examples/alpine-base.yaml apko-alpine:edge apko-alpine.tar
...
Error: failed to build layer image: initializing apk: failed to initialize apk keyring: opening system keyring: reading keyring directory: readdirent /usr/share/apk/keys/x86_64: not a directory
2023/01/05 10:16:17 error during command execution: failed to build layer image: initializing apk: failed to initialize apk keyring: opening system keyring: reading keyring directory: readdirent /usr/share/apk/keys/x86_64: not a directory

Since it seems the error occurs for the x86_64 architecture, I tried it with another architecture (aarch64) creates a similar issue (full log in appendix):

$ docker run -v "$PWD":/work cgr.dev/chainguard/apko build --debug --build-arch aarch64 examples/alpine-base.yaml apko-alpine:test apko-alpine.tar
...
2023/01/06 08:29:01 ERROR: failed to build package. the build environment has been preserved:
2023/01/06 08:29:01 melange (hello/aarch64):   workspace dir: /tmp/melange-workspace-3073749806
2023/01/06 08:29:01 melange (hello/aarch64):   guest dir: /tmp/melange-guest-4166684833
Error: failed to build package: unable to build guest: unable to generate image: initializing apk: failed to initialize apk keyring: opening system keyring: reading keyring directory: readdirent /usr/share/apk/keys/aarch64: not a directory
2023/01/06 08:29:01 error during command execution: failed to build package: unable to build guest: unable to generate image: initializing apk: failed to initialize apk keyring: opening system keyring: reading keyring directory: readdirent /usr/share/apk/keys/aarch64: not a directory

Looking for the directory /usr/share/apk/keys/x86_64/ in the file system of the container, I could find out that the named path is not a directoy but a file, containing a public key, although this directory should contain multiple public keys. This is also excpected in the code but apparently prior to that this file is created.

EDIT After some research I found an image that works as expected (image ID: sha256:61aa9cec802d5e9144eb966eea625448c993f8f9099d5bbf0aec00ee07ab3bd1) Using either docker or podman yields the wanted result.

Inspecting the image shows, that the keys are at the correct positions of the file system.

# cat /usr/share/apk/keys/x86_64
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA5qQUHmzqX9oa/RS3OSgL
YTYB1mzvVmnWC0JXsAafdgRkavJ9xRhsXMKXWPKMghZw6UnkxuxQCZlB9UhaFqed
X7SWfQ8qRgECFmdGUjoeV7P3VUkg17RU0mUKsXZZF/2mU+jCGyk5eweKcLWrk6bn
lNNF1vBJ9EXbHdfvbQsA5GfLku5tJhqRJ4FVQAGCw+1JoSKu+o3q+6xP7z6kMfR9
cp54FSj48rTJUibbAv+cPtZ3AnbpjdyarV7SYMfxp14Q/KR0CJYDihG3CFoU+TaM
dr7LUrgXaYw7m7g8hA2PY1jF8aqDqVFjVu/csPKnKVQSqNHfm4C4gljwT6TDVhPe
6xKSGKfAUNvHx2RIqSsZJCh68Af+VnfuimbHMYzGhzV4/efihpJlYsDnOXJj/PeY
SbEIYG2yoI3AYvDFyFX2OAOtQ9d4TPs4zS4aDg4R3UseXLib46FX3ZTfIi1/W0IO
eC3AFU5mK+02jq+7swcGbgzem30dFA/B3n8NQExJjF2WlGNJxFw2WnJQqKIkIh0s
km5jaxJ28xYgik+bEgxy1LsilKTowdkbEPRKv36JGdap4dOEGR+LwZsM+ontF1AO
wY6tvwtZKJWhgj7ES3BdtiC1GA2htFnP90lA7KnKbSMOxwWtSCH8spitdPdzgI8G
SxrOb0SZs6ZFAxYYe3GlvS0CAwEAAQ==
-----END PUBLIC KEY-----

Comparing this to to images such as cgr.dev/chainguard/alpine-base reveals, that this image is shipped without the standard keys but with these hard coded keys with the names of the directories.

Running using podman works:

$ podman run -v "$PWD":/work cgr.dev/chainguard/apko build examples/alpine-base.yaml apko-alpine:edge apko-alpine.tar                                                                                                                                                                *[main]
Jan  5 10:24:16.214 [INFO] loading config file: examples/alpine-base.yaml
Jan  5 10:24:16.214 [INFO] [arch:x86_64] detected https://github.com/chainguard-dev/apko@37f45aa3b853e30d84985ff988c99e3b27914da4 as VCS URL
Jan  5 10:24:16.214 [INFO] [arch:x86_64] building image 'apko-alpine:edge'
Jan  5 10:24:16.214 [INFO] [arch:x86_64] build context:
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   working directory: /tmp/apko-2690493219
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   tarball path:
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   use proot: false
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   source date: 1970-01-01 00:00:00 +0000 UTC
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   Docker mediatypes: false
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   SBOM output path: /work
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   arch: x86_64
Jan  5 10:24:16.214 [INFO] [arch:x86_64] image configuration:
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   contents:
Jan  5 10:24:16.214 [INFO] [arch:x86_64]     repositories: [https://dl-cdn.alpinelinux.org/alpine/edge/main]
Jan  5 10:24:16.214 [INFO] [arch:x86_64]     keyring:      []
Jan  5 10:24:16.214 [INFO] [arch:x86_64]     packages:     [alpine-base]
Jan  5 10:24:16.214 [INFO] [arch:x86_64]   cmd: /bin/sh -l
Jan  5 10:24:16.214 [INFO] [arch:x86_64] doing pre-flight checks
Jan  5 10:24:16.214 [INFO] [arch:x86_64] building image fileystem in /tmp/apko-2690493219
Jan  5 10:24:16.214 [INFO] [arch:x86_64] initializing apk database
Jan  5 10:24:16.214 [INFO] [arch:x86_64] [cmd:apk] [use-proot:false] [use-qemu:] running: /sbin/apk add --initdb --arch x86_64 --root /tmp/apko-2690493219
Jan  5 10:24:16.217 [INFO] [arch:x86_64] initializing apk world
Jan  5 10:24:16.217 [INFO] [arch:x86_64] initializing apk keyring
Jan  5 10:24:16.217 [INFO] [arch:x86_64] initializing apk repositories
Jan  5 10:24:16.217 [INFO] [arch:x86_64] synchronizing with desired apk world
Jan  5 10:24:16.217 [INFO] [arch:x86_64] [cmd:apk] [use-proot:false] [use-qemu:] running: /sbin/apk fix --root /tmp/apko-2690493219 --no-scripts --no-cache --update-cache --arch x86_64
Jan  5 10:24:17.246 [INFO] [arch:x86_64] [cmd:/bin/busybox] [use-proot:false] [use-qemu:] running: /usr/sbin/chroot /tmp/apko-2690493219 /bin/busybox --install -s
Jan  5 10:24:17.250 [INFO] [arch:x86_64] [cmd:/sbin/ldconfig] [use-proot:false] [use-qemu:] running: /usr/sbin/chroot /tmp/apko-2690493219 /sbin/ldconfig -v /lib
Jan  5 10:24:17.250 [WARNING] [arch:x86_64] did not generate /etc/os-release: already present
Jan  5 10:24:17.251 [INFO] [arch:x86_64] generating supervision tree
Jan  5 10:24:17.251 [INFO] [arch:x86_64] finished building filesystem in /tmp/apko-2690493219
Jan  5 10:24:17.325 [INFO] [arch:x86_64] built image layer tarball as /tmp/apko-temp-2498421197/apko-x86_64.tar.gz
&{ID:alpine IDLike: Name:Alpine Linux PrettyName:Alpine Linux v3.17 Version: VersionID:3.17.0 VersionCodename:}Jan  5 10:24:17.405 [INFO] [arch:x86_64] building OCI image from layer '/tmp/apko-temp-2498421197/apko-x86_64.tar.gz'
Jan  5 10:24:17.483 [INFO] [arch:x86_64] OCI layer digest: sha256:8c520f5c7855e069e45b333f5016745059e683363044558dc3f97a1c3ddd8566
Jan  5 10:24:17.483 [INFO] [arch:x86_64] OCI layer diffID: sha256:d2dbbb743dd9c6145b56e9688aaf9ca968919e47517fa63ca4c4050c630e2402
Jan  5 10:24:17.484 [WARNING] [arch:x86_64] multiple SBOM formats requested, uploading SBOM with media type: spdx+json
Jan  5 10:24:17.486 [INFO] [arch:x86_64] output OCI image file to apko-alpine.tar

docker version:

Client:
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.19.4
 Git commit:        3a2c30b63a
 Built:             Tue Dec 20 20:43:40 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.4
  Git commit:       42c8b31499
  Built:            Tue Dec 20 20:42:46 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323.m
 runc:
  Version:          1.1.4
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

podman version

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.3
Git Commit:   814b7b003cc630bf6ab188274706c383f9fb9915-dirty
Built:        Sun Nov 20 23:32:45 2022
OS/Arch:      linux/amd64
leongross commented 1 year ago

EDIT: Running this in an Ubuntu 22.04 virtual machine with 

$ podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

produces the same error as docker.

kaniini commented 1 year ago

Sounds like the issue is that the image we produce does not have alpine-keys in it.

leongross commented 1 year ago

Yes, this was the case. I checked again and the current image seems to be fine. Although I couldn't figure out what happened there, do you have an answer to that?

Since we use the image in our build pipeline I would like to know where this issue came from to fix it if this happens again.

kaniini commented 1 year ago

It was a regression from when we converted the image to use Wolfi. We added the missing alpine-keys package.

The next version of apko will automatically download the necessary keys from Alpine when Alpine mirrors are configured. Sorry for the inconvenience.

leongross commented 1 year ago

Thanks for finding up and fixing the error :+1: