Pack Build Fails with Extensions

JanakaSandaruwan commented 4 months ago

Summary

We are using docker extensions in our builder. The run.Dockerfile as follows

cat >>"${output_dir}/run.Dockerfile" <<EOL
FROM <python alpine image>

COPY ./requirements.txt ./requirements.txt

ARG user_id
ARG group_id
USER root
RUN apk add bash ca-certificates

$([ -n "$pip_version" ] && echo "RUN pip install pip==$pip_version")
RUN pip install -r requirements.txt --prefix /usr/local/ --ignore-installed \\
     --disable-pip-version-check --no-cache-dir --progress-bar off

USER \${user_id}:\${group_id}
EOL

However, building the app image using pack command failing with following error.

command - pack build -B <builder> test_image -v

Error Output

[detector] Run image info in analyzed metadata is: 
[detector] {"Reference":"xxx","Image":"xxx","Extend":true}
Pulling image 'xxx'
alpine: Pulling from xxx
4abcf2066143: Already exists
dca80dc46cec: Pulling fs layer
3324090550b3: Pulling fs layer
f5c92aa967a6: Pulling fs layer
34eb5116f7f2: Pulling fs layer
34eb5116f7f2: Waiting
f5c92aa967a6: Download complete
dca80dc46cec: Verifying Checksum
dca80dc46cec: Download complete
34eb5116f7f2: Verifying Checksum
34eb5116f7f2: Download complete
3324090550b3: Verifying Checksum
3324090550b3: Download complete
dca80dc46cec: Pull complete
3324090550b3: Pull complete
f5c92aa967a6: Pull complete
34eb5116f7f2: Pull complete
Digest: sha256:b7ca41d7e966f564259276ca0cb8167178193d20d00448533976e92e96271f11
Status: Downloaded newer image for xxx
ERROR: failed to build: executing lifecycle: open /tmp/extend-run-image-scratch947184908/blobs/sha256.tar: no such file or directory
Error: Process completed with exit code 1.

Environment

pack info

Pack:
  Version:  0.33.2
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.18.5

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
  experimental = true
  layout-repo-dir = "/home/runner/.pack/layout-repo"

docker info

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-1059-azure
 Operating System: Alpine Linux v3.19 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.34GiB
 Name: runners-vjf98-5pmzk
 ID: ea1eefed-89b2-416d-9e38-624f185b77e7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

natalieparellano commented 4 months ago

@JanakaSandaruwan thanks for reporting this. Do you have containerd storage enabled by chance? Before running extensions we download the run image and create a new "ephemeral" run image that has the lifecycle added as an additional layer. To grab the lifecycle layer we do a docker image save on the lifecycle image (here), and are making some assumptions about the format of the saved image.tar that I fear are no longer valid if containerd storage is enabled. open /tmp/extend-run-image-scratch947184908/blobs/sha256.tar is particularly suspicious, as sha256.tar without an actual digest is a poorly formatted path. It is probably coming from here.

Edit: I see that you have Storage Driver: overlay2 in your docker info output, so it's not containerd (though that is something we should fix). Still, something is amiss with how we are extracting the lifecycle layer...

Edit Edit: Seeing that you are on docker 25.x, the docker image save output is using the new format (https://github.com/moby/moby/pull/44598) so that explains why the filename is the weird /tmp/extend-run-image-scratch947184908/blobs/sha256.tar. I believe pack is failing to create the tar file because its name contains a parent dir that doesn't exist.

jjbustamante commented 4 months ago

@JanakaSandaruwan could you try building your app using the latest lifecycle image? add the flag --lifecycle-image buildpacksio/lifecycle:0.19.6 to your pack build command.

jjbustamante commented 4 months ago

@JanakaSandaruwan thanks for reporting this. actually, we managed to reproduce the error in our local environments when running the acceptance tests. We didn't notice before because the runner used for our acceptance tests is using Docker 24.0.9

buildpacks / pack