Run image extension does not install RPMs that are already installed in the builder

[yters]

Summary

When the run image run.Dockerfile is extended to RUN dnf install RPMs, if the RPM is already in the builder image then it is not installed in the run image. This is a problem since the run image does not have access to the RPMs in the builder image.

---

Reproduction

Steps

1. Create a build image with the following Dockerfile named <path>/build-base-python3.11:debug.

   
   FROM registry.access.redhat.com/ubi8-minimal:8.8

ARG STACK_ID="debug"

ENV CNB_USER_ID=${CNB_UID:-1000} ENV CNB_GROUP_ID=${CNB_GID:-1000}

RUN microdnf update RUN microdnf install python3.11

RUN groupadd cnb --gid ${CNB_GROUP_ID} RUN useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb

USER cnb LABEL io.buildpacks.stack.id=${STACK_ID} ENV CNB_STACK_ID=${STACK_ID} ENV CNB_USER_ID=${CNB_UID:-1000} ENV CNB_GROUP_ID=${CNB_GID:-1000}

2. Create a run image with the following Dockerfile named `<path>/run-base:debug`

FROM registry.access.redhat.com/ubi8-minimal:8.8

ARG STACK_ID="debug"

ENV APP_USER_ID=${APP_UID:-31460} ENV APP_GROUP_ID=${APP_GID:-31460}

LABEL io.buildpacks.stack.id=${STACK_ID} ENV CNB_STACK_ID=${STACK_ID}

4. Create an extension with the following files, named `<path>/extension-dnf-install:debug`.
`extension.toml`

api = "0.10"

[extension] id = "dnf-install" name = "dnf python3.11 RPM installer" description = "Extension that installs python3.11 RPM in the run image." version = "debug"

[[targets]] os = "linux/amd64" arch = "amd64"

`generate/run.Dockerfile`

ARG base_image FROM ${base_image}

USER root

RUN microdnf install python3.11

ARG user_id USER \${user_id}

5. Create a buildpack from the `hello-word` example [buildpack](https://github.com/buildpacks/samples/tree/main/buildpacks/hello-world), named `<path>/buildpack-hello-world:debug`.
6. Create a builder with the following `builder.toml`, named `<path>/builder-python3.11:debug`.

[[extensions]] uri = "/extension-dnf-install:debug" id = "dnf-install"

[[order-extensions]] [[order-extensions.group]] id = "dnf-install" version = "debug"

[[buildpacks]] uri = "/buildpack-hello-world:debug" id = "hello-world"

[[order]] [[order.group]] id = "hello-world" version = "debug"

[stack] id = "debug" run-image = "/run-base:debug" build-image = "/build-base-python3.11:debug"

7. Use the lifecycle phases in the `builder-python3.11:debug` image to create an image. Create the following files in the current directory.
`config/run.toml`

[[images]] image = "/run-base:debug"

`source/hello.sh`

!/bin/sh

echo hello

Then, run the following commands to build a container.

export image=/builder-python3.11:debug docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $(pwd)/config:/workspace/config -it --rm $image /bin/bash

These following commands are run within the container started in the immediately previous docker run command.

mkdir /kaniko export run_image=/stack-rhel8-run/amd64:debug export image=/test:debug DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/analyzer -cache-image=$image -uid=1000 -gid=1000 -run-image $run_image $image su cnb CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/detector -app=/workspace/source/ -run=/workspace/config/run.toml exit DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/restorer -cache-image=$image -uid=1000 -gid=1000 mkdir /layers/extended DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/extender -app=/workspace/source -kind=run -extended=/layers/extended

You'll see a result like this.

$ export image=/builder-python3.11:debug $ docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $ (pwd)/config:/workspace/config -it --rm $image /bin/bash
[root@102b97e9b6a4 layers]# mkdir kaniko
[root@102b97e9b6a4 layers]# rm -rf kaniko/
[root@102b97e9b6a4 layers]# mkdir /kaniko
[root@102b97e9b6a4 layers]# export run_image=/stack-rhel8-run/amd64:debug [root@102b97e9b6a4 layers]# export image=/test:debug [root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/analyzer -cache-image= $image -uid=1000 -gid=1000 -run-image $run_image $image Timer: Analyzer started at 2024-02-06T17:49:04Z
Image with name "/test:debug" not found Timer: Analyzer ran for 1.438062316s and ended at 2024-02-06T17:49:05Z [root@102b97e9b6a4 layers]# su cnb
[cnb@102b97e9b6a4 layers]$ CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/detector -app=/workspace/source/ -run=/workspace/config/run.toml
Warning: Platform requested experimental feature 'Dockerfiles' Timer: Detector started at 2024-02-06T17:49:11Z
dnf-install debug hello-world debug
Timer: Detector ran for 4.60913ms and ended at 2024-02-06T17:49:11Z Timer: Generator started at 2024-02-06T17:49:11Z
Timer: Generator ran for 841.845µs and ended at 2024-02-06T17:49:11Z [cnb@102b97e9b6a4 layers]$ exit exit
[root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/restorer -cache-image= $image -uid=1000 -gid=1000
Timer: Restorer started at 2024-02-06T17:49:21Z
Layer cache not found
Timer: Restorer ran for 1.114690805s and ended at 2024-02-06T17:49:22Z [root@102b97e9b6a4 layers]# mkdir /layers/extended
[root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lif ecycle/extender -app=/workspace/source -kind=run -extended=/layers/extended INFO[0000] Built cross stage deps: map[]
INFO[0000] Executing 0 build triggers
INFO[0000] Building stage 'base@sha256:79db881f1fa031623c738a9f0cf7a3dce4ceb2e0183c17d6364bd5b3cf0b3f02' [idx: '0', base-idx: '-1'] INFO[0000] Cmd: USER
INFO[0000] Checking for cached layer oci:/kaniko/cache/layers/cached:eba5636260b56b6f0c94d54b9a764036297f933fa03d198068c41c6e 723c055a... INFO[0000] No cached layer found for cmd RUN microdnf install python3.11 INFO[0000] Cmd: USER
INFO[0000] Unpacking rootfs as cmd RUN microdnf install python3.11 requires it. INFO[0000] Skipping unpacking as no commands require it. INFO[0000] USER root
INFO[0000] Cmd: USER
INFO[0000] No files changed in this command, skipping snapshotting. INFO[0000] RUN microdnf install python3.11
INFO[0000] Initializing snapshotter ...
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] Cmd: /bin/sh
INFO[0000] Args: [-c microdnf install python3.11]
INFO[0000] Util.Lookup returned: &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root} INFO[0000] Performing slow lookup of group ids for root INFO[0000] Running: [/bin/sh -c microdnf install python3.11]

(microdnf:72): librhsm-WARNING **: 17:49:30.110: Found 0 entitlement certificates

(microdnf:72): librhsm-WARNING **: 17:49:30.111: Found 0 entitlement certificates Nothing to do. INFO[0000] Taking snapshot of full filesystem...
INFO[0000] ARG user_id
INFO[0000] Pushing layer oci:/kaniko/cache/layers/cached:eba5636260b56b6f0c94d54b9a764036297f933fa03d198068c41c6e723c055a to cache now INFO[0000] No files changed in this command, skipping snapshotting. INFO[0000] USER \${user_id}
INFO[0000] Cmd: USER
INFO[0000] No files changed in this command, skipping snapshotting. INFO[0000] Skipping push to container registry due to --no-push flag Warning: The original user ID was app but the final extension left the user ID set to ${user_id}. Timer: Extender ran for 965.770627ms and ended at 2024-02-06T17:49:30Z

##### Current behavior
In the `extend` phase when it comes time to install `python3.11` it will say:

INFO[0000] Running: [/bin/sh -c microdnf install python3.11] Nothing to do.

despite `python3.11` not being installed in the run image.

##### Expected behavior
I expect `python3.11` to be installed in the run image since it is not installed in the base run image.

### Context

##### lifecycle version
Lifecycle version is 0.17.2.

##### platform version(s)
<!-- For example run `pack report` and `docker info` and copy output here, redacting any sensitive information. -->

$ pack report Pack: Version: 0.32.1+git-b14250b.build-5241 OS/Arch: linux/amd64

Default Lifecycle Version: 0.17.2

Supported Platform APIs: 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config: experimental = true lifecycle-image = "buildpacksio/lifecycle:0.17.1" layout-repo-dir = "/home//.pack/layout-repo"

$ docker info Client: Docker Engine - Community Version: 25.0.1 Context: default Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.) Version: v0.12.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.24.2 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 35
Running: 1
Paused: 0
Stopped: 34
Images: 51 Server Version: 25.0.1
Storage Driver: overlay2 Backing Filesystem: btrfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59 runc version: v1.1.11-0-g4bccb38 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.6.13-100.fc38.x86_64 Operating System: Fedora Linux 38 (Workstation Edition) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.03GiB Name: fedora ID: 6a61b820-9dc5-4e48-9607-b2df128276e9 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

[natalieparellano]

@yters looking at your code above, I think the issue is that you are trying to extend the run image in the context of the build image. So in this snippet:

export image=<path>/builder-python3.11:debug
docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $(pwd)/config:/workspace/config -it --rm $image /bin/bash
...

Where $image is the builder image - there is only one docker run instruction. There should be at least three (one for analyze/detect/restore/extend-build that uses $image, one for extend-run that uses $run_image, and one for export AFTER both extend phases have completed that uses $image). You would mount the same directories across all three invocations. If it's helpful here is how we do it in pack.

You may be interested in this (soon to be implemented) RFC which would make it easier to share things across the builder image and the run image: https://github.com/buildpacks/rfcs/pull/301

[yters]

Thanks for the response.

The docker run starts a terminal inside the builder docker container, and the lifecycle phase commands are run inside the container. I don't see how using the run image instead would work, since the run image doesn't have the lifecycle commands:

$ docker run --user root -it --rm <path>/run-base:debug /bin/sh
sh-4.4# ls /cnb/lifecycle
ls: cannot access '/cnb/lifecycle': No such file or directory

If I were to use the pack command to extend a run image instead of the lifecycle phase commands, should this work? I.e. the RPMs in the builder image will not conflict with the run image?

[natalieparellano]

the run image doesn't have the lifecycle commands

Ah yes, you are right. In the case of pack we download the lifecycle layer from the "lifecycle image" and append that to the run image before performing run image extension. You could accomplish something similar with volume mounts for local testing.

[yters]

Thanks, that's a good idea. I'll test it out, and close this ticket if successful.

[natalieparellano]

Any further update here? Can we close this issue?