search

buildpacks / lifecycle

Reference implementation of the Cloud Native Buildpacks lifecycle
https://buildpacks.io
Apache License 2.0
187 stars 107 forks source link

Spike: Dockerfiles POC #709

Closed natalieparellano closed 2 years ago

natalieparellano commented 3 years ago

Description

To help verify the feasibility of https://github.com/buildpacks/rfcs/pull/173 and work out some of the thorny details that would go in a spec PR, we should spike out a POC lifecycle that supports Dockerfiles. This would eventually also allow potential end users to tinker with the feature and provide their feedback.

Proposed solution

Let's create a dockerfiles-poc branch on the lifecycle with the following functionality. See linked issues for individual units of work that should be more-or-less parallelizable.

Note: "gross" code is okay. Out of scope:

natalieparellano commented 3 years ago

dockerfiles-poc branch is created. Let's make PRs into that branch (vs the main branch).

cmoulliard commented 3 years ago

Questions:

natalieparellano commented 3 years ago

@cmoulliard since support for this feature in pack will need to come after the lifecycle, we could use Tekton to test it. I'm envisioning something similar to https://github.com/buildpacks/lifecycle/issues/681

cmoulliard commented 3 years ago

we could use Tekton to test it

This is an option but we will perhaps faced to an issue as openshift will create a pod where the user is not allow to execute root privileged commands. I will check.

phracek commented 3 years ago

I would like to help with coding/writing tests in Go, But I am a newbie in Go. Do you have some parts that can be done from my side?

natalieparellano commented 3 years ago

@phracek we would love to have your help! Do any issues in the epic look interesting to you? I think that #716 and #717 would require the least intricate knowledge of the buildpacks spec, if you would like to play around with https://github.com/GoogleContainerTools/kaniko.

cmoulliard commented 3 years ago

We need also to create a sample project containing the different files such as Dockerfile, Hook TOML, ... to help the guys who will develop or test to know what the foundation should be.

natalieparellano commented 3 years ago

Please see https://github.com/buildpacks/samples/compare/dockerfiles-poc?expand=1 . It is by no means complete, but I am hoping that we could expand upon it.

Some initial validation steps:

Edit: notable features of the RFC that are not exercised: build plan interactions, "single stage" Dockerfiles e.g., build.Dockerfile or run.Dockerfile

cmoulliard commented 3 years ago

Please see https://github.com/buildpacks/samples/compare/dockerfiles-poc?expand=1 . It is by no means complete, but I am hoping that we could expand upon it.

Can you create a PR to allow us to review/comment it please ? @natalieparellano

natalieparellano commented 3 years ago

Sure, done! I made it a draft as I'm pretty sure it's not mergeable yet :)

cmoulliard commented 3 years ago

To be sure that we are on the page, can you review the following description please ?

The current lifecycle workflow executes the following steps: Detect -> Analyze -> Restore -> Build -> Export To support the Dockerfiles it will be needed that a new step is included within the workflow in order to augment the existing build and/or run images to package additional tools, libs, executables using root privileged user or to execute root commands.

Such a step should take place after the detect phase to determine based on the [[entries.requires]] if some extensions could be executed to resolve the requirements.

Example A. During the detect phase execution, it appears that the project to be build is using Maven as compilation/packaging tool and Java is the language framework. Then detect will generate within the plan.toml the following entries

  [[entries.requires]]
    name = "Maven"
    [entries.requires.metadata]
      version = "3.8.1"

  [[entries.requires]]
    name = "JDK"
    [entries.requires.metadata]
      version = "11"

REMARK: Ideally the name (OR ID) of the required entry should map the name of the package listed part of the SBoM AND PURL (https://cyclonedx.org/use-cases/).

Next, the new step that we could call extend will check if there is a matching between the id + version of the extensions and the entries = requirements. If this is the case, then the extension(s) will be executed to perform a container build using either the Dockerfile provided within the extension folder or to execute the build/bin able to produce a Dockerfile on the fly

REMARK: If a build/detect is part of the extension, then it could be used to collect the ENV or Parameters to be used to execute the container build command, to check the OS (amd, arm, ...), ....

WDYT: @natalieparellano @jkutner @aemengo @sclevine

aemengo commented 3 years ago

Minor comment: as of PLATFORM: 0.7, Analyze and Detect are switched. It's now: Analyze -> Detect -> Restore -> Build -> Export. Otherwise, that was my understanding.

cmoulliard commented 3 years ago

Otherwise, that was my understanding.

So the new step should be inserted here: https://github.com/buildpacks/lifecycle/blob/8c2edb2a85f5aaa1f46babcaf300fe3a84c85505/cmd/lifecycle/creator.go#L242 ? @aemengo

aemengo commented 3 years ago

@cmoulliard Yes, that seems appropriate.

Especially because this is still a POC, it would probably be expedient to make a discrete step there and then we could revise after.

It would be a big help, if you're able to get this moving for the community 🙂

cmoulliard commented 3 years ago

FYI: We need also a new acceptance test case validating some of the scenario we will test using the extend phase

cmoulliard commented 3 years ago

Question: Should we use the docker client part of the lifecycle Dockerfiles change to perform an container build or to execute a command using the container client (docker, podman, ....) @aemengo @sclevine @natalieparellano @jabrown85

jabrown85 commented 3 years ago

@cmoulliard the RFC says that is up to the platform - not exactly sure how that is intended to work. I would think that lifecycle ships with a kaniko based implementation.

cmoulliard commented 3 years ago

with a kaniko based implementation.

AFAIK, kaniko is shipped with their own image and cannot be used as go lib within the lifecycle application to execute a build - see: https://github.com/GoogleContainerTools/kaniko#running-kaniko

The lib to be used and which could help us is buildah (already used by podman): https://github.com/containers/podman/blob/main/cmd/podman/images/build.go

WDYT ? @jabrown85 @aemengo @natalieparellano

cmoulliard commented 3 years ago

FYI: I create a simple buildah go app able to build a dockerfile - https://github.com/redhat-buildpacks/poc/blob/main/bud/bud.go#L85

cmoulliard commented 3 years ago

Questions:

@natalieparellano @jabrown85 @aemengo @sclevine

jabrown85 commented 3 years ago
  • Will the new feature build image from Dockerfiles of lifecycle publish the build or run image(s) published on a registry ?

I don't think so, no. The idea, as I understand it, is to execute the Dockerfile commands live on the build image that lifecycle is already executing on (inside of builder) right before executing buildpacks.

For the run image, the new layers created during the execution of the Dockerfile would carry over in a volume so the exporter would take run image + extension layers + buildpack layers.

Does that make sense?

cmoulliard commented 3 years ago

I don't think so, no. The idea, as I understand it, is to execute the Dockerfile commands live on the build image that lifecycle is already executing on (inside of builder) right before executing buildpacks.

Make sense and that will simplify our life ;-) if we dont have to push it somewhere

cmoulliard commented 3 years ago

For the run image, the new layers created during the execution of the Dockerfile would carry over in a volume so the exporter would take run image + extension layers + buildpack layers.

Is the exporter able to publish the newly image created then ?

jabrown85 commented 3 years ago

Is the exporter able to publish the newly image created then ?

Yes, the exporter gets registry credentials mounted from the platform to push the resulting app image. It creates a new manifest using the manifest of the run image + buildpack layers and exports that manifest and layers to the destination (docker, registry).

cmoulliard commented 3 years ago

is to execute the Dockerfile commands live on the build image

We should also define a convention to name the image newly created as it could be cached to be reused for a next build and by consequence first searched before to execute the command to apply the docker file ? WDYT

jabrown85 commented 3 years ago

Caching is not identified in the RFC today. I think caching may start off as a platform-specific feature.

cmoulliard commented 3 years ago

Caching is not identified in the RFC today. I think caching may start off as a platform-specific feature.

Ok but then it will be needed when pack build or kpack build will take place to execute every time the build Dockerfiles ? @jabrown85

jabrown85 commented 3 years ago

Ok but then it will be needed when pack build or kpack build will take place to execute every time the build Dockerfiles ? @jabrown85

That is how I understand it. Each build.Dockerfile, may need to re-executed on each build. Any optimizations to that have yet to be identified AFAIK. A platform could execute the build.Dockerfile and then run builder on top of that image and use whatever caching/registry they want, but it isn't defined. A platform could execute the build.Dockerfile instructions in the same container that builder runs on. If a platform used kaniko to apply the instructions, the platform may be able store the snapshots in a platform specific cache volume for subsequent rebuilds.

cmoulliard commented 3 years ago

I was able to create a simple POC which supports the concept to apply a Dockerfile. It uses a go buildah lib which can create an image from a Dockerfiles, next extract the layer(s) content. The code used is equivalent to the buildah bud command.

https://github.com/redhat-buildpacks/poc/blob/main/k8s/manifest.yml#L72-L135

Remark: As it is needed to modify the path to access the layer(s) content extracted within a volume (e.g. /layers --> export PATH="/layers/usr/bin:$PATH", ...), then the RFC perhaps should include an additional ARG or ENV VAR (CNB_EXTENSION_LAYERS) to let to specify where the layers will be extracted in order to change the $PATH during the build step

WDYT: @natalieparellano @sclevine @aemengo @jabrown85

cmoulliard commented 3 years ago

I talked a lot with Giuseppe Scrivano today (podman project) and I think that I found the right tools/projects (skopeo, umoci) to manage the images after the dockerfiles have been applied. As umoci supports to unpack/repack an image, we could imagine to merge the content generated by the execution of the Dockerfiles into a snapshot builder image that next lifecycle will use to perform the build. Same thing could also take place for the runtime image if additional stuffs should be added.

  1. Buildah bud 
    
cat <<EOF > Dockerfile
FROM registry.access.redhat.com/ubi8:8.4-211

RUN yum install -y --setopt=tsflags=nodocs nodejs && \ rpm -V nodejs && \ yum -y clean all EOF REPO="buildpack-poc" sudo buildah bud -f Dockerfile -t $REPO .


2. Skopeo
We can extract locally the content of an image

GRAPH_DRIVER="overlay" TAG=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $2;}') IMAGE_ID=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $3;}') sudo skopeo copy containers-storage:$IMAGE_ID oci:$(pwd)/$IMAGE_ID:$TAG


2. Umoci
We extract the blob and next we can merge them into the image using `repack`

sudo umoci unpack --image $IMAGE_ID:$TAG bundle

**Remark**: If we use Tekton, then no need to develop something else as we could apply some pre-steps (= initcontainer) to perform the execution of steps 1-2 and 3. For local development using pack then, that will be a different story !!

End to end script tested

sudo rm -f _temp && mkdir _temp pushd _temp

cat < Dockerfile FROM registry.access.redhat.com/ubi8:8.4-211

RUN yum install -y --setopt=tsflags=nodocs nodejs && \ rpm -V nodejs && \ yum -y clean all EOF

REPO="buildpack-poc" sudo buildah bud -f Dockerfile -t $REPO .

GRAPH_DRIVER="overlay" TAG=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $2;}') IMAGE_ID=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $3;}') sudo skopeo copy containers-storage:$IMAGE_ID oci:$(pwd)/$IMAGE_ID:$TAG

sudo ls -la $IMAGE_ID sudo ls -la $IMAGE_ID/blobs/sha256/

sudo ../umoci unpack --image $IMAGE_ID:$TAG bundle

sudo ls -la bundle sudo ls -la bundle/rootfs total 4 dr-xr-xr-x. 18 root root 242 Sep 14 16:20 . drwx------. 3 root root 142 Oct 29 14:50 .. lrwxrwxrwx. 1 root root 7 Apr 23 2020 bin -> usr/bin dr-xr-xr-x. 2 root root 6 Apr 23 2020 boot drwxr-xr-x. 2 root root 6 Sep 14 16:19 dev drwxr-xr-x. 50 root root 4096 Oct 29 14:46 etc drwxr-xr-x. 2 root root 6 Apr 23 2020 home lrwxrwxrwx. 1 root root 7 Apr 23 2020 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Apr 23 2020 lib64 -> usr/lib64 drwx------. 2 root root 6 Sep 14 16:19 lost+found drwxr-xr-x. 2 root root 6 Apr 23 2020 media drwxr-xr-x. 2 root root 6 Apr 23 2020 mnt drwxr-xr-x. 2 root root 6 Apr 23 2020 opt drwxr-xr-x. 2 root root 6 Sep 14 16:19 proc dr-xr-x---. 3 root root 213 Sep 14 16:38 root drwxr-xr-x. 5 root root 66 Oct 29 14:46 run lrwxrwxrwx. 1 root root 8 Apr 23 2020 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Apr 23 2020 srv drwxr-xr-x. 2 root root 6 Sep 14 16:19 sys drwxrwxrwt. 2 root root 58 Oct 29 14:46 tmp drwxr-xr-x. 13 root root 155 Oct 29 14:46 usr drwxr-xr-x. 19 root root 249 Sep 14 16:20 var

popd


Remark: buildah, skopeo and umoci should be installed to play with the technology before we integrate them within the lifecycle
jabrown85 commented 3 years ago

As @sclevine mentioned in slack, I think we should concentrate on one implementation at a time if possible. Stephen listed two distinct implementations that we could concentrate on. I, personally, would like to concentrate on the pure userspace version first.

Here is what I imagined could happen at a high level.

For build extensions, a new phase executor build-extender would exist between detector and builder that would parse the Dockerfile(s) and execute the steps directly - not involving any docker daemon or images. It would assume the platform executing the dockerfiles is ensuring the same builder image is used throughout a build. After each dockerfile is executed, a diff of the changes could be stored into a volume. Then, builder would run as root and restore these changes prior to buildpack execution. It would drop permissions prior to executing buildpacks. If the diffing and storing is too expensive, we could start out by running the dockerfile steps in builder and then dropping permissions before executing the buildpacks. This has fewer moving parts and is likely a good place to start a PoC, but things like caching are more difficult.

For run extensions, a new phase run-extender would exist before exporter and after detector. A container would run against the run image, with the run-extender mounted in (or it could already be there). The run-extender like the build-extender would execute the dockerfile steps. Before each new buildpack, the resulting diff of changes would be stored into a volume. After all extensions have processed, the new phase exits. The exporter now has access to both extension layers and buildpack layers and would be responsible for stitching them up against the run image directly against the registry. I'm not sure I understand how run-extender could happen in a single container flow (e.g. creator), but maybe @sclevine had ideas around that.

cmoulliard commented 3 years ago

Then, builder would run as root and restore these changes prior to buildpack execution. It would drop permissions prior to executing buildpacks.

If I understand correctly what you say here, the idea is to execute the lifecycle builder step as 'root and end of the step, the permissions should be restored to what it is defined according to the images CNB_** parameters - correct ?

If the answer is yes, then I suppose that it will be needed that part of Tekton, kpack builds(= openshift or kubernetes builds executed using a pod), that the privileged option is enabled (https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) - correct ?

@jabrown85

natalieparellano commented 2 years ago

Please see https://github.com/buildpacks/lifecycle/pull/786 for a lifecycle that goes part of the way toward meeting the acceptance criteria outlined in this comment by:

This could be paired with the "extender" POC that @cmoulliard is working on.

natalieparellano commented 2 years ago

I think this spike has more or less served its purpose. Through https://github.com/buildpacks/lifecycle/pull/802 we were able to identify many of the changes that would be necessary, which are outlined in https://github.com/buildpacks/spec/pull/298.

https://github.com/buildpacks/spec/pull/308 and https://github.com/buildpacks/spec/pull/307 break this down further into changes that would be required for "phase 1" of the implementation (using Dockerfiles to switch the runtime base image, only). https://github.com/buildpacks/lifecycle/issues/849 can be used to track the work for this.

With all that said I think we can close this issue, and related spike issues.