Closed cmoulliard closed 4 months ago
So, the suggestion is that you can create a builder with a lifecycle specified as an image tag? specified just like you can with pack build ... --lifecycle-image
?
Or is it, more simply, that you can create a builder with a uri to a custom lifecycle archive? Because that's already possible to today, via the uri:
key: https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-_optional_
@aemengo do you mind if I move this issue over to pack?
It's been a while since I checked, but I believe that the uri
key expects to receive the path to a local (or maybe also remote?) .tgz file. It would be suuuuuper convenient if you could also specify uri: buildpacksio/lifecycle:919b8ad-linux-arm64
and then pack
just copies the lifecycle layer out of that image onto the builder it's creating.
Edit: this would avoid the need to compile or separately locate and download the lifecycle artifact for that SHA.
@natalieparellano I don't mind.
@cmoulliard To be clear, we're curious if the uri
key is enough to fit your use-case. Or do you require something more.
So, the suggestion is that you can create a builder with a lifecycle specified as an image tag?
Yes ;-)
Or is it, more simply, that you can create a builder with a uri to a custom lifecycle archive? Because that's already possible to today, via the
uri:
key: https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-_optional_
The limitation of the current approach is that it is needed to re-build a new builder image if we would like to use, when we will do a build, lifecycle creator v1.0 or v1.1 or v1.0-SNAPSHOT or v1.0-rc1 .... We dont have this problem using pack build
client locally as we can pass as parameter the lifecycle image to be used
To be clear, we're curious if the
uri
key is enough to fit your use-case. Or do you require something more.
I'm looking about something more to avoid to have to build a new builder image if I would like to use a different lifecycle creator
@cmoulliard Ok, thank you for clarifying.
So the suggestion is the builder can change the lifecycle in use (via a tag), at runtime? during the build? And this can happen independent of the pack CLI, it can happen on all platforms? say on Kubernetes?
I think this might call for a spec change, perhaps an RFC.
Do I miss a step. I built a new lifecycle and created a tgz file but process to create the builder
image fails
---> Creating builder for builder quarkus-native
Pulling image redhat/buildpacks-stack-quarkus-run:native
Pulling image redhat/buildpacks-stack-quarkus-build:native
Creating builder redhat/buildpacks-builder-quarkus-native:latest from build-image redhat/buildpacks-stack-quarkus-build:native
Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz
Looking up buildpack ../../buildpacks/quarkus
Downloading buildpack from URI: file:///Users/cmoullia/temp/quarkus-buildpacks/buildpacks/quarkus
Creating builder with the following buildpacks:
-> io.quarkus.buildpacks.buildpack@0.0.1
Adding buildpack io.quarkus.buildpacks.buildpack@0.0.1 (diffID=sha256:31e79cc2bea1be709f23e1dc7cdb152ea73b802b221ffe70f055b010430a0695)
Successfully created builder image redhat/buildpacks-builder-quarkus-native:latest
Tip: Run pack build <image-name> --builder redhat/buildpacks-builder-quarkus-native:latest to use this builder
Builder redhat/buildpacks-builder-quarkus-native:latest is already trusted
---> Creating builder for builder quarkus-jvm
Pulling image redhat/buildpacks-stack-quarkus-run:jvm
Pulling image redhat/buildpacks-stack-quarkus-build:jvm
Creating builder redhat/buildpacks-builder-quarkus-jvm:latest from build-image redhat/buildpacks-stack-quarkus-build:jvm
ERROR: failed to create builder: fetch lifecycle: invalid lifecycle: could not find entry path 'lifecycle.toml': not exist
Steps
$HOME/code/cncf/lifecycle
make clean build-linux-amd64
tar -czvf lifecycle.gz out/linux-amd64/lifecycle
edit `builder.toml` file of the builder project
[lifecycle]
uri = "$HOME/code/cncf/lifecycle/lifecycle.gz"
Ah, use make clean build package linux-amd64
, that will add the lifecycle.toml.
Ah, use
make clean build package linux-amd64
, that will add the lifecycle.toml.
No luck
make clean build package build-linux-amd64
> Cleaning workspace...
rm -rf /Users/cmoullia/code/cncf/lifecycle/out
> Building lifecycle/lifecycle for linux/amd64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle
GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/lifecycle -a ./cmd/lifecycle
> Creating phase symlinks for linux/amd64...
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/detector
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/analyzer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/restorer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/builder
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/exporter
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/rebaser
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/creator
> Building lifecycle/launcher for linux/amd64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle
GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/launcher -a ./cmd/launcher
test $(du -m /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/launcher|cut -f 1) -le 3
> Building lifecycle/lifecycle for linux/arm64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle
GOARCH=arm64 CGO_ENABLED=0 go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/lifecycle -a ./cmd/lifecycle
> Creating phase symlinks for linux/arm64...
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/detector
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/analyzer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/restorer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/builder
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/exporter
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/rebaser
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/creator
> Building lifecycle/launcher for linux/arm64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle
GOARCH=arm64 CGO_ENABLED=0 go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/launcher -a ./cmd/launcher
test $(du -m /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/launcher|cut -f 1) -le 3
> Building lifecycle/lifecycle for windows/amd64...
go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/lifecycle.exe -a ./cmd/lifecycle
> Creating phase symlinks for Windows...
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/detector.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/analyzer.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/restorer.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/builder.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/exporter.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/rebaser.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/creator.exe
> Building lifecycle/launcher for windows/amd64...
go build -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/launcher.exe -a ./cmd/launcher
> Packaging lifecycle for linux/amd64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.x86-64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
> Packaging lifecycle for linux/arm64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.arm64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
> Packaging lifecycle for windows/amd64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+windows.x86-64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
make: Nothing to be done for `build-linux-amd64'.
~/code/cncf/lifecycle main ls -la out/linux-amd64/lifecycle
total 27048
drwxr-xr-x 11 cmoullia staff 352 Oct 7 19:53 .
drwxr-xr-x 3 cmoullia staff 96 Oct 7 19:53 ..
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 analyzer -> lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 builder -> lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 creator -> lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 detector -> lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 exporter -> lifecycle
-rwxr-xr-x 1 cmoullia staff 2478080 Oct 7 19:53 launcher
-rwxr-xr-x 1 cmoullia staff 11370496 Oct 7 19:53 lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 rebaser -> lifecycle
lrwxr-xr-x 1 cmoullia staff 9 Oct 7 19:53 restorer -> lifecycle
~/code/cncf/lifecycle main ls -la out/linux-amd64/
total 0
drwxr-xr-x 3 cmoullia staff 96 Oct 7 19:53 .
drwxr-xr-x 11 cmoullia staff 352 Oct 7 19:53 ..
drwxr-xr-x 11 cmoullia staff 352 Oct 7 19:53 lifecycle
For whatever reason the lifecycle.toml file is only created in the packaging step, so it will be part of out/lifecycle-v0.12.0-rc.1-6+de98d3f+linux.x86-64.tgz
but not out/linux-amd64
.
I provided the incorrect command before, make clean build package
or make clean build-linux-amd64 package-linux-amd64
should be sufficient.
So the suggestion is the builder can change the lifecycle in use (via a tag), at runtime? during the build? And this can happen
Ideally, at runtime, when by example we execute a build using a container, pod and where we would like to override the lifecycle pakaged within the image
@cmoulliard pack
won't be able to do what you're asking. If you're daring, next steps would be to create an RFC proposal and we'd likely discuss it at an upcoming community working group meeting
@cmoulliard
pack
won't be able to do what you're asking.
This is not what I'm asking. What I would like is to be able to specify the version of the lifecycle to be used when a pod is doing a build.
- name: export
image: registry.local:5000/redhat-builder/quarkus
command:
- /cnb/lifecycle/exporter
args:
That will certainly then need that a pre-step is added (= initContainer) to load the needed go binary before to launch the build ...
Again, it's still not clear what you want. If you:
Want the "builder to download the specified lifecycle" into its container, at runtime, this will require an RFC proposal. It goes against the principles of reproducibility to have core components on a container image be switched out, after the container is built.
Want to change the image used for the export
pod in your k8s definition, then you can simply do that:
- name: export
# image: registry.local:5000/redhat-builder/quarkus
image: buildpacksio/lifecycle:919b8ad-linux-arm64
command:
- /cnb/lifecycle/exporter
args:
This is exactly how the pack
behaves when you use the --lifecycle-image
flag during pack build
. Only in cases when the builder is not trusted, and only for the analyze, restore, export phases
Want to use initContainers
to download the lifecycle binary to a volume and so that you can swap out the /cnb/lifecycle/...
directories in the following steps, then you can simply do that.
For whatever reason the lifecycle.toml file is only created in the packaging step, so it will be part of
out/lifecycle-v0.12.0-rc.1-6+de98d3f+linux.x86-64.tgz
but notout/linux-amd64
.I provided the incorrect command before,
make clean build package
ormake clean build-linux-amd64 package-linux-amd64
should be sufficient.
Thanks. That did the job :-)
#lifecycle project
make clean build-linux-amd64 package-linux-amd64
# and next creating a new builder
Creating builder redhat/buildpacks-builder-quarkus-native:latest from build-image redhat/buildpacks-stack-quarkus-build:native
Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz
Looking up buildpack ../../buildpacks/quarkus
where `builder.toml`
[lifecycle]
uri = "file:///Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.x86-64.tgz"
WARNING: The message logged using the verbose mode by pack builder
gives the impression that the version packaged is Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz
instead of what we pass as uri.
That should be improved as the only way now to figure out which version is packaged is to do:
docker run --rm -it redhat/buildpacks-builder-quarkus-jvm /cnb/lifecycle/creator -version
0.12.0-rc.1-4+919b8add
@ameyer-pivotal @natalieparellano
@cmoulliard To be clear, we're curious if the
uri
key is enough to fit your use-case.
uri is enough.
Isue: uri do not work for the lifecycle property if we pass an image
[lifecycle]
uri = "buildpacksio/lifecycle:919b8ad-linux-arm64"
--->
pack builder ...
ERROR: failed to create builder: fetch lifecycle: invalid lifecycle: open lifecycle blob: read blob at path '/Users/cmoullia/temp/quarkus-buildpacks/builders/quarkus-jvm/buildpacksio/lifecycle:919b8ad-linux-arm64': stat /Users/cmoullia/temp/quarkus-buildpacks/builders/quarkus-jvm/buildpacksio/lifecycle:919b8ad-linux-arm64: no such file or directory
@aemengo
@cmoulliard Write RFC
Hi @cmoulliard
Checking this issue, I think the issue now is that you find a bug referencing an lifecycle image using the uri
field, right? as you mentioned here
In that case, I can remove the status/requires-rfc
label and convert this one into a bug
I can remove the
Yes
Closing as a duplicate of https://github.com/buildpacks/pack/issues/2076
Description
When we would like to use a different lifecycle release from the list of the published releases - see: https://github.com/buildpacks/lifecycle/releases), then we can set the property
version
within thebuilder.toml
file to include it within the builder image :Such a version will then generate a request to download the following artifact "
Downloading from https://github.com/buildpacks/lifecycle/releases/download/v0.12.0/lifecycle-v0.12.0+linux.x86-64.tgz
"How would it be possible to use a lifecycle version which is not yet released but where image has been created from the github commit id ?
Example for the commit: 919b8ad
I think that we should investigate to propose a solution as by example this is is something that we can do using
pack build
using this parameterpack build ... --lifecycle-image buildpacksio/lifecycle:919b8ad-linux-arm64
but not if we execute the lifecycle creator part of kubernetes pod