search

buildpacks / lifecycle

Reference implementation of the Cloud Native Buildpacks lifecycle
https://buildpacks.io
Apache License 2.0
186 stars 105 forks source link

Support ARM #435

Open ekcasey opened 3 years ago

ekcasey commented 3 years ago

GOAL: Support buildpack builds for ARM images

WARNING Not yet ready to implement, may require RFCs, spec changes and/or imgutil changes.

Lifecycle Behavior

analyzer

  1. If the previous image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
  2. Warn and proceed without analyzing the previous image if there the image exists but has the wrong architecture.

Because analyzer may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

  1. Record the information in detector and pass it to analyzer.
  2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

exporter

  1. If the run-image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
  2. Fail if there is no run-image with the correct architecture
  3. If there was a previous image, validate that the previous image architecture and the run image architecture match.

Because exporter may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

  1. Record the information in a previous phase and pass it to analyzer.
  2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

rebaser

  1. If the run-image is a multiarch image, select the child manifest that matches the app image architecture
  2. If the run-image is not a multiarch image rebaser should validate that the architecture matches the app-image before rebasing.

Lifecycle Artifacts

Publish ARM lifecycle artifacts

Add the following new Makefile targets:

And update the existing targets:

Local ARM image builds

Add the following new Makefile targets:

Docker desktop should support both building for multiple architectures. I do not know if a daemon running on a x86 linux machine will allow us to load arm64 images.

Note We cannot publish arm windows images becausemcr.microsoft.com/windows/servercore:ltsc2019 does not include an arm manifest (unlikemcr.microsoft.com/windows/nanoserver:1809 which we recently moved away from).

Publish ARM images

Instead of publish buildpacksio/lifecycle:<commit>-linux images for each build, publish

The buildpacksio/lifecycle:<commit> multiarch image should contain the windows image and both linux architectures.

When we release a new lifecycle version we should retag appropriate images with

And ensure that both buildpacksio/lifecycle:latest and buildpacksio/lifecycle:<version> include both linux architectures.

dfreilich commented 3 years ago

This issue focuses on support for arm64. There's an issue open by pack (https://github.com/buildpacks/pack/issues/907) re: arm64 and armv7 support – should I open a separate issue for armv7, or can we tack that on to this?

dfreilich commented 3 years ago

Also, for clarity sake, what RFC(s) would you imagine we would need to drive this through?

natalieparellano commented 3 years ago

Possibly relevant exploration around building ARM images on x86: https://github.com/buildpacks/samples/pull/60/files

dfreilich commented 3 years ago

Would we need separate lifecycle images for darwin/arm64 as well? If so, should I open up a new issue for that? Related: https://github.com/buildpacks/pack/issues/1003

tibeer commented 3 years ago

Ran into this issue today. Just for your information: I do not use lifecycle directly but rather by trying to use Hashicorps waypoint to spin up a Kubernetes deployment (of any container) on my local Kubernetes cluster. I am working on a M1 Mac.

ryuheechul commented 3 years ago

👍🏼 for this issue and I'm pretty much repeating what @tibeer said above.

Just to report a use-case, currently https://learn.hashicorp.com/tutorials/waypoint/get-started-docker?in=waypoint/get-started-docker would not work on M1 laptops with this error.

I can see there was this issue, https://github.com/buildpacks/pack/issues/1003 (in another repo) relating to m1 was resolved but that seems to be for the native binary building not container image building. And this, https://github.com/buildpacks/pack/issues/1171 also to seems be a related issue.

❯ waypoint up

» Building...
Creating new buildpack-based image using builder: heroku/buildpacks:18
✓ Creating pack client
❌ Building image
 │ 2021/06/10 21:32:23.990480 DEBUG:  Pulling image index.docker.io/heroku/buildpac
 │ ks:18

...

 │ 2021/06/10 21:32:27.168139 DEBUG:  -> heroku/nodejs-function-invoker@0.1.5
 │ 2021/06/10 21:32:27.168142 DEBUG:  -> heroku/nodejs-npm@0.4.3
 │ 2021/06/10 21:32:27.168145 DEBUG:  -> heroku/nodejs-typescript@0.2.2
 │ 2021/06/10 21:32:27.168150 DEBUG:  -> heroku/nodejs-engine@0.7.3
 │ 2021/06/10 21:32:27.168154 DEBUG:  -> evergreen/fn@0.3.0
 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries
imjasonh commented 3 years ago

I looked into this a bit out of curiosity, and it looked like building the lifecycle binary into an image would be fairly straightforward to accomplish with ko, not just for linux/arm64 but also more exotic architectures like linux/ppc64le -- basically anything go build supports, plus what you can find a base image for -- and after https://github.com/google/ko/pull/374 for Windows as well.

Essentially, ko does what your image build process already does: it go builds the binary for the desired output os+architecture, tars it up, appends it to a base image, and either puts it in the local daemon or pushes it to a registry.

The only hitch seems to be the use of cgo in priv/user_linux.go. If there's any way to avoid that, I think you could get multi-arch lifecycle images for lifecycle relatively easily with ko.

buildpacksio/pack doesn't have this issue, so you could get a multi-arch image for that even easier.

I'd be happy to help push this forward if this seems like a reasonable path. I understand the use of cgo might be unavoidable, or that changing the release process might be undesirable.

micahyoung commented 3 years ago

I feel like it's a great suggestion @imjasonh. Having more standardization and tooling around generating these foundational images would be a big win (selfishly, particularly for the Windows images). I'm not sure how maintainers might feel about migrating to ko, but if you're up for it, I think starting a separate issue here would be enough to kick off the discussion. I'll gladly add my +1.

natalieparellano commented 3 years ago

@ryuheechul it looks like your issue most closely relates to https://github.com/buildpacks/pack/issues/1197. Just to confirm, are you trying to build a linux/amd64 image? If so, I think a change in pack will unblock you. Is there any way to configure Waypoint to trust the provided builder?

There seem to be several threads on this issue, I wonder if it's worth breaking them out:

natalieparellano commented 3 years ago

Just wanted to let anyone interested know that there is an #arm channel in buildpacks Slack. This thread provides a somewhat mind-boggling matrix of possible supported scenarios. Please feel free to add your thoughts here!

ryuheechul commented 3 years ago

@natalieparellano Thanks for your comment. Btw, I'm not trying to build anything, all the logs are triggered by Waypoint itself and based on the error logs at the bottom,

 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

I think it's simply because the docker image is not built for linux/armX as I can see it here, https://hub.docker.com/r/buildpacksio/lifecycle/tags.

But then, I'm only guessing at this point so I might be wrong.

warlockdn commented 3 years ago

Do you guys have anything planned for arm architecture ? I also came across this issue while using waypoint.

imjasonh commented 3 years ago

After #659 (and #660 😓), the latest lifecycle image (buildpacksio/lifecycle:caab214) built from head now includes a linux/arm64 image:

$ crane manifest buildpacksio/lifecycle:caab214 | jq '.manifests[].platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.17763.2029"
}
warlockdn commented 3 years ago

does this solve no matching manifest for linux/arm64/v8 issue ?

natalieparellano commented 3 years ago

@warlockdn I believe that issue is more likely to be solved by the fix for https://github.com/buildpacks/pack/issues/1197, which will be available in pack 0.20.0.

warlockdn commented 3 years ago

Wow. That's just great to hear. Looking forward. @natalieparellano 0.20 releases next ?

natalieparellano commented 3 years ago

@warlockdn yep! 0.20.0 should be out next week or the following.

willianpaixao commented 3 years ago

Hey @natalieparellano, pack-v0.20.0 was released but I couldn't find buildpacks/pack#1197 in the Changelog. Is that right?

natalieparellano commented 3 years ago

@willianpaixao I believe it's fixed by https://github.com/buildpacks/pack/pull/1216. Does upgrading resolve your issue? Please let us know!

cihantas commented 3 years ago

@natalieparellano It appears like the issue isn't resolved. Using v0.11.4 on an M1 Mac still results in the same error message.

natalieparellano commented 3 years ago

@cihantas just confirming you are using pack v0.20.0 (v0.11.4 is the lifecycle version).

Edit: also just to confirm - are you trying to build an amd64 image?

cihantas commented 3 years ago

@natalieparellano Yes, I can confirm I'm using v0.11.4.

I'm using pack in the context of Hashicorp Waypoint. Not sure if I'm building the amd64 version. Is there a way for me to verify this?

aemengo commented 3 years ago

@cihantas Thanks for calling this out. This is a great find.

It seems that there's been some confusion (myself included) about how Hashicorp Waypoint works. Upgrading the pack CLI on your local machine won't have any effect because Waypoint is using pack via its golang API library. As of right now, Waypoint's pack version is pinned to v0.18.1.

There's the quicker fix which is to make a PR bumping Waypoint's pinned pack version to v0.20.0, like @natalieparellano suggested.

But I think you've highlighted a nascent problem with the pack Golang API library: the "trusted builder code path" isn't exercised at all. For trusted builders (like their default heroku/buildpacks:18), the lifecycle shouldn't even be downloaded. And more importantly, it means that the lifecycle's newer creator workflow isn't getting exercised.

--

If no one else wants to have the fun, I'll make the appropriate PRs in Waypoint, and pack, GitHub repositories - within the next couple days.

EDIT:

aemengo commented 3 years ago

@cihantas @natalieparellano and other interested folks,

Just wanted to highlight that hashicorp released waypoint v0.5.0+ which should have the architecture fix. The following error should no longer be present:

 # ...
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries
imjasonh commented 3 years ago

buildpacksio/lifecycle is still not released with arm64 support: https://explore.ggcr.dev/?image=buildpacksio%2Flifecycle

680 should fix this (after another release)

donnyv12 commented 3 years ago

another M1 Mac user who would love to use buildpacks here :wave:

imjasonh commented 3 years ago

As of v0.12.0-rc.1, the lifecycle image includes a linux/arm64 build -- the image is buildpacksio/lifecycle:0.12.0-rc.1.

For Waypoint users on M1, I'm not sure how to configure Waypoint to use this image, or if it's even configurable. If someone can manage to figure that out and try out the 0.12 RC and report back, that would be really helpful.

edit: Or will this require pack to bump its dep on lifecycle, and waypoint to bump its dep on pack, and do a release? 🤷

aemengo commented 3 years ago

@imjasonh The Waypoint issue is orthogonal to this "Support ARM" issue (and should be resolved with the latest Waypoint release). Let's please open another issue about Waypoint if someone is still finding issue.

agracey commented 2 years ago

What's the status of this? I see a lot of arm related build steps and artifacts in the makefile. If I build a builder with the arm binaries and ran it on a cluster that's arm only, would it fail?

natalieparellano commented 2 years ago

@agracey thanks for the feedback - we should probably close this issue or update it to include only the work yet to be done. If you have a builder where the base image, lifecycle, and buildpacks are all based on arm, and you run it on arm, it should produce an arm image.

The work still to be done is of the nature of edge cases that may arise when e.g., the lifecycle is running on one architecture but the desired output image is of another architecture. There is a matrix of possibilities that we need to evaluate to determine if they can be supported.

xlf12 commented 2 years ago

We have started migrating our development boxes from x86 to Apple Silicon. A packing on arm towards x86 would be welcome ;-)

... just to put 2 cent in this discussion ;-)

agracey commented 2 years ago

That's good to hear! Thank you for the update.

vnobo commented 1 year ago

How is it going?

dlion commented 1 year ago

@vnobo @jjbustamante is actually working on this topic, you can follow this tracking issue https://github.com/buildpacks/rfcs/issues/294 to follow the progress

jjbustamante commented 1 year ago

I want to summarize some stuffs because this issue contains a lot of different threads and things to be done.
I think the original idea for this ticket was completed, the latest version of lifecycle and pack released till date are multi-arch images.

> crane manifest buildpacksio/lifecycle:0.17.0 | jq '.manifests[].platform'
{
  "architecture": "arm64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.17763.4737"
}
> crane manifest buildpacksio/pack:0.30.0 | jq '.manifests[].platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm64",
  "os": "linux"
}

With that been said, I like to point out this comment from @natalieparellano

If you have a builder where the base image, lifecycle, and buildpacks are all based on arm, and you run it on arm, it should produce an arm image.

We already know you can run pack build on ARM and it works correctly. @dmikusa and @dashaun have these incredible repos that are very helpful

What is missing? We want to deliver some tooling and capabilities to the CNB ecosystem that allow Buildpack authors to package and distribute their Builders and Buildpacks supporting multi-arch and that's the goal being tracked by the issue mentioned above from @dlion, also we want to actually add support in pack to run pack build --platform, or something like that, to output an application image for different os/arch (check last kubecon 2022 talk) from @AidanDelaney, but we are still not there.