search

tektoncd / pipeline

A cloud-native Pipeline resource.
https://tekton.dev
Apache License 2.0
8.44k stars 1.77k forks source link

Add support for other architectures #856

Open vdemeester opened 5 years ago

vdemeester commented 5 years ago

Kubernetes support more architecture than x86_64, Tekton Pipeline should support those too.

What this means is :

Architecture to support

Architecture supported

Marking this as an epic as it is a big chunk of work and is gonna need to be split up into several issues.

lisa commented 5 years ago

As part of my investigation that @vdemeester mentioned I had to change the base images in .ko.yaml*, even though busybox has an arm64 manifest. I believe this is because in google/ko's commands/config.go which exhibits underlying behaviour in go-containertools (described https://github.com/GoogleContainerTools/kaniko/pull/646 as it relates to kaniko) to not pick image manifests according to what arch is currently making the request. In short, this issue likely blocks on either go-containerregistry doing something different with its default platform or having ko passing the desired platform along as in the kaniko PR.

vdemeester commented 5 years ago

cc @ImJasonH @mattmoor (for the ko part)

imjasonh commented 5 years ago

https://github.com/google/ko/issues/6 cc @jonjohnsonjr

jonjohnsonjr commented 5 years ago

I added some more details to that bug with what I had in mind.

vincent-pli commented 5 years ago

@vdemeester @ImJasonH What about this project? do we have ppc64 supported?

vdemeester commented 5 years ago

@vincent-pli as of today, we only support linux/amd64. We should aim to support more combinaison of GOOS and GOARCH though.

mattmoor commented 5 years ago

The container tools nerd in me would love to see this happen, but I'm concerned about our current testing capability for this matrix.

@jonjohnsonjr can probably guide someone through this (he has an "ugly" PoC), but we probably won't be able to prioritize this before Knative Serving "v1"s.

jonjohnsonjr commented 5 years ago

Published what I was playing with here: https://github.com/google/ko/pull/38

If anyone's interested in picking this up, I'm happy to chat.

vincent-pli commented 5 years ago

@lisa I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

lisa commented 5 years ago

@lisa I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

@vincent-pli Yes, it worked. The build process (./hack/release.sh) built a controller image which I'm storing as thedoh/arm64-tektoncd-pipeline-cmd-controller.

skaegi commented 4 years ago

I know there is work in-house on doing a Tekton for s390x. Is the idea here that our build/release process would also have images for these other architectures (oops... nevermind I see one image to rule them all above)? What are we looking for here then? Do we want a tick mark when we can confirm Tekton releases works on a particular architecture?

apasrini commented 4 years ago

@skaegi We are right now trying Tekton for s390x. Is it supported or it is yet to be complete? Or is it available in a branch so that we could try?

vdemeester commented 4 years ago

@apasrini It's not "officially" supported, as "we do not ship s390x images" but it should be possible. We (Red Hat & IBM) are working on having OpenShift Pipelines supporting s390x. In parallel we should work upstream to make sure it supports it to (cc @barthy1). To be able to build and release s390x (and other platforms) for tekton, we depend on https://github.com/google/ko/pull/38 — or we should modify our release process to be able to not use ko for other architectures in the meantime.

@afrittoli @ImJasonH @sbwsg @dibyom we also could try to use a patch version of ko (based on https://github.com/google/ko/pull/38) to try this out (and gather feedback for ko so that it gets in at some point).

barthy1 commented 4 years ago

@vdemeester I've played a little bit with the latest ko version (without taking into account https://github.com/google/ko/pull/38) A little bit strange behaviour takes place, but there is some dirty way to use ko for building s390x even now. So what I can see: 1) s390x host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> s390x image and "Architecture": "s390x" in manifest 2) s390x host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> s390x image and "Architecture": "amd64" in manifest 3) amd64 host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> amd64 image and "Architecture": "s390x" in manifest 4) amd64 host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> amd64 image and "Architecture": "amd64" in manifest

Result: First option provides a way to have s390x image build with current ko, but it requires to have s390x hardware

mattmoor commented 4 years ago

I'd love to get Jon's PR moving forwards, and we were just talking last week about some thoughts here. Let's discuss there?

easmithdev commented 4 years ago

Is there any update surrounding ARM64 support for the tekton controller and webhook images, as well as pipeline workloads? ARM64 should be 1st class citizen in regard to architecture support.

jonjohnsonjr commented 3 years ago

https://github.com/google/ko/pull/38 has landed which should unblock this, if anyone wants to jump on it.

Also happy to hear any feedback if that's not sufficient. I still need to write docs and stuff but it's pretty straightforward to play around with:

  1. Set a multi-platform image as the base image.
  2. Use --platform=all
docularxu commented 3 years ago

I did a native Arm64 build of teckton/pipeline on Arm64 servers. Also, I had docker images built, everything is run and tested on Arm64. Is there anyway I can contribute my work to you?

The description of my work can be found here: Build tekton-v0.15.2 on Arm64.md

For the tests, I run a lighthouse+tekton pipeline.

tekton-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

vdemeester commented 3 years ago

/remove-lifecycle stale /lifecycle frozen /area roadmap

This is part of a long-term plan to support multiple architecture, and work is already in progress on the topic

julweber commented 3 years ago

+1, see https://github.com/tektoncd/pipeline/issues/4202 also

ankuo commented 2 years ago

Is there any update surrounding ARM64 support for the tekton controller and webhook images, as well as pipeline workloads? Whether a related image is provided

mattmoor commented 2 years ago

arm64 should work just fine. I generally build my own distro downstream, but I have been running on RPi4 and Graviton clusters without any known issues with Tekton. I'd open issues for anything you see.

mattmoor commented 2 years ago

(that said, I'm pretty sure upstream ships at least amd64/arm64, but likely also s390x/ppc64le/arm)

ankuo commented 2 years ago

@mattmoor Is there an official mirror for arm64 version available

mattmoor commented 2 years ago

All of the images should be multi-arch, from latest release:

# crane manifest gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.33.1@sha256:590777b63ef2ab6027cb819c5f81f9bd32f1e38ca2e8a5da8115e0603db96f85 | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:56d4c294506151f6447eedd18ece7d75942e5a7ef0a37bfba0bd2d3f31d50a7b",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:e9eb0e795b17070e3b8d9290b3c0543a98cf510f6b4cdc723eca79e7ed2f4877",
      "platform": {
        "architecture": "arm",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:cc6e03bd33208231c241345d7b6c4c681027f53fd0cb0a604d232e8f23d114e1",
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:f061293cdc773c8cd41e5e031e58f7fde2269244626ac8bed864e7566b8005a1",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:c625d63d1893128593d0ed1e024ffee3ae5f964a1d69a4501e4988ad272e9686",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      }
    }
  ]
}
ankuo commented 2 years ago

@mattmoor Thanks for getting back to me so quickly. Forgive me. I don't know how to get the download link corresponding to these images, and how to find the download link corresponding to the historical version

mattmoor commented 2 years ago

The images referenced from release.yaml here are all multi-architecture images, which means your cluster will download the appropriate images based on its architecture.

In short: the amd64 yaml is the arm64 yaml since the images all support both architectures.

Hopefully that clarifies?

ankuo commented 2 years ago

@mattmoor This is so cool！ I gave it a try and it worked. Thank you again. Thank you.

afrittoli commented 1 year ago

We do have nightly tests for ppc64 and s390x as we have plans to add PR testing as well. We currently have no infrastructure available to test amd64 and windows.

vdemeester commented 1 year ago

We do have nightly tests for ppc64 and s390x as we have plans to add PR testing as well. We currently have no infrastructure available to test amd64 and windows.

@afrittoli You mean aarch64 or arm64 (amd64 being our "default") 😝

imjasonh commented 1 year ago

We currently have no infrastructure available to test amd64 and windows.

Both linux/arm64 and windows/amd64 are available on GKE, if an enterprising someone felt like making those nodes available in a testing or dogfood cluster. 😄

jerop commented 1 year ago

@vdemeester now that we've added support for windows, can we consider this issue completed? (triaging old issues)

vdemeester commented 1 year ago

Probably, although " bonus : e2e test on cluster of these architectures" is not fixed, meaning, we know it compiles for all those architectures, but we do not really know if they actually works (by runinng e2e tests on them).