search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.29k stars 1.54k forks source link

Support arm64 #166

Closed vielmetti closed 3 years ago

vielmetti commented 5 years ago

Device under test is a Packet c1.large.arm 96-core arm64 machine running Ubuntu 18.04.

ed@ed-2a-bcc-llvm:~$ go version
go version go1.11.2 linux/arm64
ed@ed-2a-bcc-llvm:~$ go get sigs.k8s.io/kind
ed@ed-2a-bcc-llvm:~$ go/bin/kind create cluster
Creating cluster 'kind-1' ...
 ✓ Ensuring node image (kindest/node:v1.12.2)  
 ✓ [kind-1-control-plane] Creating node container 📦 
 ✗ [kind-1-control-plane] Fixing mounts 🗻 
Error: failed to create cluster: exit status 1
Usage:  
  kind create cluster [flags]

Flags:  
      --config string   path to a kind config file
  -h, --help            help for cluster
      --image string    node docker image to use for booting the cluster
      --name string     cluster context name (default "1")
      --retain          retain nodes for debugging when cluster creation fails
      --wait duration   Wait for control plane node to be ready (default 0s)

Global Flags:
      --loglevel string   logrus log level [panic, fatal, error, warning, info, debug] (default "warning")

failed to create cluster: exit status 1
ed@ed-2a-bcc-llvm:~$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:52:41 2018
 OS/Arch:           linux/arm64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:17:01 2018
  OS/Arch:          linux/arm64
  Experimental:     false
vielmetti commented 5 years ago

Looking at https://github.com/kubernetes-sigs/kind/blob/master/pkg/cluster/context.go#L196-L206 where the code errors out, there's a TODO for logging from @BenTheElder .

vielmetti commented 5 years ago

Looks like the underlying reason is that kindest/node is not multiarchitecture.

ed@ed-2a-bcc-llvm:~$ docker run --rm mplatform/mquery kindest/node:v1.12.2
Image: kindest/node:v1.12.2
 * Manifest List: No
 * Supports: amd64/linux
tao12345666333 commented 5 years ago

Yes, now we only hava amd64 architecture image. https://github.com/kubernetes-sigs/kind/blob/master/images/base/Dockerfile#L28-L33

neolit123 commented 5 years ago

/priority important-longterm /kind feature

BenTheElder commented 5 years ago

@dims hacked up a working version of this: https://paste.fedoraproject.org/paste/gdlF9fqXeSADK-aPN-sEbw/raw :tada: I think we might want to put a little more thought into how to do this well like #188, but this should be doable. Tentatively tracking this for kind 1.0 in Q1 2019.

pbnj commented 5 years ago

We're yet to see details around this announcement, but it ought to simplify the process of building ARM images significantly: https://techcrunch.com/2019/04/24/docker-partners-with-arm/

BenTheElder commented 5 years ago

yes!

also somehow forgot to update this issue, we have arm64 support, just no published images yet (that will need some more thinking...) if you build images yourself kind should work on arm64 today 😅

BenTheElder commented 5 years ago

somewhat, we then either have to publish different images for arm or sort out image manifests and what that pipeline looks like. ideally we don't always require building both but do when we publish pre-built images.

there's some other options where we stop needing RUN outside of the base image that would make this all simpler as well.

On Wed, Apr 24, 2019 at 10:15 PM Peter Benjamin notifications@github.com wrote:

Once we can cross-build images for ARM from x86 with native docker tooling, would that solve the image publishing problem?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kind/issues/166#issuecomment-486522493, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHADKYJ3JR7PJ4SNNFQP5LPSE477ANCNFSM4GJSJUKQ .

vielmetti commented 5 years ago

My fervent hope is that the new Docker tooling will make multi-architecture manifests much easier to produce. The current setup for most projects is just a bit complex.

BenTheElder commented 5 years ago

So FWIW I did figure out how to work manifest-tool I think, this definitely seems feasible this year, if a bit clunky, the trickiest part now is we need to write some tooling to cross compile the kind image (or coordinate with kicking off a build on packet or ... 🤔).

I think I'd like to get this into GCB based publishing with a cross-compile so we can start automating publishing mulit-arch node images, I punted looking further into that while working on the breaking image changes in #461, but those are in now :-)

aojea commented 5 years ago

@BenTheElder we can use CircleCI or Travis to do that, I see that another project under kubernetes-sig has the integrations enabled https://github.com/kubernetes-sigs/kubeadm-dind-cluster

I have experience building pipelines on those, it will be relatively easy to create a pipeline there to automatically publish the images based on PR or per tag

BenTheElder commented 5 years ago

We want them to be pushed based on Kubernetes repo changes instead.

The tricky part will not be setting up Travis etc., We have Prow, GCB, ... but we need the credential to not be available to presubmits, we need the triggering to be right, and we need to build even for PPC64LE ideally, which I doubt Travis etc. are running natively on, so we need to cross compile.

Kubernetes uses some tricks for doing this w/ binfmt_misc and qemu that work very portably.

From:Antonio Ojea notifications@github.com Date:Fri, May 3, 2019, 02:04 To:kubernetes-sigs/kind Cc:Benjamin Elder, Mention

@BenTheElder https://github.com/BenTheElder we can use CircleCI or Travis

to do that, I see that another project under kubernetes-sig has the integrations enabled https://github.com/kubernetes-sigs/kubeadm-dind-cluster

I have experience building pipelines on those, it will be relatively easy to create a pipeline there to automatically publish the images based on PR or per tag

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kind/issues/166#issuecomment-489020422, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHADK7V2OSSMKVES7RIO7LPTP53RANCNFSM4GJSJUKQ .

aojea commented 5 years ago

hmm, I think I have to read more about prow https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md

BenTheElder commented 5 years ago

Minor update: I've ensured the ip-masq-agent pushes multi-arch (manifest) images upstream before we adopted it, and our own tiny networking daemon is cross compiled and pushing multi-arch images.

These were simpler than Kubernetes node images, but at least we have some more multi-arch samples and we continue to nominally work on arm64.

I think the next step is to support building node images from Kubernetes release tarballs so we can consume Kubernetes's upsteam cross compilation output and save time building & publishing. Then we start building manifest list images from those.

vielmetti commented 5 years ago

@BenTheElder Can you take a look and provide current status for this? I'd very much like to identify the list of dependencies to get this work complete, so that we can reasonably assess whether it has any chance to be ready for Kubecon or Rejekts, and if so assess that against all other things.

BenTheElder commented 4 years ago

Hi, sorry I missed this comment.

KIND should work fine in that most of what we publish and all of the dependencies work on arm BUT you must build your own base + node image..

These are somewhat expensive and complex builds and the demand is low so it has not been worth making our build even more complex and slow and reducing our velocity at the moment.

Interested parties can build their own, which is similarly required if you want completely arbitrary Kubernetes versions at the moment.

We do prebuild a handful of node images at each release (one for each lastest patch version of 1.11.X forward) but we aren't doing broad pre-building yet.

CI is problematic, the Kubernetes project's CI does not run natively on ARM and I personally do not have time to maintain anything further, I don't think out other regular contributors do either, and as the Kubernetes infra work group is not even close to having the community run CI etc....

There was contributed CI set up previously, but it has not been kept maintained so we can't rely on the signal. https://testgrid.k8s.io/conformance-all#kind,%20v1.14%20(dev,%20ARM64)

edmondsw commented 4 years ago

I'm very interested in this for ppc64le and have colleagues who want to use this for s390x. Based on the latest comment here, we'll see if we can build the images ourselves, but it would be awesome if we could just pull them from docker hub in the future, like you can for amd64.

@BenTheElder earlier you wrote:

we need to build even for PPC64LE ideally, which I doubt Travis etc. are running natively on, so we need to cross compile.

so I just wanted to let you know... Travis does actually support native builds for ppc64le and s390x now. Here's the announcement from last year: https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z

BenTheElder commented 4 years ago

unfortunately currently kubernetes CI is not on something like travis and there is significant friction to using it: https://github.com/kubernetes/test-infra/tree/master/prow

right now we're not pushing a lot of images to docker hub cross build or otherwise because we still have to round out some things with testing kubernetes itself for which this is not related, but I'd be willing to look at starting with the base image at least for cross builds. we need to build that infrequently and the build is not super expensive.

we do cross build for ~all architectures oureselves already:

node image build is more problematic.

BenTheElder commented 4 years ago

tenative exploration of base image cross build. https://github.com/kubernetes-sigs/kind/pull/1366 this is really slow by comparison even on my workstation only adding amd64 and we currently have no way to verify that the images work on those architectures, so I'm not committing to it yet.

zhlhahaha commented 4 years ago

Hi @BenTheElder we have successfully run kind-e2e on ARM, and plane to submit the test result to testgirds. Any other help we can do to?

BenTheElder commented 4 years ago

Hi, I think a great next step would be to look at the kube-cross image, bazel is looking like it will be removed upstream and the cross image is now built with community infra.

zhlhahaha commented 4 years ago

Hi @BenTheElder , I saw there is a PR (link) of building kube-cross image for multi-arch. I will do some research on it.

BenTheElder commented 4 years ago

FYI on https://github.com/kubernetes/kubernetes/issues/88553

On Tue, Mar 3, 2020 at 8:33 PM zhlhahaha notifications@github.com wrote:

Hi @BenTheElder https://github.com/BenTheElder , I saw there is a PR ( link https://github.com/kubernetes/kubernetes/pull/85355) of building kube-cross image for multi-arch. I will do some research on it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kind/issues/166?email_source=notifications&email_token=AAHADK2VJ7M5XOS65CTQSA3RFXK2BA5CNFSM4GJSJUK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENWKASQ#issuecomment-594321482, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHADK55LE5RCBF2DYWZ473RFXK2BANCNFSM4GJSJUKQ .

BenTheElder commented 4 years ago

https://github.com/kubernetes-sigs/kind/pull/1346 fixes image preloading when the image is not properly metada-ed as arm (many cross built images are only correct in the manifest list), we fixed this in the pause image at least.

vielmetti commented 4 years ago

@BenTheElder - you note that "many cross built images are only correct in the manifest list" - can you provide an example of same? I suspect that means figuring out the pattern for the cross-built image creation and fixing that build process many places upstream.

BenTheElder commented 4 years ago

see #1346 discussion / back links, if you do a normal docker build where you do from <some other arch image> w/ quemu, the manifest metadata inherits somewhat from the host instead of the intended arch. buildx doesn't seem to have this issue. the manifest list will point to the right image but that image's manifest will have the wrong arch.

this broke users with bleeding edge cri-o and kubernetes because cri-o started validating more of the metadata

BenTheElder commented 4 years ago

I'm starting to re-evaluate how we can support platforms again, though it is on my backburner. Is there any reason to expect ARM (not ARM64) users? It looks like these day even raspberry pi is arm64?

alexellis commented 4 years ago

The primary OS for Raspberry Pi is 32-bit with a 64-bit beta, if it's trivial to support armhf (32-bit) then you should, and you'll make a lot of users (millions) happy.

If you need any help @BenTheElder then don't hesitate to ask - almost all of the OpenFaaS projects and utilities are multi-arch.

BenTheElder commented 4 years ago

thanks. that's unfortunate re: rpi OS

I have multi-arch builds for most things, but the node images are not normal builds and are a bit expensive atm (building kubernetes...). building for armhf locally even on my large workstation is especially slow for every image so far, we may revisit this in the future when we have less need for kind iteration velocity.

right now upstream CI health is at its worst so we will not be able to put more into this for a bit.

On Thu, Jun 18, 2020 at 6:50 AM Alex Ellis notifications@github.com wrote:

The primary OS for Raspberry Pi is 32-bit with a 64-bit beta, if it's trivial to support armhf (32-bit) then you should, and you'll make a lot of users (millions) happy.

If you need any help @BenTheElder https://github.com/BenTheElder then don't hesitate to ask - almost all of the OpenFaaS projects and utilities are multi-arch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kind/issues/166#issuecomment-646029140, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHADK5AGZNRRY2ERC3VCJ3RXILQ3ANCNFSM4GJSJUKQ .

alexellis commented 4 years ago

Why is it unfortunate that armhf is still in use? You'd be doing the community a service even publishing an ARM64 image.

BenTheElder commented 4 years ago

er, it's unfortunate that arm32 is still in use for the reasons mentioned above (build time).

we're already building the base image etc. for multiple platforms.

the node image is not possible to cross build at the moment (it can't be accomplished with a dockerimage so no throwing buildx at it)

u2bo commented 4 years ago

is the node had already support arm64?

zhlhahaha commented 4 years ago

is the node had already support arm64?

Kind can run on arm64, but you need to build node-image by yourself.

scottmalkie commented 3 years ago

Hello, I am attempting to build my own node-image on an arm64 host. It looks like the kind build node-image starts with the file build/run.sh in the kubernetes source, which starts with FROM k8s.gcr.io/build-image/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG ... which does not provide an arm64 image.

Do I need to actually make the entirety of kubernetes first before attempting to build a node-image?

Any help would be greatly appreciated. Thanks!

Terminal output:

[ssm-user bin]$ arch
aarch64
[ssm-user bin]$ ./kind build node-image
Starting to build Kubernetes
+++ [0914 18:13:40] Verifying Prerequisites....
+++ [0914 18:13:40] Building Docker image kube-build:build-66fac25301-5-v1.15.0-1
+++ Docker build command failed for kube-build:build-66fac25301-5-v1.15.0-1

Sending build context to Docker daemon  8.704kB
Step 1/16 : FROM k8s.gcr.io/build-image/kube-cross:v1.15.0-1
v1.15.0-1: Pulling from build-image/kube-cross
no matching manifest for linux/arm64/v8 in the manifest list entries

To retry manually, run:

docker build -t kube-build:build-66fac25301-5-v1.15.0-1 --pull=false /home/ssm-user/go/src/k8s.io/kubernetes/_output/images/kube-build:build-66fac25301-5-v1.15.0-1

!!! [0914 18:13:40] Call tree:
!!! [0914 18:13:40]  1: build/run.sh:31 kube::build::build_image(...)
Failed to build Kubernetes: failed to build binaries: command "build/run.sh make all 'WHAT=cmd/kubeadm cmd/kubectl cmd/kubelet'" failed with error: exit status 1
ERROR: error building node image: failed to build kubernetes: failed to build binaries: command "build/run.sh make all 'WHAT=cmd/kubeadm cmd/kubectl cmd/kubelet'" failed with error: exit status 1
Command Output: +++ [0914 18:13:40] Verifying Prerequisites....
+++ [0914 18:13:40] Building Docker image kube-build:build-66fac25301-5-v1.15.0-1
+++ Docker build command failed for kube-build:build-66fac25301-5-v1.15.0-1

Sending build context to Docker daemon  8.704kB
Step 1/16 : FROM k8s.gcr.io/build-image/kube-cross:v1.15.0-1
v1.15.0-1: Pulling from build-image/kube-cross
no matching manifest for linux/arm64/v8 in the manifest list entries

To retry manually, run:

docker build -t kube-build:build-66fac25301-5-v1.15.0-1 --pull=false /home/ssm-user/go/src/k8s.io/kubernetes/_output/images/kube-build:build-66fac25301-5-v1.15.0-1

!!! [0914 18:13:40] Call tree:
!!! [0914 18:13:40]  1: build/run.sh:31 kube::build::build_image(...)
BenTheElder commented 3 years ago

Do I need to actually make the entirety of kubernetes first before attempting to build a node-image?

No, we wrap the upstream build inside kind build node-image.

It looks like the kind build node-image starts with the file build/run.sh in the kubernetes source, which starts with FROM k8s.gcr.io/build-image/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG ... which does not provide an arm64 image.

Yes, parts of the upstream build only support building from amd64 currently. There are upstream issues discussing thi.s

Right now you'll have to build using bazel I think. There's an option for that in `kind build node-image.

u2bo commented 3 years ago

@vielmetti @BenTheElder @scottmalkie build in arm64 arch. ''' kind build node-image --image node:v1.18 \ --kube-root /root/go/src/k8s.io/kubernetes-release-1.18 \ --type bazel \ --base-image kindest/base:v20200826-5c3ff118 '''

got this error .How to solve it? thx

''' INFO: Repository debian-iptables-arm64 instantiated at: no stack (--record_rule_instantiation_callstack not enabled) Repository rule container_pull defined at: /root/.cache/bazel/_bazel_root/ebf03545b4e1140a2d71eccbc6690d4b/external/io_bazel_rules_docker/container/pull.bzl:173:33: in INFO: Repository debian-base-arm64 instantiated at: no stack (--record_rule_instantiation_callstack not enabled) Repository rule container_pull defined at: /root/.cache/bazel/_bazel_root/ebf03545b4e1140a2d71eccbc6690d4b/external/io_bazel_rules_docker/container/pull.bzl:173:33: in ERROR: An error occurred during the fetch of repository 'debian-iptables-arm64': Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/ebf03545b4e1140a2d71eccbc6690d4b/external/io_bazel_rules_docker/container/pull.bzl", line 128, column 71, in _impl fail("Pull command failed: %s (%s)" % (result.stderr, " ".join(args))) Error in join: expected string for sequence element 0, got 'path' ERROR: /root/go/src/k8s.io/kubernetes-release-1.18/build/BUILD:61:22: //build:kube-proxy-internal depends on @debian-iptables-arm64//image:image in repository @debian-iptables-arm64 which failed to fetch. no such package '@debian-iptables-arm64//image': expected string for sequence element 0, got 'path' ERROR: Analysis of target '//build:docker-artifacts' failed; build aborted: Analysis failed INFO: Elapsed time: 26.151s INFO: 0 processes. FAILED: Build did NOT complete successfully (117 packages loaded, 1535 targets configured) FAILED: Build did NOT complete successfully (117 packages loaded, 1535 targets configured)

'''

u2bo commented 3 years ago

@scottmalkie build success? Can you share it?

MichaelCade commented 3 years ago

Yes that image build for ARM64 would be really useful if you can share

stevensu1977 commented 3 years ago

There have any official arm64 image ?

tcnghia commented 3 years ago

I believe building using Bazel currently can't work on ARM64 due to https://github.com/bazelbuild/rules_docker/issues/1155

mattmoor commented 3 years ago

🤔 Bazel shipped arm64 support for Envoy, and distroless is building (and testing) containers on arm64 with rules_docker.

mattmoor commented 3 years ago

I see, the problem is container_pull. It looks like binaries were added for arm64, and s390x, but never added to the workspace rules. This should be relatively straightforward to fix. Happy to help review PRs.

tcnghia commented 3 years ago

@BenTheElder is it possible to build node image using the K8s released binaries (https://dl.k8s.io/v1.19.0/kubernetes-node-linux-arm64.tar.gz)?

BenTheElder commented 3 years ago

@tcnghia not currently no. that would be https://github.com/kubernetes-sigs/kind/issues/381 which is pending a fleshed out design. there will also be some drawbacks (e.g. our binaries are not built the same as the default builds, your images will be larger etc.)

BenTheElder commented 3 years ago

There are upstream discussions in https://github.com/kubernetes/kubernetes about building Kubernetes for other platforms w/o cross building from AMD64, please try to participate there if you're working on this aspect.

The upstream build / testing efforts for non-amd64 platforms are still under developed enough that with my Kubernetes hat on (not KIND) I'm inclined to continue suggesting that the project should not ship these binaries anyhow as they are in no way qualified before release.

With my KIND hat on, since they are shipped for now, and downstream of Kubernetes people want to themselves test on these platforms, it makes sense to try to support it in KIND ...

.. but as long as Kubernetes has no release-blocking tests on any non-amd64 platforms many of us are unlikely to be interested in expending a lot of effort on these platforms. Interested vendors have come and gone without working with the release team on this and the KIND maintainers have barely had time to sort out more pressing and obviously useful / supportable concerns.

tcnghia commented 3 years ago

@u2bo @MichaelCade I confirmed that kind build node-image --type bazel will work after bazelbuild/rules_docker#1155 is fixed. This is what I do:

  1. checkout K8s source code at 1.19 or later.
  2. apply these changes https://github.com/tcnghia/kubernetes/compare/master...tcnghia:use-arm64-rules-docker (or you can also create a patched rules_docker yourself based on the change here https://github.com/bazelbuild/rules_docker/compare/v0.14.4...tcnghia:arm64-old).
  3. go to src/go/k8s.io/kubernetes
  4. run kind build node-image --type bazel
  5. tag the newly built kindest/node image and push to your repo.
u2bo commented 3 years ago

@u2bo @MichaelCade I confirmed that kind build node-image --type bazel will work after bazelbuild/rules_docker#1155 is fixed. This is what I do:

  1. checkout K8s source code at 1.19 or later.
  2. apply these changes tcnghia/kubernetes@master...tcnghia:use-arm64-rules-docker (or you can also create a patched rules_docker yourself based on the change here bazelbuild/rules_docker@v0.14.4...tcnghia:arm64-old).
  3. go to src/go/k8s.io/kubernetes
  4. run kind build node-image --type bazel
  5. tag the newly built kindest/node image and push to your repo.

@tcnghia use your config build and get some error. it's sames like kubelet file not output ?

ERROR: /root/go/src/k8s.io/kubernetes/cmd/kubelet/BUILD:10:10: GoLink cmd/kubelet/kubelet_/kubelet failed (Exit 1): builder failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/013dd613323da188b43f955d6f89ac02/sandbox/linux-sandbox/5718/execroot/io_k8s_kubernetes && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=arm64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder '-param=bazel-out/aarch64-fastbuild-ST-f6ff168d88b9/bin/cmd/kubelet/kubelet_/kubelet-0.params' -- -extld /usr/bin/gcc '-buildid=redacted' -extldflags '-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm')
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox builder failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/013dd613323da188b43f955d6f89ac02/sandbox/linux-sandbox/5718/execroot/io_k8s_kubernetes && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=arm64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder '-param=bazel-out/aarch64-fastbuild-ST-f6ff168d88b9/bin/cmd/kubelet/kubelet_/kubelet-0.params' -- -extld /usr/bin/gcc '-buildid=redacted' -extldflags '-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm')
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
link: error running subcommand: signal: killed
INFO: Elapsed time: 64.553s, Critical Path: 61.90s
INFO: 1 process: 1 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
tcnghia commented 3 years ago

@u2bo I created a .travis.yml here to build the node image https://github.com/tcnghia/playground/blob/master/.travis.yml (see successful build here) . If you like you can clone that file and add a step to push the built image to your own repository. Obviously, this isn't officially supported.

@vielmetti I've sent PR https://github.com/bazelbuild/rules_docker/pull/1671 to rules_docker. Once that merges & releases, if you can help bring https://github.com/kubernetes/kubernetes to use the latest release of rules_docker I think we are good to go with building KinD arm64 node images using kind build node-image --type bazel.

u2bo commented 3 years ago

@u2bo I created a .travis.yml here to build the node image https://github.com/tcnghia/playground/blob/master/.travis.yml (see successful build here) . If you like you can clone that file and add a step to push the built image to your own repository. Obviously, this isn't officially supported.

@vielmetti I've sent PR bazelbuild/rules_docker#1671 to rules_docker. Once that merges & releases, if you can help bring https://github.com/kubernetes/kubernetes to use the latest release of rules_docker I think we are good to go with building KinD arm64 node images using kind build node-image --type bazel.

@tcnghia thanks very much. can you share your image?

tcnghia commented 3 years ago

@u2bo docker.io/tcnghia/kindest-arm64-node:1.19.3 should work