KUSTOMIZE_VERSION ?= v3.8.7 on bump-kb.yaml breaking the "Install the CRDs into the cluster" step on the kubebuilder Quick Start

[tavaresrodrigo]

Bug Report

What did you do?

I'm following the https://book.kubebuilder.io/quick-start.html and during the "Install the CRDs into the cluster:" step, make install I experience the error below:

$ make install         
~/projects/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 ~/projects/guestbook/bin
Version v3.8.7 does not exist.
make: *** [~/projects/guestbook/bin/kustomize] Error 1

What did you expect to see?

I expected that the "make install" command would execute successfully allowing me to proceed to the run stage.

What did you see instead? Under which circumstances?

$ make install         
~/projects/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 ~/projects/guestbook/bin
Version v3.8.7 does not exist.
make: *** [~/projects/guestbook/bin/kustomize] Error 1

Environment

Operator type:

/language go

Kubernetes cluster type:

Minikube

$ operator-sdk version operator-sdk version: "v1.20.1", commit: "1780d438cfd87382d034c72703a80d9073b7b6d8", kubernetes version: "v1.23", go version: "go1.17.10", GOOS: "darwin", GOARCH: "arm64"

`$ go version go1.18.2 darwin/arm64``

$ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:36:49Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/arm64"}

Troubleshooting details

Having a look at the bumb-kb.yaml I could see the KUSTOMIZE_VERSION ?= v3.8.7.

Investigating the install_kustomize.sh I noticed the release_url which is fetching the kustomize releases, however the oldest version the GitHub api is returning is v4.4.0.

Possible Solution

If I manually set the v4.5.5(latest) or v4.4.0(oldest) in the bump-kb.yaml file:

KUSTOMIZE_VERSION ?= v4.4.0

I'm able to "make install" and "make run" the controller.

I'm not quite sure what is the strategy to maintain the versions, if we should get the latest version or the oldest one in this case, but I would be happy to contribute with my first PR if you folks decide this is a decent workaround.

I'm also seeking to get engaged and contribute so any help or guidance to discuss an even better fix will be appreciated.

[camilamacedo86]

Hi @tavaresrodrigo,

GOOS: "darwin", GOARCH: "arm64" is not supported by Kubebuilder or SDK. You might want to look at the issue raised: https://github.com/kubernetes-sigs/kubebuilder/issues/1932

Regards the kustomize dep only, you can work around it by using the v4; however, from v3 and v4, it has breaking changes, and we cannot ensure that it works well with the default stable scaffolds (such as go/v3).

In Kubebuilder (master) branch, we added a new kustomize/v2-alpha plugin that uses the v4; see: https://github.com/kubernetes-sigs/kubebuilder/pull/2583. From master in Kubebuilder you are able to now:

kubebuilder init --plugins=kustomize/v2-alpha,base.go.kubebuilder.io/v3

We will probably also provide as follow up a new golang plugin like go/v4-alpha, which will allow users to get the same result by running kubebuilder/operator-sdk init --plugins=go/v4-alpha.

However, note that in order to support darwin/arm64 fully we need more changes, and it is not only about the kustomize version, see: https://github.com/kubernetes-sigs/kubebuilder/issues/1932

c/c @jmrodri @rashmigottipati @ryantking @everettraven @asmacdo @fabianvf

[tavaresrodrigo]

Hi @camilamacedo86, thank you 😃. I've tried on a t3.small and everything worked as expected. Do you think it might be worth handling this error? I think it would be a better user experience if the error was more precise, clearly describing the issue.

So instead of:

"Version v3.8.7 does not exist."

maybe:

 "GOOS: "darwin", GOARCH: "arm64" is not supported by Kubebuilder or SDK" (as you said.)

Thank you for the collaboration, and sorry if my suggestion makes no sense for the project, I'm just getting started in the Framework and there is still a lot to learn, let me know if I can contribute in case you decide this is interesting for the project though, I'm trying to learn more about operators and I think contributing is the best way to do that.

[camilamacedo86]

Hi @tavaresrodrigo,

See: https://github.com/kubernetes-sigs/kubebuilder/pull/2698 (This PR address your suggestion)

c/c @rashmigottipati @ryantking @everettraven ^ looking for PTAL.

[tavaresrodrigo]

This is great! Thank you @camilamacedo86 🙏 .

[everettraven]

closing this issue since @camilamacedo86 had the upstream PR merged.

[everettraven]

@tavaresrodrigo Would you be interested in helping test a change upstream in Kubebuilder that tries to make modifications to the scaffolded Makefile so that M1 Mac users can use Kustomize v3.8.7 by building it locally instead of attempting to pull a prebuilt release?

If you are interested, there is an upstream PR to address it here: https://github.com/kubernetes-sigs/kubebuilder/pull/2706

[algogrit]

Hi @tavaresrodrigo,

GOOS: "darwin", GOARCH: "arm64" is not supported by Kubebuilder or SDK. You might want to look at the issue raised: kubernetes-sigs/kubebuilder#1932

Regards the kustomize dep only, you can work around it by using the v4; however, from v3 and v4, it has breaking changes, and we cannot ensure that it works well with the default stable scaffolds (such as go/v3).

In Kubebuilder (master) branch, we added a new kustomize/v2-alpha plugin that uses the v4; see: kubernetes-sigs/kubebuilder#2583. From master in Kubebuilder you are able to now:

kubebuilder init --plugins=kustomize/v2-alpha,base.go.kubebuilder.io/v3

We will probably also provide as follow up a new golang plugin like go/v4-alpha, which will allow users to get the same result by running kubebuilder/operator-sdk init --plugins=go/v4-alpha.

However, note that in order to support darwin/arm64 fully we need more changes, and it is not only about the kustomize version, see: kubernetes-sigs/kubebuilder#1932

c/c @jmrodri @rashmigottipati @ryantking @everettraven @asmacdo @fabianvf

This should be mentioned in the kubebuilder book as well.

[algogrit]

By the way, kustomize binaries for amd64 work well even on Apple silicon cause of Rosetta Stone 2.

In the interim, just changing the install script to use amd64 binaries even on darwin/arm64 should work.

Sent from a pocket screen

On 29-Jul-2022, at 01:49, Camila Macedo @.***> wrote:

 Hi @algogrit,

I think we could create an FAQ section for the SDK side. But we might before do the following:

Note that we also need to release kubebuilder (which will happen very soon), we have the go/v4-alpha plugin that provides the same result of --plugins=kustomize/v2-alpha,base.go.kubebuilder.io/v3) to use Kustomize v4 and support Apple Silicom.

Then, we will bump Kubebuilder into the SDK side and begin to provide it. However, the target make bundle for Golang is not working with, see #5898. Therefore we will need to check it for Ansible and Helm as well.

Because of this, it is hard to doc now. But I agree. We will need to provide info about it.

c/c @everettraven @Jesus

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.