Kubebuilder is not listed in prerequisites

deepak1725 commented 4 years ago

Bug Report

Kubebuilder is Tightly coupled in requirements, which is fine but...

kubebuilder is not listed in Prerequisites
make docker-build ... expects kubebuilder in usr/local/kubebuilder only otherwise it raises an error, which I don't think is a good idea. User may have kubebuilder anywhere in PATH.

What did you do? make docker-build $IMG

What did you expect to see? resolves kubebuilder from $PATH not from any Specific Location.

What did you see instead? Under which circumstances?

Failure [0.002 seconds]
[BeforeSuite] BeforeSuite 
/home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:52

  Unexpected error:
      <*fmt.wrapError | 0xc00046c2a0>: {
          msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory",
          err: {
              Op: "fork/exec",
              Path: "/usr/local/kubebuilder/bin/etcd",
              Err: 0x2,
          },
      }
      failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory
  occurred

  /home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:62
------------------------------

Ran 0 of 0 Specs in 0.002 seconds
FAIL! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (0.00s)
FAIL
coverage: 0.0% of statements
FAIL    github.com/deepak1725/crda-operator/controllers 0.010s
FAIL
make: *** [Makefile:30: test] Error 1

Environment

operator-sdk version: 0.19
go version: 0.14
Kubernetes version information: 1.18
Kubernetes cluster kind: minikube
Are you writing your operator in ansible, helm, or go? go

Possible Solution

Additional context Add any other context about the problem here.

camilamacedo86 commented 4 years ago

Hi @deepak1725,

I am unable to reproduce this issue. Kubebuilder should not be a pre-requirement and shows that your project is asking its binary to setup the Env Test.

Could you please let us know the steps performed to generate this issue? Could you please let me know if you have kubebuilder installed when you create the project?

deepak1725 commented 4 years ago

@camilamacedo86 My System didn't had kubebuilder

Steps:

I tried make docker-build $IMG
Got Error: As Specified above.
then I installed it in the separate Path than specified in Official Kubebuilder docs. ie. I installed in /usr/local/bin/kubebuilder instead of /usr/local/kubebuilder/
Exported Kubebuilder and it is recognized from every end of PC.
I tried again make docker-build $IMG, but the error was still not resolved.
Finally I have moved binaries from my custom path to officially specified Kubebuilder path, which finally Resolves Error.
Conclusion acc. to me: SDK has Hard-Coded Kubebuilder path which relies solely on Official specified Path.

camilamacedo86 commented 4 years ago

Hi @deepak1725,

Really thank you for raise that. As you know, SDK is in a process to be integrated with KB which means that its project layouts will be aligned. More info : Integrating Kubebuilder and Operator SDK. But, IMHO kubebuilder cannot be set as a pre-requirement because the kubebuilder version used by SDK as dependency can be diff of the version installed.

The problem faced by you is because Env Test requires the apiserver, kubectl and etcd binaries to run and these are shipped with kubebuilder binary. You can customize its set up to use them from another location as well. To understand it further as to see how you can change its conjugation see: https://github.com/kubernetes-sigs/kubebuilder/issues/686#issuecomment-647376924.

We will discuss this scenario in the bug triage meeting on Monday. Feel free to attend if you wish.

camilamacedo86 commented 4 years ago

PR to fix it for v3 in Kubebuilder; https://github.com/kubernetes-sigs/kubebuilder/pull/1600

camilamacedo86 commented 4 years ago

Hi @deepak1725,

See that we added a script to upstream to do this setup and we have a pr against master https://github.com/operator-framework/operator-sdk/pull/3510 and against 0.19.x https://github.com/operator-framework/operator-sdk/pull/3517 to update the docs.

camilamacedo86 commented 4 years ago

All done. We can improve the docs in the future as well. Really tks for raise that.

deepak1725 commented 4 years ago

@camilamacedo86 are we going to cherry-pick this in 0.19 later..?

camilamacedo86 commented 4 years ago

It is applied to the 0.19 already. See; https://v0-19-x.sdk.operatorframework.io/docs/golang/references/env-test-setup/ and the PR: https://github.com/operator-framework/operator-sdk/pull/3517

camilamacedo86 commented 4 years ago

We are re-open this one. After the kubebuilder/controller-runtime meeting was decided that we should not re-build the bins and instead of that get them from https://storage.googleapis.com/kubebuilder-tools. @joelanford has been working in an script against controller-runtime: https://github.com/kubernetes-sigs/controller-runtime/pull/1092

And then, we imagine as a final solution something such as:

setup_envtest.sh:

ENVTEST_ASSETS_DIR=$(pwd)/test/assets
fetch_envtest_assets() {
    OS=$(uname -s | tr '[:upper:]' '[:lower:]')
    ARCH=$(uname -m | sed 's/x86_64/amd64/')
    mkdir -p $ENVTEST_ASSETS_DIR
    curl -fsL "https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-1.16.4-${OS}-${ARCH}.tar.gz" | tar zx --strip-components=1 -C $ENVTEST_ASSETS_DIR
}
setup_envtest_env() {
    export TEST_ASSET_KUBE_APISERVER=$ENVTEST_ASSETS_DIR/bin/kube-apiserver
    export TEST_ASSET_ETCD=$ENVTEST_ASSETS_DIR/bin/etcd
    export TEST_ASSET_KUBECTL=$ENVTEST_ASSETS_DIR/bin/kubectl
}

Then in a makefile (or wherever) source setup_envtest.sh; fetch_envtest_assets; setup_envtest_env

Also, in the PRs https://github.com/operator-framework/operator-sdk/pull/3551 and https://github.com/operator-framework/operator-sdk/pull/3543 we are marking the Env Test doc as TBD as the step to setup it until we have a better solution.

estroz commented 4 years ago

This shouldn't block v1.0 since the steps to get envtest tools are encoded in a script and documented. Moving this to v1.1.0.

jimdickinson commented 3 years ago

I just hit this on 1.0, and I don't see this explained in the docs. Is this the right page? https://sdk.operatorframework.io/docs/building-operators/golang/references/envtest-setup/

Might want to weave a link to that into the tutorial, although it's quite possible I missed it.

camilamacedo86 commented 3 years ago

Hi @jimdickinson,

In the quickstart, you will see that a step was added to solve this problem for now. See: https://sdk.operatorframework.io/docs/building-operators/golang/quickstart/#configuring-your-test-environment

zonnie commented 3 years ago

@camilamacedo86 - the link you sent leads to the Quickstart page but the section is missing. I tried to look for the workaround anyway but was unable to find it...any quick way to avoid this issue ?

camilamacedo86 commented 3 years ago

The above link shows the latest version (1.1). For this version, the setup is no longer required. See that your project is built with the target as: https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/memcached-operator/Makefile#L28-L33.

However, if you see any problem could you please raise a new issue.

kansberry commented 3 years ago

I am using operator-sdk version 1.3.0 and get this error message: Unexpected error: <*fmt.wrapError | 0xc00045cd60>: { msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory", err: { Op: "fork/exec", Path: "/usr/local/kubebuilder/bin/etcd", Err: 0x2, }, } failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory occurred

and here are the lines in the Makefile that seem to be discussed above: ENVTEST_ASSETS_DIR=$(shell pwd)/testbin test: generate fmt vet manifests mkdir -p ${ENVTEST_ASSETS_DIR} test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

Any suggestion as to what I am doing wrong would be appreciated.

estroz commented 3 years ago

@kansberry make test is giving you that error? If you add

SHELL = /bin/bash

to your Makefile, what happens?

kansberry commented 3 years ago

No, the command I am running is "make docker-build IMG=registry.hub.docker.com/$USERNAME/memcached-operator:v0.0.1". I also created an environment variable of SHELL=/bin/bash, as suggested, but this didn't fix the issue. I tried falling back to 1.2.0, and I get the same error.

CORRECTION. I missed the part about adding the value to the Makefile. I just did that, and it worked.

estroz commented 3 years ago

~@kansberry can you file a new issue with a link to/paste of your Makefile.~ Just saw your edit above, this is a known problem upstream and is being worked on. See #4203

Phiph commented 3 years ago

@kansberry make test is giving you that error? If you add
SHELL = /bin/bash
to your Makefile, what happens?

Thanks Estroz. I was experiencing the same problem as the OP. Doing your suggestion solved the problem!

Environment - Kali linux WSL2 instance.

techjacker commented 3 years ago

I'm still experiencing this issue

ghost commented 3 years ago

OS: WSL2

Still does not work!

Commit - https://github.com/evergreen-trading-systems/yta-operator/commit/64a13f71d2b05bc45e4285e6942e3249af85790a#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52

StephanSalas commented 3 years ago

Yep, I think this is a user-experience issue. If I have to do some research to rename a path for kubebuilder manually through some riga-maru, I don't think most users will find this edge case very friendly. I'd suggest an automated solution to prevent this error.

dh-ironsec commented 3 years ago

These helped me:


USE_EXISTING_CLUSTER (boolean): if set to true, envtest will use an existing cluster
TEST_ASSET_KUBE_APISERVER (string): path to the api-server binary to use
TEST_ASSET_ETCD (string): path to the etcd binary to use
TEST_ASSET_KUBECTL (string): path to the kubectl binary to use
KUBEBUILDER_ASSETS (string): directory containing the binaries to use (api-server, etcd and kubectl). Defaults to /usr/local/kubebuilder/bin.
KUBEBUILDER_CONTROLPLANE_START_TIMEOUT (string supported by time.ParseDuration): timeout for test control plane to start. Defaults to 20s.
KUBEBUILDER_CONTROLPLANE_STOP_TIMEOUT (string supported by time.ParseDuration): timeout for test control plane to start. Defaults to 20s.
KUBEBUILDER_ATTACH_CONTROL_PLANE_OUTPUT (boolean): if set to true, the control plane's stdout and stderr are attached to os.Stdout and os.Stderr

operator-framework / operator-sdk

Kubebuilder is not listed in prerequisites #3461

Bug Report