search

kubeflow / katib

Automated Machine Learning on Kubernetes
https://www.kubeflow.org/docs/components/katib
Apache License 2.0
1.5k stars 441 forks source link

Failed to start envtest on Mac with macOS 12 Monterey #1858

Closed tenzen-y closed 2 years ago

tenzen-y commented 2 years ago

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] When I run make test, etcd does not start and the test crashes.

{"level":"error","ts":1651553200.312333,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
2022/05/03 13:46:40 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
stacktrace ```shell $ make test go test ./pkg/... ./cmd/... -coverprofile coverage.out # github.com/shirou/gopsutil/cpu ../../../../1.17.8/pkg/mod/github.com/shirou/gopsutil@v2.20.7+incompatible/cpu/cpu_darwin_cgo.go:13:5: warning: 'TARGET_OS_MAC' is not defined, evaluates to 0 [-Wundef-prefix=TARGET_OS_] ? github.com/kubeflow/katib/pkg/apis/controller [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/common [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/common/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/experiments [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/experiments/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/suggestions [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/suggestions/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/trials [no test files] ? github.com/kubeflow/katib/pkg/apis/controller/trials/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/apis/manager/health [no test files] ? github.com/kubeflow/katib/pkg/apis/manager/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/apis/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/cert-generator/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/cert-generator/v1beta1/consts [no test files] ok github.com/kubeflow/katib/pkg/cert-generator/v1beta1/generate 2.975s coverage: 81.6% of statements ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/fake [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/scheme [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/common/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/common/v1beta1/fake [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/experiments/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/experiments/v1beta1/fake [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/suggestions/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/suggestions/v1beta1/fake [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/trials/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/trials/v1beta1/fake [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/experiments [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/experiments/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/internalinterfaces [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/suggestions [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/suggestions/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/trials [no test files] ? github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/trials/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/listers/experiments/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/listers/suggestions/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/client/controller/listers/trials/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/common/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/controller.v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/controller.v1beta1/consts [no test files] {"level":"error","ts":1651552960.309113,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553020.309948,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553080.3106759,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553140.311522,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553200.312333,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} 2022/05/03 13:46:40 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start FAIL github.com/kubeflow/katib/pkg/controller.v1beta1/experiment 301.173s ok github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/manifest 1.034s coverage: 75.9% of statements ? github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/suggestion [no test files] ? github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/util [no test files] {"level":"error","ts":1651552921.0477371,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651552941.048397,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651552961.048811,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651552981.049254,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553001.049622,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} 2022/05/03 13:43:21 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start FAIL github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion 101.779s 2022/05/03 13:43:20 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start FAIL github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/composer 101.519s ok github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestionclient 0.552s coverage: 98.1% of statements {"level":"error","ts":1651552961.4102862,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553021.411232,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553081.411948,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553141.412628,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} {"level":"error","ts":1651553201.413376,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"} 2022/05/03 13:46:41 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start FAIL github.com/kubeflow/katib/pkg/controller.v1beta1/trial 302.123s ? github.com/kubeflow/katib/pkg/controller.v1beta1/trial/managerclient [no test files] ok github.com/kubeflow/katib/pkg/controller.v1beta1/trial/util 1.440s coverage: 32.9% of statements ? github.com/kubeflow/katib/pkg/controller.v1beta1/util [no test files] ? github.com/kubeflow/katib/pkg/db/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/db/v1beta1/common [no test files] ok github.com/kubeflow/katib/pkg/db/v1beta1/mysql 0.504s coverage: 70.4% of statements ? github.com/kubeflow/katib/pkg/metricscollector/v1beta1/common [no test files] ok github.com/kubeflow/katib/pkg/metricscollector/v1beta1/file-metricscollector 2.214s coverage: 54.4% of statements ? github.com/kubeflow/katib/pkg/mock/v1beta1/api [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/db [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/experiment/manifest [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/experiment/suggestion [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/suggestion/suggestionclient [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/trial/managerclient [no test files] ? github.com/kubeflow/katib/pkg/mock/v1beta1/util/katibclient [no test files] ? github.com/kubeflow/katib/pkg/new-ui/v1beta1 [no test files] ok github.com/kubeflow/katib/pkg/suggestion/v1beta1/goptuna 1.019s coverage: 37.8% of statements ? github.com/kubeflow/katib/pkg/ui/v1beta1 [no test files] ok github.com/kubeflow/katib/pkg/util/v1beta1/env 0.217s coverage: 100.0% of statements ? github.com/kubeflow/katib/pkg/util/v1beta1/katibclient [no test files] ok github.com/kubeflow/katib/pkg/util/v1beta1/katibconfig 0.785s coverage: 97.4% of statements ? github.com/kubeflow/katib/pkg/webhook/v1beta1 [no test files] ? github.com/kubeflow/katib/pkg/webhook/v1beta1/common [no test files] ? github.com/kubeflow/katib/pkg/webhook/v1beta1/experiment [no test files] ok github.com/kubeflow/katib/pkg/webhook/v1beta1/experiment/validator 1.474s coverage: 98.6% of statements --- FAIL: TestGetMetricsCollectorArgs (100.00s) inject_webhook_test.go:249: failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start inject_webhook_test.go:255: Unexpected error: <*errors.errorString | 0xc0004d6140>: { s: "must specify Config", } must specify Config occurred --- FAIL: TestGetKatibJob (100.00s) inject_webhook_test.go:696: failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start inject_webhook_test.go:702: Unexpected error: <*errors.errorString | 0xc00024af60>: { s: "must specify Config", } must specify Config occurred FAIL coverage: 32.0% of statements FAIL github.com/kubeflow/katib/pkg/webhook/v1beta1/pod 200.934s ? github.com/kubeflow/katib/cmd/cert-generator/v1beta1 [no test files] ok github.com/kubeflow/katib/cmd/db-manager/v1beta1 1.027s coverage: 36.1% of statements ? github.com/kubeflow/katib/cmd/katib-controller/v1beta1 [no test files] ? github.com/kubeflow/katib/cmd/metricscollector/v1beta1/file-metricscollector [no test files] ? github.com/kubeflow/katib/cmd/new-ui/v1beta1 [no test files] ? github.com/kubeflow/katib/cmd/suggestion/goptuna/v1beta1 [no test files] ? github.com/kubeflow/katib/cmd/ui/v1beta1 [no test files] FAIL make: *** [test] Error 1 ```

According to the Go upstream wiki, macOS Monterey has discontinued support for programs built with versions earlier than Go 1.10.x.

If you reached this page because you saw an error message like the following printed by a Go program running on macOS 12 Monterey:

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x20594e0, 0x21)
  /usr/local/go/src/runtime/panic.go:619 +0x81 fp=0x7ff7bfeff878 sp=0x7ff7bfeff858 pc=0x1029751
runtime.goenvs()
  /usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff8a8 sp=0x7ff7bfeff878 pc=0x10272d3
runtime.schedinit()
  /usr/local/go/src/runtime/proc.go:496 +0xa4 fp=0x7ff7bfeff900 sp=0x7ff7bfeff8a8 pc=0x102c014
runtime.rt0_go(0x7ff7bfeff930, 0x3, 0x7ff7bfeff930, 0x1000000, 0x3, 0x7ff7bfeffab0, 0x7ff7bfeffabf, 0x7ff7bfeffac3, 0x0, > 0x7ff7bfeffacc, ...)
  /usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff908 sp=0x7ff7bfeff900 pc=0x1052c64

then you are running a program built with an old version of Go (Go 1.10 or before). You will need to update your program or rebuild it with a newer version of Go.

I checked the etcd version on Ubuntu machine and found that the kubebuilder v2.3.0 used by Katib comes bundled with etcd built with go 1.10.7, so etcd could not be started.

runtime stack: runtime.throw(0x1bed3fa, 0x21) /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff2d8 sp=0x7ff7bfeff2b8 pc=0x102a871 runtime.goenvs() /usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff308 sp=0x7ff7bfeff2d8 pc=0x10283f3 runtime.schedinit() /usr/local/go/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff370 sp=0x7ff7bfeff308 pc=0x102d166 runtime.rt0_go(0x7ff7bfeff3a0, 0x2, 0x7ff7bfeff3a0, 0x1000000, 0x2, 0x7ff7bfeff5a8, 0x7ff7bfeff5c8, 0x0, 0x7ff7bfeff5d2, 0x7ff7bfeff5e2, ...) /usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff378 sp=0x7ff7bfeff370 pc=0x1056474


- On Ubuntu machine
```shell
$ ./etcd --version
etcd Version: 3.3.11
Git SHA: 2cf9e51d2
Go Version: go1.10.7
Go OS/Arch: linux/amd64

So I downloaded kubebuilder-tools for K8s v1.20.2 from GCS bucket maintained by controller-runtime project, extracted only etcd, and installed it in /usr/local/kubebuilder/bin/etcd, I was able to run make test successfully.

What did you expect to happen: Envtest succeeds.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

tenzen-y commented 2 years ago

We may have to consider upgrading the kubebuilder version from v2.3.0 to v3.x.y.

/cc @kubeflow/wg-automl-leads

tenzen-y commented 2 years ago

Since kubebuilder supports K8s v1.22 in v3.2.0, I will upgrade kubebuilder version to v3.2.0.

/assign

⚠️ Breaking Changes (go/v3) update deps to use k8s 1.22 (https://github.com/kubernetes-sigs/kubebuilder/pull/2340) This is only a breaking change if using v1beta1 CRDs or webhook configs

https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v3.2.0