Error while setting up karmada locally

rgrupesh commented 1 year ago

What happened: error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz

What you expected to happen: Karmada should run locally.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: Installed all the required prerequisites(Go,kubectl,kind)

Environment:

Karmada version:
kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
Others: Ubuntu 22.04 LTS, OS Type = 64 bit, GNOME Version=42.1

`

hack/local-up-karmada.sh Preparing: 'kind' existence check - passed Preparing: 'kubectl' existence check - passed Preparing kindClusterConfig in path: /tmp/tmp.5NwB5q1PkZ Creating cluster karmada-host and the log file is in /tmp/karmada/karmada-host.log Creating cluster member1 and the log file is in /tmp/karmada/member1.log Creating cluster member2 and the log file is in /tmp/karmada/member2.log Creating cluster member3 and the log file is in /tmp/karmada/member3.log make: Entering directory '/home/rgrupesh/nHack/karmada' set -e;\ target=$(echo karmada-aggregated-apiserver);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:36:20Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-aggregated-apiserver !!! Building karmada-aggregated-apiserver for linux/amd64:

CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:36:20Z' -o _output/bin/linux/amd64/karmada-aggregated-apiserver github.com/karmada-io/karmada/cmd/aggregated-apiserver
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
docker build --build-arg BINARY=karmada-aggregated-apiserver --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 6109a580438a Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> fb27e5c451e9 Successfully built fb27e5c451e9 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
set +x set -e;\ target=$(echo karmada-controller-manager);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:39:01Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-controller-manager !!! Building karmada-controller-manager for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:39:01Z' -o _output/bin/linux/amd64/karmada-controller-manager github.com/karmada-io/karmada/cmd/controller-manager ^[[B
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
docker build --build-arg BINARY=karmada-controller-manager --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MBB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 01cc30731c8d Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 2b3311bff1cc Successfully built 2b3311bff1cc Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
set +x set -e;\ target=$(echo karmada-scheduler);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:47:07Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-scheduler !!! Building karmada-scheduler for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:47:07Z' -o _output/bin/linux/amd64/karmada-scheduler github.com/karmada-io/karmada/cmd/scheduler
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest
docker build --build-arg BINARY=karmada-scheduler --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 9b73178625dc Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 07428db96a48 Successfully built 07428db96a48 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest
set +x set -e;\ target=$(echo karmada-descheduler);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:49:15Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-descheduler !!! Building karmada-descheduler for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:49:15Z' -o _output/bin/linux/amd64/karmada-descheduler github.com/karmada-io/karmada/cmd/descheduler
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest
docker build --build-arg BINARY=karmada-descheduler --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> d71eedaa0dd4 Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 045c91d3399f Successfully built 045c91d3399f Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest
set +x set -e;\ target=$(echo karmada-webhook);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:49:47Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-webhook !!! Building karmada-webhook for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:49:47Z' -o _output/bin/linux/amd64/karmada-webhook github.com/karmada-io/karmada/cmd/webhook
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
docker build --build-arg BINARY=karmada-webhook --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> b9d6e450c74a Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 940a54ddb75c Successfully built 940a54ddb75c Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
set +x set -e;\ target=$(echo karmada-agent);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:50:17Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-agent !!! Building karmada-agent for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:50:17Z' -o _output/bin/linux/amd64/karmada-agent github.com/karmada-io/karmada/cmd/agent
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest
docker build --build-arg BINARY=karmada-agent --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 1d017c9c9cc8 Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 6fbaf8991d04 Successfully built 6fbaf8991d04 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest
set +x set -e;\ target=$(echo karmada-scheduler-estimator);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:50:57Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-scheduler-estimator !!! Building karmada-scheduler-estimator for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:50:57Z' -o _output/bin/linux/amd64/karmada-scheduler-estimator github.com/karmada-io/karmada/cmd/scheduler-estimator
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest
docker build --build-arg BINARY=karmada-scheduler-estimator --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> ef3763eb50d3 Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 1f26e7e61209 Successfully built 1f26e7e61209 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest
set +x set -e;\ target=$(echo karmada-interpreter-webhook-example);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:51:21Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-interpreter-webhook-example !!! Building karmada-interpreter-webhook-example for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:51:21Z' -o _output/bin/linux/amd64/karmada-interpreter-webhook-example github.com/karmada-io/karmada/examples/customresourceinterpreter/webhook
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest
docker build --build-arg BINARY=karmada-interpreter-webhook-example --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> e432a60a770d Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 25bb143204b5 Successfully built 25bb143204b5 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest
set +x set -e;\ target=$(echo karmada-search);\ make $target GOOS=linux;\ VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target make[1]: Entering directory '/home/rgrupesh/nHack/karmada' LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:51:47Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-search !!! Building karmada-search for linux/amd64:
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-110-g3039ebbb -X github.com/karmada-io/karmada/pkg/version.gitCommit=3039ebbba31358fc6d8a1a0374cef09dfc17cf5e -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-06-17T03:51:47Z' -o _output/bin/linux/amd64/karmada-search github.com/karmada-io/karmada/cmd/karmada-search
set +x make[1]: Leaving directory '/home/rgrupesh/nHack/karmada' Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest
docker build --build-arg BINARY=karmada-search --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64 Sending build context to Docker daemon 586MB Step 1/4 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 2/4 : ARG BINARY ---> Using cache ---> db94c77a9f44 Step 3/4 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 5c69e6096cae Step 4/4 : COPY ${BINARY} /bin/${BINARY} ---> 824dd933dcbd Successfully built 824dd933dcbd Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest
set +x make: Leaving directory '/home/rgrupesh/nHack/karmada' Waiting for the host clusters to be ready... Waiting for kubeconfig file /home/rgrupesh/.kube/karmada.config and clusters karmada-host to be ready... Context "kind-karmada-host" renamed to "karmada-host". Cluster "kind-karmada-host" set. Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest" with ID "sha256:2b3311bff1cc2d14338e15c52c92abb94bb5079fc53d49c8958f85b09c2a488b" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest" with ID "sha256:07428db96a48fde8fd8021a1e3c15404aba221da5a375f7f9249ba8832d8cd8d" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest" with ID "sha256:045c91d3399f0aa371f304ea71b8e3e775697c464170620cef6a8820c5720716" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest" with ID "sha256:940a54ddb75cbe7d1e1aeaf4b198f997b89c06e8585c810e85f882bd19d7bd2e" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest" with ID "sha256:1f26e7e6120910fe29a2c2967d257f90b4d22a74723cb38ca81f448d32b70048" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest" with ID "sha256:fb27e5c451e9d907829914ad598619ebaa673a562ff92ce0e6c1289e75f10765" not yet present on node "karmada-host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest" with ID "sha256:824dd933dcbd7dc31d51cf9bb3f60b9132baf382b678ca45585af37b7c905b61" not yet present on node "karmada-host-control-plane", loading... go install github.com/cloudflare/cfssl/cmd/...@v1.5.0
github.com/mattn/go-sqlite3

sqlite3-binding.c: In function ‘sqlite3SelectNew’: sqlite3-binding.c:123303:10: warning: function may return address of local variable [-Wreturn-local-addr] 123303 | return pNew; | ^~~~ sqlite3-binding.c:123263:10: note: declared here 123263 | Select standin; | ^~~ ..........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++..+...+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++...+.......+.........+......+.....+...+......+....+.........+..+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++.+...+..........+...+...............+......+........+...+...+....+..+.+..+...+...+.+..............+................+.....+...+...+....+......+...+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++.+......+...+......+......+......+..+...+.........+.+..+.......+.........+.........+...............+..+.........+.......+..+...+.......+..+...+.+...............+.....+.+...+...........+.........+.+.....+.+...+...+..+............+...............+......+.........+....+..+...+.+...+........+.......+...+........+...................+..+.+.....+.........+.+...+...........+.+............+............+......+........+......+................+..+.......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.........+...+.....+......+....+.....+.+..+...+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++..+.+.....+.+............+......+..+.........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++.....+.+..+.......+.....+.........+....+........+...+.+.....+......+.......+..+............................+...+.....+...+.+.....+.+.........+........+......+....+..+...................+.....+.......+...+......+.....+...+.+.....+.+..............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .....+.....+...+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++......+...........+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++..+.........................+........+.............+.....+.......+...+...........+.+.........+.....+...+.+..+...+....+............+.....+......+...................+.....+...+.+..+....+........+......+.......+..+......+.........+..........+.....+......+....+..+.+.....+...............+.........+................+..+.+......+........+............+.+..+.......+...+..+.............+..+....+.....+..........+.........+...........+....+.................+....+.....+.+......+.....+....+...+..+.+.....+.......+...+...+............+..+...+....+...........+..........+..+......+................+.....+....+.....+.+...+..+.............+...+.....+............+.+..+.........+....+..+...+..........+.....+................+......+...+.......................................+.....+.+...........+....+.........+.....+......+...............+...............+.+...+...+........+.+..................+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2022/06/17 09:39:50 [INFO] generate received request 2022/06/17 09:39:50 [INFO] received CSR 2022/06/17 09:39:50 [INFO] generating key: rsa-2048 2022/06/17 09:39:50 [INFO] encoded CSR 2022/06/17 09:39:50 [INFO] signed certificate with serial number 198355104992488371909900420224866949315351921923 2022/06/17 09:39:50 [INFO] generate received request 2022/06/17 09:39:50 [INFO] received CSR 2022/06/17 09:39:50 [INFO] generating key: rsa-2048 2022/06/17 09:39:50 [INFO] encoded CSR 2022/06/17 09:39:50 [INFO] signed certificate with serial number 507646481658017151173849241513702037749951887290 namespace/karmada-system created serviceaccount/karmada-controller-manager created serviceaccount/karmada-scheduler created serviceaccount/karmada-webhook created clusterrole.rbac.authorization.k8s.io/karmada-controller-manager created clusterrolebinding.rbac.authorization.k8s.io/karmada-controller-manager created secret/karmada-cert-secret created secret/kubeconfig created secret/webhook-cert created statefulset.apps/etcd created service/etcd-client created service/etcd created wait the etcd ready... error: no matching resources found kubectl wait --for=condition=Ready --timeout=30s pods -l app=etcd -n karmada-system failed, retrying(1 times) error: timed out waiting for the condition on pods/etcd-0 kubectl wait --for=condition=Ready --timeout=30s pods -l app=etcd -n karmada-system failed, retrying(2 times) pod/etcd-0 condition met

Apply dynamic rendered apiserver service in /tmp/tmp.zdk55t4nNW/karmada-apiserver.yaml. deployment.apps/karmada-apiserver created service/karmada-apiserver created wait the karmada-apiserver ready... error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(1 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(2 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(3 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(4 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(5 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(6 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(7 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(8 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(9 times) error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed, retrying(10 times) kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-apiserver -n karmada-system failed error: timed out waiting for the condition on pods/karmada-apiserver-85c99f5c57-4xqdz kubectl describe info: Name: karmada-apiserver-85c99f5c57-4xqdz Namespace: karmada-system Priority: 2000001000 Priority Class Name: system-node-critical Node: karmada-host-control-plane/172.18.0.3 Start Time: Fri, 17 Jun 2022 09:40:42 +0545 Labels: app=karmada-apiserver pod-template-hash=85c99f5c57 Annotations: <none> Status: Running IP: 172.18.0.3 IPs: IP: 172.18.0.3 Controlled By: ReplicaSet/karmada-apiserver-85c99f5c57 Containers: karmada-apiserver: Container ID: containerd://fff4febe54271eba92ea5224ec1dfd73e3deefacbaee24f632462515a6623b32 Image: k8s.gcr.io/kube-apiserver:v1.22.10 Image ID: k8s.gcr.io/kube-apiserver@sha256:ac68f1c1c7aa78df15703a705daaa7925da9b8840ae69f1426d9c047a5e47c17 Port: <none> Host Port: <none> Command: kube-apiserver --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/server-ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/server-ca.crt --etcd-certfile=/etc/kubernetes/pki/karmada.crt --etcd-keyfile=/etc/kubernetes/pki/karmada.key --etcd-servers=https://etcd-client.karmada-system.svc.cluster.local:2379 --bind-address=0.0.0.0 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/karmada.crt --kubelet-client-key=/etc/kubernetes/pki/karmada.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --disable-admission-plugins=StorageObjectInUseProtection,ServiceAccount --runtime-config= --secure-port=5443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/karmada.key --service-account-signing-key-file=/etc/kubernetes/pki/karmada.key --service-cluster-ip-range=10.96.0.0/12 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --tls-cert-file=/etc/kubernetes/pki/karmada.crt --tls-private-key-file=/etc/kubernetes/pki/karmada.key State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Fri, 17 Jun 2022 09:46:01 +0545 Finished: Fri, 17 Jun 2022 09:46:21 +0545 Ready: False Restart Count: 5 Requests: cpu: 250m Liveness: http-get https://:5443/livez delay=10s timeout=15s period=10s #success=1 #failure=8 Readiness: http-get https://:5443/readyz delay=0s timeout=15s period=1s #success=1 #failure=3 Environment: <none> Mounts: /etc/kubernetes/pki from k8s-certs (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8x7zc (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: k8s-certs: Type: Secret (a volume populated by a Secret) SecretName: karmada-cert-secret Optional: false kube-api-access-8x7zc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: :NoExecute op=Exists Events: Type Reason Age From Message

Normal Scheduled 5m42s Normal Pulling 5m36s Normal Pulled 5m24s Warning Unhealthy 5m2s Warning Unhealthy 5m2s Warning Unhealthy 5m2s Normal Started Normal Created Warning Unhealthy Warning Unhealthy 4m38s Warning Unhealthy 4m38s Warning Unhealthy Warning BackOff Normal Pulled kubectl logs info: Flag --insecure-port I0617 04:01:01.153295 I0617 04:01:01.153550 I0617 04:01:01.540586 I0617 04:01:01.541213 I0617 04:01:01.541221 I0617 04:01:01.541924 I0617 04:01:01.541932 Error: context deadline exceeded

` default-scheduler Successfully assigned karmada-system/karmada-apiserver-85c99f5c57-4xqdz to karmada-host-control-plane kubelet Pulling image "k8s.gcr.io/kube-apiserver:v1.22.10" kubelet Successfully pulled image "k8s.gcr.io/kube-apiserver:v1.22.10" in 12.064760942s kubelet Liveness probe failed: Get "https://172.18.0.3:5443/livez": read tcp 172.18.0.3:45962->172.18.0.3:5443: read: connection reset by peer kubelet Liveness probe failed: Get "https://172.18.0.3:5443/livez": dial tcp 172.18.0.3:5443: connect: connection refused kubelet Readiness probe failed: Get "https://172.18.0.3:5443/readyz": read tcp 172.18.0.3:45960->172.18.0.3:5443: read: connection reset by peer 4m59s (x2 over 5m23s) kubelet Started container karmada-apiserver 4m59s (x2 over 5m23s) kubelet Created container karmada-apiserver 4m38s (x3 over 5m12s) kubelet Readiness probe failed: Get "https://172.18.0.3:5443/readyz": net/http: TLS handshake timeout kubelet Liveness probe failed: Get "https://172.18.0.3:5443/livez": read tcp 172.18.0.3:45982->172.18.0.3:5443: read: connection reset by peer kubelet Readiness probe failed: Get "https://172.18.0.3:5443/readyz": read tcp 172.18.0.3:45984->172.18.0.3:5443: read: connection reset by peer 4m37s (x7 over 5m2s) kubelet Readiness probe failed: Get "https://172.18.0.3:5443/readyz": dial tcp 172.18.0.3:5443: connect: connection refused 4m32s (x2 over 4m36s) kubelet Back-off restarting failed container 24s (x5 over 4m59s) kubelet Container image "k8s.gcr.io/kube-apiserver:v1.22.10" already present on machine has been deprecated, This flag has no effect now and will be removed in v1.24. 1 server.go:553] external host was not specified, using 172.18.0.3 1 server.go:161] Version: v1.22.10 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer 1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. 1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

prodanlabs commented 1 year ago

From the startup log of kube-apiserver, it seems that the server has insufficient memory.

RainbowMango commented 1 year ago

Thanks @prodanlabs, but which logs?

@rgrupesh How much memory do you get on your machine? I guess at least 8G

rgrupesh commented 1 year ago

Yes, 8 gigs.

prodanlabs commented 1 year ago

kubectl logs info:

Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.

I0617 04:01:01.153295 1 server.go:553] external host was not specified, using 172.18.0.3

I0617 04:01:01.153550 1 server.go:161] Version: v1.22.10

I0617 04:01:01.540586 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer

I0617 04:01:01.541213 1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I0617 04:01:01.541221 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

I0617 04:01:01.541924 1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I0617 04:01:01.541932 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

Error: context deadline exceeded

@RainbowMango When kube-apiserver starts, it prints information about connecting etcd. It exits before connecting to etcd, so I suspect that it is a lack of resources.

hi @rgrupesh , Is it possible to change the log level to --v=9, see more detailed logs.

rgrupesh commented 1 year ago

@prodanlabs I increased my swap area from 4 to 8 GBs but in vain. How do I change the log level?

RainbowMango commented 1 year ago

How do I change the log level?

You can edit the karmada-apiserver by command: kubectl edit deployments.apps -n karmada-system karmada-apiserver

set the --v=9 to the .spec.template.spec.containers.command, like:

      - command:
        - kube-apiserver
        - --requestheader-username-headers=X-Remote-User
        - --tls-cert-file=/etc/kubernetes/pki/karmada.crt
        - --tls-private-key-file=/etc/kubernetes/pki/karmada.key
        - --v=9  // add this flag

rgrupesh commented 1 year ago

Screenshot from 2022-06-21 18-06-43

rgrupesh commented 1 year ago

Screenshot from 2022-06-21 17-50-06

prodanlabs commented 1 year ago

The current IP of the server is 172.23.0.2, but it is not in the trusted list of mTLS certificates.

I guess the previous IP address was 172.23.0.3

RainbowMango commented 1 year ago

@prodanlabs Do you know how to fix it? @rgrupesh just run hack/local-up-karmada.sh and didn't set the certificate by himself.

prodanlabs commented 1 year ago

@prodanlabs Do you know how to fix it? @rgrupesh just run hack/local-up-karmada.sh and didn't set the certificate by himself.

I recommend reinstalling

Execute hack/undeploy-karmada.sh to remove

RainbowMango commented 1 year ago

@rgrupesh What's going on now?

rgrupesh commented 1 year ago

@RainbowMango Still stuck with that error. I tried reinstalling.

rgrupesh commented 1 year ago

The current IP of the server is 172.23.0.2, but it is not in the trusted list of mTLS certificates.

I guess the previous IP address was 172.23.0.3

Any solution to this? @RainbowMango @prodanlabs

RainbowMango commented 1 year ago

Sorry, I've no idea. :(

prodanlabs commented 1 year ago

The current IP of the server is 172.23.0.2, but it is not in the trusted list of mTLS certificates. I guess the previous IP address was 172.23.0.3

Any solution to this? @RainbowMango @prodanlabs

Hi @caniszczyk ， You can try adding a virtual IP.

enp0s25 is the name of the network card, execute ip a to see

ip addr del 172.23.0.3/24 dev enp0s25

prodanlabs commented 1 year ago

ip addr del 172.23.0.3/24 dev enp0s25

sorry, its my fault, it should be add

ip addr add 172.23.0.3/24 dev enp0s25

RainbowMango commented 1 year ago

@rgrupesh as we talked at slack, I suggested you set up Karmada on a virtual machine, not on your laptop. How is it going?

rgrupesh commented 1 year ago

@RainbowMango I created a VM on Azure. It is still not working.

RainbowMango commented 1 year ago

Why, are you facing the same problem?

rgrupesh commented 1 year ago

@RainbowMango VM Info:

H/W path      Device       Class          Description
=====================================================
                           system         Virtual Machine (None)
/0                         bus            Virtual Machine
/0/0                       memory         1MiB BIOS
/0/4                       processor      Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
/0/6                       memory         16GiB System Memory
/0/6/0                     memory         1GiB 
/0/6/1                     memory         15GiB 
/0/2          enP64564s1   network        MT27710 Family [ConnectX-4 Lx Virtual Function]
/0/1                       communication  PnP device PNP0501
/0/3                       communication  PnP device PNP0501
/0/5                       system         PnP device PNP0b00
/0/7          scsi0        storage        
/0/7/0.0.0    /dev/sda     disk           32GB Virtual Disk
/0/7/0.0.0/1  /dev/sda1    volume         29GiB EXT4 volume
/0/7/0.0.0/e  /dev/sda14   volume         4095KiB BIOS Boot partition
/0/7/0.0.0/f  /dev/sda15   volume         105MiB Windows FAT volume
/0/7/0.0.1    /dev/sdb     disk           34GB Virtual Disk
/0/7/0.0.1/1  /dev/sdb1    volume         31GiB EXT4 volume
/1            eth0         network        Ethernet interface
/2            veth85bd3be  network        Ethernet interface
/3            veth7267d66  network        Ethernet interface
/4            veth16074fb  network        Ethernet interface
/5            veth3b9abcd  network        Ethernet interface

Log:

Preparing: 'kind' existence check - not pass
go install sigs.k8s.io/kind@v0.14.0
Preparing: 'kubectl' existence check - passed
Preparing kindClusterConfig in path: /tmp/tmp.iw4k6qzGhy
Creating cluster karmada-host and the log file is in /tmp/karmada/karmada-host.log
Creating cluster member1 and the log file is in /tmp/karmada/member1.log
Creating cluster member2 and the log file is in /tmp/karmada/member2.log
Creating cluster member3 and the log file is in /tmp/karmada/member3.log
make: Entering directory '/home/rgrupesh/karmada'
set -e;\
target=$(echo karmada-aggregated-apiserver);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:49:50Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-aggregated-apiserver
!!! Building karmada-aggregated-apiserver for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:49:50Z' -o _output/bin/linux/amd64/karmada-aggregated-apiserver github.com/karmada-io/karmada/cmd/aggregated-apiserver
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
+ docker build --build-arg BINARY=karmada-aggregated-apiserver --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> ceb9758d797f
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 9f0591a94aa0
Successfully built 9f0591a94aa0
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
+ set +x
set -e;\
target=$(echo karmada-controller-manager);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:50:22Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-controller-manager
!!! Building karmada-controller-manager for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:50:22Z' -o _output/bin/linux/amd64/karmada-controller-manager github.com/karmada-io/karmada/cmd/controller-manager
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
+ docker build --build-arg BINARY=karmada-controller-manager --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> 4f6ea70bf6b6
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> df2070127c65
Successfully built df2070127c65
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
+ set +x
set -e;\
target=$(echo karmada-scheduler);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:50:46Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-scheduler
!!! Building karmada-scheduler for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:50:46Z' -o _output/bin/linux/amd64/karmada-scheduler github.com/karmada-io/karmada/cmd/scheduler
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest
+ docker build --build-arg BINARY=karmada-scheduler --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> 582e9474a2c7
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 4c26270c617b
Successfully built 4c26270c617b
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest
+ set +x
set -e;\
target=$(echo karmada-descheduler);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:51:14Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-descheduler
!!! Building karmada-descheduler for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:51:14Z' -o _output/bin/linux/amd64/karmada-descheduler github.com/karmada-io/karmada/cmd/descheduler
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest
+ docker build --build-arg BINARY=karmada-descheduler --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> f558103714a1
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 22d8ce0f60fd
Successfully built 22d8ce0f60fd
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest
+ set +x
set -e;\
target=$(echo karmada-webhook);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:51:40Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-webhook
!!! Building karmada-webhook for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:51:40Z' -o _output/bin/linux/amd64/karmada-webhook github.com/karmada-io/karmada/cmd/webhook
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
+ docker build --build-arg BINARY=karmada-webhook --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> 8a1ac46b4a53
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 9bd382363cd7
Successfully built 9bd382363cd7
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
+ set +x
set -e;\
target=$(echo karmada-agent);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:00Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-agent
!!! Building karmada-agent for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:00Z' -o _output/bin/linux/amd64/karmada-agent github.com/karmada-io/karmada/cmd/agent
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest
+ docker build --build-arg BINARY=karmada-agent --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> 9672e4ad996b
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 95787cc139aa
Successfully built 95787cc139aa
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest
+ set +x
set -e;\
target=$(echo karmada-scheduler-estimator);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:16Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-scheduler-estimator
!!! Building karmada-scheduler-estimator for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:16Z' -o _output/bin/linux/amd64/karmada-scheduler-estimator github.com/karmada-io/karmada/cmd/scheduler-estimator
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest
+ docker build --build-arg BINARY=karmada-scheduler-estimator --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> bad5cb926810
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 2348c1d233b4
Successfully built 2348c1d233b4
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest
+ set +x
set -e;\
target=$(echo karmada-interpreter-webhook-example);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:31Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-interpreter-webhook-example
!!! Building karmada-interpreter-webhook-example for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:31Z' -o _output/bin/linux/amd64/karmada-interpreter-webhook-example github.com/karmada-io/karmada/examples/customresourceinterpreter/webhook
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest
+ docker build --build-arg BINARY=karmada-interpreter-webhook-example --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> ef33f7c25d66
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> 54a70eaddbb2
Successfully built 54a70eaddbb2
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest
+ set +x
set -e;\
target=$(echo karmada-search);\
make $target GOOS=linux;\
VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada BUILD_PLATFORMS=linux/amd64 hack/docker.sh $target
make[1]: Entering directory '/home/rgrupesh/karmada'
LDFLAGS='-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:46Z' BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-search
!!! Building karmada-search for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.2.0-325-gfa4fecaa -X github.com/karmada-io/karmada/pkg/version.gitCommit=fa4fecaa244703201e6989e23b2a290f8d232774 -X github.com/karmada-io/karmada/pkg/version.gitTreeState=clean -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-07-29T03:52:46Z' -o _output/bin/linux/amd64/karmada-search github.com/karmada-io/karmada/cmd/karmada-search
+ set +x
make[1]: Leaving directory '/home/rgrupesh/karmada'
Building image for linux/amd64: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest
+ docker build --build-arg BINARY=karmada-search --tag swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest --file hack/../cluster/images/Dockerfile hack/../_output/bin/linux/amd64
Sending build context to Docker daemon  583.1MB
Step 1/4 : FROM alpine:3.15.1
 ---> e9adb5357e84
Step 2/4 : ARG BINARY
 ---> Using cache
 ---> b0f4e244ce6a
Step 3/4 : RUN apk add --no-cache ca-certificates
 ---> Using cache
 ---> b262a8bcf010
Step 4/4 : COPY ${BINARY} /bin/${BINARY}
 ---> b35ccabd52e9
Successfully built b35ccabd52e9
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest
+ set +x
make: Leaving directory '/home/rgrupesh/karmada'
Waiting for the host clusters to be ready...
Waiting for kubeconfig file /root/.kube/karmada.config and clusters karmada-host to be ready...
Context "kind-karmada-host" renamed to "karmada-host".
Cluster "kind-karmada-host" set.
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest" with ID "sha256:df2070127c6519f5a813f31862d1bb878b90caefb8aefffc3f8d8eb8aa9ba61f" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest" with ID "sha256:4c26270c617b5aa450ab072b03ddae6709d320f89d6c25b232864f0769945364" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest" with ID "sha256:22d8ce0f60fdf53fa3094315ff387dd43d6cbc3e9b38fa50a14fbebe2c0ab5bd" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest" with ID "sha256:9bd382363cd7a3e5ec1df3d0bab9969b8c9b30a8f5f1a3aa8409fb168187e8e2" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest" with ID "sha256:2348c1d233b489acc6ebe48b1a309f57eeedc8e89ca64acf7c055750cb10cf4c" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest" with ID "sha256:9f0591a94aa0379de988d91e60a5fe89956dcfcfebc258f5b41fe66bbac7c827" not yet present on node "karmada-host-control-plane", loading...
Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-search:latest" with ID "sha256:b35ccabd52e9b9306599ac7c088705b453f688c08a7bde2de65a0de61624de6b" not yet present on node "karmada-host-control-plane", loading...
go install github.com/cloudflare/cfssl/cmd/...@v1.5.0
go: downloading github.com/cloudflare/cfssl v1.5.0
go: downloading github.com/go-sql-driver/mysql v1.4.0
go: downloading github.com/lib/pq v1.3.0
go: downloading github.com/mattn/go-sqlite3 v1.10.0
go: downloading github.com/cloudflare/go-metrics v0.0.0-20151117154305-6a9aea36fb41
go: downloading github.com/zmap/zlint/v2 v2.2.1
go: downloading github.com/jmoiron/sqlx v1.2.0
go: downloading github.com/GeertJohan/go.rice v1.0.0
go: downloading golang.org/x/crypto v0.0.0-20201012173705-84dcc777aaee
go: downloading github.com/kisielk/sqlstruct v0.0.0-20150923205031-648daed35d49
go: downloading github.com/google/certificate-transparency-go v1.0.21
go: downloading github.com/cloudflare/redoctober v0.0.0-20171127175943-746a508df14c
go: downloading github.com/zmap/zcrypto v0.0.0-20200911161511-43ff0ea04f21
go: downloading golang.org/x/net v0.0.0-20201010224723-4f7140c49acb
go: downloading github.com/jmhodges/clock v0.0.0-20160418191101-880ee4c33548
go: downloading github.com/daaku/go.zipexe v1.0.0
go: downloading github.com/golang/protobuf v1.3.1
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/weppos/publicsuffix-go v0.13.0
go: downloading github.com/getsentry/raven-go v0.0.0-20180121060056-563b81fc02b7
go: downloading github.com/certifi/gocertifi v0.0.0-20180118203423-deb3ae2ef261
go: downloading github.com/pkg/errors v0.8.0
Generating a RSA private key
..+++++
..................................+++++
writing new private key to '/root/.karmada/server-ca.key'
-----
Generating a RSA private key
.........................................................................................................+++++
....+++++
writing new private key to '/root/.karmada/front-proxy-ca.key'
-----
2022/07/29 03:54:37 [INFO] generate received request
2022/07/29 03:54:37 [INFO] received CSR
2022/07/29 03:54:37 [INFO] generating key: rsa-2048
2022/07/29 03:54:37 [INFO] encoded CSR
2022/07/29 03:54:37 [INFO] signed certificate with serial number 125627075058778793067147212178427315668327342880
2022/07/29 03:54:37 [INFO] generate received request
2022/07/29 03:54:37 [INFO] received CSR
2022/07/29 03:54:37 [INFO] generating key: rsa-2048
2022/07/29 03:54:37 [INFO] encoded CSR
2022/07/29 03:54:37 [INFO] signed certificate with serial number 132071276715813476564522344276237698658425075366
namespace/karmada-system created
serviceaccount/karmada-controller-manager created
serviceaccount/karmada-scheduler created
serviceaccount/karmada-webhook created
clusterrole.rbac.authorization.k8s.io/karmada-controller-manager created
clusterrolebinding.rbac.authorization.k8s.io/karmada-controller-manager created
secret/karmada-cert-secret created
secret/kubeconfig created
secret/webhook-cert created
statefulset.apps/etcd created
service/etcd-client created
service/etcd created
wait the etcd ready...
pod/etcd-0 condition met

Apply dynamic rendered apiserver service in /tmp/tmp.JOd2u0IW7f/karmada-apiserver.yaml.
deployment.apps/karmada-apiserver created
service/karmada-apiserver created
wait the karmada-apiserver ready...
pod/karmada-apiserver-7bc75b454c-4dvkl condition met

Karmada API Server's IP is: 172.18.0.5, host cluster type is: local
Cluster "karmada-apiserver" set.
User "karmada-apiserver" set.
Context "karmada-apiserver" created.
deployment.apps/karmada-kube-controller-manager created
deployment.apps/karmada-aggregated-apiserver created
service/karmada-aggregated-apiserver created
wait the karmada-aggregated-apiserver ready...
pod/karmada-aggregated-apiserver-75c95c5fb-hvnkn condition met
pod/karmada-aggregated-apiserver-75c95c5fb-lcqz8 condition met
deployment.apps/karmada-search created
service/karmada-search created
wait the karmada-search ready...
pod/karmada-search-59f47bc47b-r5xhm condition met
pod/karmada-search-59f47bc47b-wnmtw condition met
namespace/karmada-system created
customresourcedefinition.apiextensions.k8s.io/clusteroverridepolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/clusterpropagationpolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/clusterresourcebindings.work.karmada.io created
customresourcedefinition.apiextensions.k8s.io/federatedresourcequotas.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/multiclusteringresses.networking.karmada.io created
customresourcedefinition.apiextensions.k8s.io/overridepolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/propagationpolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/resourcebindings.work.karmada.io created
customresourcedefinition.apiextensions.k8s.io/resourceinterpreterwebhookconfigurations.config.karmada.io created
customresourcedefinition.apiextensions.k8s.io/serviceexports.multicluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/serviceimports.multicluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/works.work.karmada.io created
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-config created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-config created
apiservice.apiregistration.k8s.io/v1alpha1.cluster.karmada.io created
service/karmada-aggregated-apiserver created
wait the karmada-aggregated-apiserver Available...
apiservice.apiregistration.k8s.io/v1alpha1.cluster.karmada.io condition met
apiservice.apiregistration.k8s.io/v1alpha1.search.karmada.io created
service/karmada-search created
wait the karmada-search Available...
apiservice.apiregistration.k8s.io/v1alpha1.search.karmada.io condition met
clusterrole.rbac.authorization.k8s.io/cluster-proxy-admin created
clusterrolebinding.rbac.authorization.k8s.io/cluster-proxy-admin created
Switched to context "karmada-host".
deployment.apps/karmada-controller-manager created
deployment.apps/karmada-scheduler created
deployment.apps/karmada-descheduler created
deployment.apps/karmada-webhook created
service/karmada-webhook created
wait the karmada-controller-manager ready...
pod/karmada-controller-manager-56c54786db-g4rxv condition met
pod/karmada-controller-manager-56c54786db-lknnw condition met
wait the karmada-scheduler ready...
pod/karmada-scheduler-6cb58d89f5-5f45x condition met
pod/karmada-scheduler-6cb58d89f5-xn2p2 condition met
wait the kube-controller-manager ready...
pod/karmada-kube-controller-manager-5cc595654f-xrh8w condition met
wait the karmada-webhook ready...
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(1 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(2 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(3 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(4 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(5 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(6 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(7 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(8 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(9 times)
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed, retrying(10 times)
kubectl wait --for=condition=Ready --timeout=30s pods -l app=karmada-webhook -n karmada-system failed
pod/karmada-webhook-c8657fd8-4sqrq condition met
error: timed out waiting for the condition on pods/karmada-webhook-c8657fd8-m6kvk
kubectl describe info:
Name:         karmada-webhook-c8657fd8-4sqrq
Namespace:    karmada-system
Priority:     0
Node:         karmada-host-control-plane/172.18.0.5
Start Time:   Fri, 29 Jul 2022 03:55:06 +0000
Labels:       app=karmada-webhook
              pod-template-hash=c8657fd8
Annotations:  
Status:       Running
IP:           10.244.0.17
IPs:
  IP:           10.244.0.17
Controlled By:  ReplicaSet/karmada-webhook-c8657fd8
Containers:
  karmada-webhook:
    Container ID:  containerd://03b14dde0b084e54867780705630efa0a375a873ff51939e3ba52d939fce2b0c
    Image:         swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
    Image ID:      docker.io/library/import-2022-07-29@sha256:72cfb249a31571fd13224a61d2c8d76a1e8303931aac0983b6ec547bf885f285
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /bin/karmada-webhook
      --kubeconfig=/etc/kubeconfig
      --bind-address=0.0.0.0
      --secure-port=8443
      --cert-dir=/var/serving-cert
      --v=4
    State:          Running
      Started:      Fri, 29 Jul 2022 03:55:12 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      http-get https://:8443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    
    Mounts:
      /etc/kubeconfig from kubeconfig (rw,path="kubeconfig")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vpqvj (ro)
      /var/serving-cert from cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kubeconfig:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubeconfig
    Optional:    false
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-cert
    Optional:    false
  kube-api-access-vpqvj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node-role.kubernetes.io/master op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m48s  default-scheduler  Successfully assigned karmada-system/karmada-webhook-c8657fd8-4sqrq to karmada-host-control-plane
  Normal  Pulled     5m42s  kubelet            Container image "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest" already present on machine
  Normal  Created    5m42s  kubelet            Created container karmada-webhook
  Normal  Started    5m42s  kubelet            Started container karmada-webhook

Name:         karmada-webhook-c8657fd8-m6kvk
Namespace:    karmada-system
Priority:     0
Node:         karmada-host-control-plane/172.18.0.5
Start Time:   Fri, 29 Jul 2022 03:55:06 +0000
Labels:       app=karmada-webhook
              pod-template-hash=c8657fd8
Annotations:  
Status:       Running
IP:           10.244.0.18
IPs:
  IP:           10.244.0.18
Controlled By:  ReplicaSet/karmada-webhook-c8657fd8
Containers:
  karmada-webhook:
    Container ID:  containerd://b4029148d8c015575c713e1c0eeb3e6ce73a2330c4aff87f0d293338b42cf812
    Image:         swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest
    Image ID:      docker.io/library/import-2022-07-29@sha256:72cfb249a31571fd13224a61d2c8d76a1e8303931aac0983b6ec547bf885f285
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /bin/karmada-webhook
      --kubeconfig=/etc/kubeconfig
      --bind-address=0.0.0.0
      --secure-port=8443
      --cert-dir=/var/serving-cert
      --v=4
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 29 Jul 2022 03:58:17 +0000
      Finished:     Fri, 29 Jul 2022 03:58:18 +0000
    Ready:          False
    Restart Count:  5
    Readiness:      http-get https://:8443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    
    Mounts:
      /etc/kubeconfig from kubeconfig (rw,path="kubeconfig")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nrp45 (ro)
      /var/serving-cert from cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kubeconfig:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubeconfig
    Optional:    false
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-cert
    Optional:    false
  kube-api-access-nrp45:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node-role.kubernetes.io/master op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  5m48s                 default-scheduler  Successfully assigned karmada-system/karmada-webhook-c8657fd8-m6kvk to karmada-host-control-plane
  Normal   Pulled     4m4s (x5 over 5m42s)  kubelet            Container image "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest" already present on machine
  Normal   Created    4m4s (x5 over 5m42s)  kubelet            Created container karmada-webhook
  Normal   Started    4m4s (x5 over 5m42s)  kubelet            Started container karmada-webhook
  Warning  BackOff    40s (x27 over 5m40s)  kubelet            Back-off restarting failed container
kubectl logs info:
I0729 03:55:12.536450       1 webhook.go:82] karmada-webhook version: version.Info{GitVersion:"v1.2.0-325-gfa4fecaa", GitCommit:"fa4fecaa244703201e6989e23b2a290f8d232774", GitTreeState:"clean", BuildDate:"2022-07-29T03:51:40Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
I0729 03:55:12.588868       1 webhook.go:112] registering webhooks to the webhook server
I0729 03:55:12.590140       1 internal.go:362] "Starting server" kind="health probe" addr="[::]:8000"
I0729 03:55:12.590215       1 internal.go:362] "Starting server" path="/metrics" kind="metrics" addr="[::]:8080"
I0729 03:55:12.590389       1 shared_informer.go:285] caches populated
I0729 03:58:18.049641       1 internal.go:362] "Starting server" kind="health probe" addr="[::]:8000"
I0729 03:58:18.049687       1 internal.go:362] "Starting server" path="/metrics" kind="metrics" addr="[::]:8080"
I0729 03:58:18.049789       1 shared_informer.go:285] caches populated
I0729 03:58:18.049884       1 internal.go:567] "Stopping and waiting for non leader election runnables"
I0729 03:58:18.049911       1 internal.go:571] "Stopping and waiting for leader election runnables"
I0729 03:58:18.049921       1 internal.go:577] "Stopping and waiting for caches"
I0729 03:58:18.049936       1 internal.go:581] "Stopping and waiting for webhooks"
I0729 03:58:18.049945       1 internal.go:585] "Wait completed, proceeding to shutdown the manager"
E0729 03:58:18.049957       1 webhook.go:129] webhook server exits unexpectedly: too many open files
E0729 03:58:18.049995       1 run.go:74] "command failed" err="too many open files"

RainbowMango commented 1 year ago

E0729 03:58:18.049957 1 webhook.go:129] webhook server exits unexpectedly: too many open files

Please check your system limitations:

$ ulimit -a

And look at the open files limitation.

Maybe you need to enlarge it by ulimit -n <number>.

rgrupesh commented 1 year ago

Should I still increase? If so, by how much?

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63976
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63976
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

RainbowMango commented 1 year ago

You can increase it to a larger number and have a try. I set it to 65535 on my machine.

open files                      (-n) 65535

rgrupesh commented 1 year ago

Thanks, now local Karmada is running in VM. Screenshot from 2022-07-29 13-19-44

But to access to the dashboard, I need get Bearer Token. I ran into this error when I tried to generate that. Screenshot from 2022-07-29 13-19-59

RainbowMango commented 1 year ago

Congrats!!

For the dashboard thing please talk to @chinmaym07 or open an issue in karmada-io/dashboard. Just a reminder, you should use the feature branch ng-dashboard-react, not main.

RainbowMango commented 1 year ago

/close

karmada-bot commented 1 year ago

@RainbowMango: Closing this issue.

In response to [this](https://github.com/karmada-io/karmada/issues/2020#issuecomment-1198988546): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

Wang-Kai commented 1 year ago

You can increase it to a larger number and have a try. I set it to 65535 on my machine.
open files                      (-n) 65535

I met the same issue.

haiker2011 commented 10 months ago

I met the same issue even set open files to 65535

haiker2011 commented 10 months ago

/reopen

karmada-bot commented 10 months ago

@haiker2011: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/karmada-io/karmada/issues/2020#issuecomment-1582234719): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

RainbowMango commented 10 months ago

@haiker2011 Please check this document: https://karmada.io/docs/troubleshooting/trouble-shooting/#karmada-webhook-keeps-on-crashing-due-to-too-many-open-files

haiker2011 commented 10 months ago

@RainbowMango Thanks, I have solved this issue use this document

karmada-io / karmada

Error while setting up karmada locally #2020

github.com/mattn/go-sqlite3