search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.12k stars 806 forks source link

Join cluster error #1291

Closed chenrc0529-ai closed 2 years ago

chenrc0529-ai commented 2 years ago

[root@k8s-master ~]# kubectl karmada join kubernetes-admin --kubeconfig=/etc/karmada/karmada-apiserver.config --cluster-kubeconfig=$HOME/.kube/config W0120 16:17:30.264037 12485 cluster.go:106] failed to create cluster(kubernetes-admin). error: Cluster.cluster.karmada.io "kubernetes-admin" is invalid: [spec.secretRef.namespace: Required value, spec.secretRef.name: Required value, spec.impersonatorSecretRef.namespace: Required value, spec.impersonatorSecretRef.name: Required value] W0120 16:17:30.264245 12485 cluster.go:50] failed to create cluster(kubernetes-admin). error: Cluster.cluster.karmada.io "kubernetes-admin" is invalid: [spec.secretRef.namespace: Required value, spec.secretRef.name: Required value, spec.impersonatorSecretRef.namespace: Required value, spec.impersonatorSecretRef.name: Required value] Error: failed to create cluster(kubernetes-admin) object. error: Cluster.cluster.karmada.io "kubernetes-admin" is invalid: [spec.secretRef.namespace: Required value, spec.secretRef.name: Required value, spec.impersonatorSecretRef.namespace: Required value, spec.impersonatorSecretRef.name: Required value]

kubectl-karmada version： make kubectl-karmada by the latest codes in github [root@k8s-master ~]# kubectl-karmada version kubectl karmada version: version.Info{GitVersion:"", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-01-20T02:30:56Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

RainbowMango commented 2 years ago

Can you tell us the versions of both Karmada and kubectl-karmada?

Karmada version:
kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
chenrc0529-ai commented 2 years ago

Can you tell us the versions of both Karmada and kubectl-karmada?

Karmada version:
kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):

make kubectl-karmada by the latest codes

RainbowMango commented 2 years ago

What's the output of kubeclt karmada version?

Which version Karmada do you use?

chenrc0529-ai commented 2 years ago

What's the output of kubeclt karmada version?

Which version Karmada do you use?

[root@k8s-master ~]# kubectl karmada version kubectl karmada version: version.Info{GitVersion:"", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-01-20T02:30:56Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

XiShanYongYe-Chang commented 2 years ago

Hi @chenrc0529-ai , have you ever found the reason why you failed?

chenrc0529-ai commented 2 years ago

Hi @chenrc0529-ai , have you ever found the reason why you failed?

Not yet

chenrc0529-ai commented 2 years ago

Can you tell us the versions of both Karmada and kubectl-karmada?

Karmada version:
kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):

How can i solve this issue?

RainbowMango commented 2 years ago

Still don't know which version you are using.

RainbowMango commented 2 years ago

kubectl karmada version: version.Info{GitVersion:"", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-01-20T02:30:56Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

No version info.

chenrc0529-ai commented 2 years ago

kubectl-karmada

I pull the latest codes from github in Jan 13. Because i cannot download the image， so i modify the image source, then make kubectl-karmada in local.

chenrc0529-ai commented 2 years ago

kubectl karmada version: version.Info{GitVersion:"", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-01-20T02:30:56Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

No version info. I pull the latest codes from github in Jan 13. Because i cannot download the image， so i modify the image source, then make kubectl-karmada in local.

SZT0728 commented 2 years ago

kubectl karmada version: version.Info{GitVersion:"", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-01-20T02:30:56Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

No version info.

i met the same question. i remember that it's can join cluster a few days ago. but now it show the errmsg: W0121 16:58:03.484629 20954 cluster.go:106] failed to create cluster(kubernetes-admin). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource W0121 16:58:03.484873 20954 cluster.go:50] failed to create cluster(kubernetes-admin). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource Error: failed to create cluster(kubernetes-admin) object. error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource

XiShanYongYe-Chang commented 2 years ago

Hi @SZT0728, the problem you have is different from the one described above. We have already removed cluster.karmada.io validate in webhook with PR #1152.

SZT0728 commented 2 years ago

Hi @SZT0728, the problem you have is different from the one described above. We have already removed cluster.karmada.io validate in webhook with PR #1152.

so,i build the kubectl-karmada from the master latest source??

XiShanYongYe-Chang commented 2 years ago

@SZT0728 , you can check validatingwebhookconfigurations on your environment:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config -oyaml

Check whether it is the latest,

XiShanYongYe-Chang commented 2 years ago

Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "validating-config" not found how can i solve this?

Get it when you connect to karmada-apiserver, for example: kubectl --context karmada-apiserver get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config -oyaml

SZT0728 commented 2 years ago

Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "validating-config" not found how can i solve this?

Get it when you connect to karmada-apiserver, for example: kubectl --context karmada-apiserver get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config -oyaml

apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: creationTimestamp: "2022-01-21T08:43:13Z" generation: 1 labels: app: validating-config managedFields:

SZT0728 commented 2 years ago

Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "validating-config" not found how can i solve this?

Get it when you connect to karmada-apiserver, for example: kubectl --context karmada-apiserver get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config -oyaml

now i join the cluster, it show the error like this issue' describe failed to create cluster(kubernetes). error: Cluster.cluster.karmada.io "kubernetes" is invalid: [spec.secretRef.namespace: Required value, spec.secretRef.name: Required value, spec.impersonatorSecretRef.namespace: Required value, spec.impersonatorSecretRef.name: Required value] ...

prodanlabs commented 2 years ago

hi @chenrc0529-ai is your problem solved?

prodanlabs commented 2 years ago

hi @SZT0728 Is ectd not persistent and restarted?

SZT0728 commented 2 years ago

hi @SZT0728 Is ectd not persistent and restarted?

no, it's normal

chenrc0529-ai commented 2 years ago

hi @chenrc0529-ai is your problem solved?

Not yet

prodanlabs commented 2 years ago

hi @chenrc0529-ai is your problem solved?

Not yet

Can you take a look at your apiservice?

root@dev-k8s-master01:~# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get apiservices | grep karmada
v1alpha1.cluster.karmada.io            karmada-system/karmada-aggregated-apiserver   True        9d
v1alpha1.config.karmada.io             Local                                         True        9d
v1alpha1.policy.karmada.io             Local                                         True        9d
v1alpha1.work.karmada.io               Local                                         True        9d
v1alpha2.work.karmada.io               Local                                         True        9d
chenrc0529-ai commented 2 years ago

kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get apiservices | grep karmada

[root@k8s-master ~]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get apiservices | grep karmada v1alpha1.cluster.karmada.io karmada-system/karmada-aggregated-apiserver True 4d v1alpha1.config.karmada.io Local True 4d v1alpha1.policy.karmada.io Local True 4d v1alpha1.work.karmada.io Local True 4d v1alpha2.work.karmada.io Local True 4d

RainbowMango commented 2 years ago

Hi @chenrc0529-ai The error log is similar to #1306. Can you help test it on your side?

chenrc0529-ai commented 2 years ago

kubectl karmada join kubernetes-admin --kubeconfig=/etc/karmada/karmada-apiserver.config --cluster-kubeconfig=$HOME/.kube/config

I modify the file pkg/util/secret.go, and rebuild the kubectl-karmada， the error still exist.

Hi @chenrc0529-ai The error log is similar to #1306. Can you help test it on your side?

I modify the file pkg/util/secret.go, and rebuild the kubectl-karmada， the error still exist.

RainbowMango commented 2 years ago

Well, @chenrc0529-ai Do you know how reproduce it ? I can help reproduce it on my side.

By the way, does the cluster kubernetes-admin ever joined before?( Is it the first time joining this cluster to Karmada?)

MingZhang-YBPS commented 2 years ago

I met similar issue, and the log in karmada-aggregated-apiserver says W0211 12:58:51.501491 1 dispatcher.go:150] Failed calling webhook, failing closed cluster.karmada.io: failed calling webhook "cluster.karmada.io": the server could not find the requested resource

YuxiJin-tobeyjin commented 2 years ago

Same issue with the latest release, while using v0.10.1 there is no such problem

kubectl karmada version
kubectl karmada version: version.Info{GitVersion:"v1.0.1", GitCommit:"ce2267c5d04a5e97368285fd64fdbe5fa285856e", GitTreeState:"clean", BuildDate:"2022-01-21T09:37:08Z", GoVersion:"go1.16.13", Compiler:"gc", Platform:"linux/amd64"}
kubectl config rename-context "kubernetes-admin@cluster.local" "host" --kubeconfig=host-config
Context "kubernetes-admin@cluster.local" renamed to "host".
kubectl karmada join host --cluster-kubeconfig=host-config 
W0216 11:52:16.931388    8410 cluster.go:106] failed to create cluster(host). error: the server could not find the requested resource (post clusters.cluster.karmada.io)
W0216 11:52:16.931859    8410 cluster.go:50] failed to create cluster(host). error: the server could not find the requested resource (post clusters.cluster.karmada.io)
Error: failed to create cluster(host) object. error: the server could not find the requested resource (post clusters.cluster.karmada.io)

Maybe need to wait for a new release? Do we have a plan?

XiShanYongYe-Chang commented 2 years ago

Hi @chenrc0529-ai @SZT0728 @chenrc0529-ai @MingZhang-YBPS @YuxiJin-tobeyjin everybody. Can someone provide a full process for reproducing the problem?

I use the v1.0.1 version and make kubectl-karmada on my local site, but didn't reproduce the error.

There are a few questions that need to be confirmed:

YuxiJin-tobeyjin commented 2 years ago

Is the cluster added for the first time? // yes

kubectl karmada version // as follows

kubectl karmada version
kubectl karmada version: version.Info{GitVersion:"v1.0.1", GitCommit:"ce2267c5d04a5e97368285fd64fdbe5fa285856e", GitTreeState:"clean", BuildDate:"2022-01-21T09:37:08Z", GoVersion:"go1.16.13", Compiler:"gc", Platform:"linux/amd64"}

whether the karmada version is upgraded from an earlier version // no, deploy from scratch, just deploy and undeploy sometimes (undeploy with "bash -x undeploy-karmada.sh ~/.kube/config kubernetes-admin@cluster.local" )

Reproduce:

1、deploy a local k8s cluster

2、bash remote-up-karmada.sh ~/.kube/config kubernetes-admin@cluster.local CLUSTER_IP_ONLY=true

Generating a 2048 bit RSA private key
......................+++
....................................................................................+++
writing new private key to '/root/.karmada/server-ca.key'
-----
Generating a 2048 bit RSA private key
.+++
....................+++
writing new private key to '/root/.karmada/front-proxy-ca.key'
-----
2022/02/16 15:11:12 [INFO] generate received request
2022/02/16 15:11:12 [INFO] received CSR
2022/02/16 15:11:12 [INFO] generating key: rsa-2048
2022/02/16 15:11:13 [INFO] encoded CSR
2022/02/16 15:11:13 [INFO] signed certificate with serial number 370804119998102905444109242487124500837254195160
2022/02/16 15:11:13 [INFO] generate received request
2022/02/16 15:11:13 [INFO] received CSR
2022/02/16 15:11:13 [INFO] generating key: rsa-2048
2022/02/16 15:11:13 [INFO] encoded CSR
2022/02/16 15:11:13 [INFO] signed certificate with serial number 532867452045251919217757510499376049896225951800
namespace/karmada-system created
serviceaccount/karmada-controller-manager created
serviceaccount/karmada-scheduler created
serviceaccount/karmada-webhook created
clusterrole.rbac.authorization.k8s.io/karmada-controller-manager unchanged
clusterrolebinding.rbac.authorization.k8s.io/karmada-controller-manager unchanged
secret/karmada-cert-secret created
secret/kubeconfig created
secret/webhook-cert created
statefulset.apps/etcd created
service/etcd-client created
service/etcd created
wait the etcd ready...
pod/etcd-0 condition met

Apply dynamic rendered apiserver service in /tmp/tmp.gHGDw6kEUh/karmada-apiserver.yaml.
deployment.apps/karmada-apiserver created
service/karmada-apiserver created
wait the karmada-apiserver ready...
pod/karmada-apiserver-786875ff54-9ksf8 condition met

Karmada API Server's IP is: 10.3.68.67, host cluster type is: remote
Cluster "karmada-apiserver" set.
User "karmada-apiserver" set.
Context "karmada-apiserver" created.
deployment.apps/karmada-kube-controller-manager created
deployment.apps/karmada-aggregated-apiserver created
service/karmada-aggregated-apiserver created
wait the karmada-aggregated-apiserver ready...
pod/karmada-aggregated-apiserver-86cb967c86-6wtrq condition met
namespace/karmada-system created
customresourcedefinition.apiextensions.k8s.io/clusteroverridepolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/clusterpropagationpolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/clusterresourcebindings.work.karmada.io created
customresourcedefinition.apiextensions.k8s.io/overridepolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/propagationpolicies.policy.karmada.io created
customresourcedefinition.apiextensions.k8s.io/resourcebindings.work.karmada.io created
customresourcedefinition.apiextensions.k8s.io/resourceinterpreterwebhookconfigurations.config.karmada.io created
customresourcedefinition.apiextensions.k8s.io/serviceexports.multicluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/serviceimports.multicluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/works.work.karmada.io created
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-config created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-config created
apiservice.apiregistration.k8s.io/v1alpha1.cluster.karmada.io created
service/karmada-aggregated-apiserver created
wait the karmada-aggregated-apiserver Available...
apiservice.apiregistration.k8s.io/v1alpha1.cluster.karmada.io condition met
Switched to context "kubernetes-admin@cluster.local".
deployment.apps/karmada-controller-manager created
deployment.apps/karmada-scheduler created
deployment.apps/karmada-webhook created
service/karmada-webhook created
wait the karmada-controller-manager ready...
pod/karmada-controller-manager-b754b59d7-94wt6 condition met
wait the karmada-scheduler ready...
pod/karmada-scheduler-c48f85d94-zfslp condition met
wait the kube-controller-manager ready...
pod/karmada-kube-controller-manager-6758ff89b-bt8bc condition met
wait the karmada-webhook ready...
pod/karmada-webhook-85d564b96-ps5qs condition met

Karmada is installed.

Kubeconfig for karmada in file: /root/.kube/config, so you can run:
  export KUBECONFIG="/root/.kube/config"
Or use kubectl with --kubeconfig=/root/.kube/config
Please use 'kubectl config use-context karmada-apiserver' to switch the cluster of karmada control plane
And use 'kubectl config use-context kubernetes-admin@cluster.local' for debugging karmada installation

3、kubectl config rename-context "kubernetes-admin@cluster.local" "host" --kubeconfig=host-config

Context "kubernetes-admin@cluster.local" renamed to "host".

4、kubectl karmada join host --cluster-kubeconfig=host-config 

W0216 15:13:55.964392   18057 cluster.go:106] failed to create cluster(host). error: the server could not find the requested resource (post clusters.cluster.karmada.io)
W0216 15:13:55.964561   18057 cluster.go:50] failed to create cluster(host). error: the server could not find the requested resource (post clusters.cluster.karmada.io)
Error: failed to create cluster(host) object. error: the server could not find the requested resource (post clusters.cluster.karmada.io)

Hope this helps.

XiShanYongYe-Chang commented 2 years ago

Hi @YuxiJin-tobeyjin, I follow your step, it works ok on my side.

Can you get clusters correctly with the command:

kubectl get clusters --v=6
YuxiJin-tobeyjin commented 2 years ago 
kubectl config use-context  karmada-apiserver 
Switched to context "karmada-apiserver".

kubectl get clusters --v=6
I0216 18:42:18.105326   21340 loader.go:375] Config loaded from file:  /root/.kube/config
I0216 18:42:18.182650   21340 round_trippers.go:443] GET https://10.3.68.67:5443/apis/cluster.karmada.io/v1alpha1/clusters?limit=500 200 OK in 54 milliseconds
No resources found
XiShanYongYe-Chang commented 2 years ago

kubectl config use-context karmada-apiserver Switched to context "karmada-apiserver".

Is this context not used when you join a cluster?

YuxiJin-tobeyjin commented 2 years ago

Yes, of course i did, the logs are as follows:

kubectl config use-context karmada-apiserver
Switched to context "karmada-apiserver".

kubectl karmada join host --cluster-kubeconfig=host-config 
W0216 14:18:22.478715    8192 cluster.go:106] failed to create cluster(host). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource
W0216 14:18:22.480569    8192 cluster.go:50] failed to create cluster(host). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource
Error: failed to create cluster(host) object. error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource

Maybe the reason is validating-config, as mentioned above i've deployed and undeployed karmada many times, but it seems validating-config keeps unchanged, i will try it again tomorrow

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config
NAME                WEBHOOKS   AGE
validating-config   6          5d18h
XiShanYongYe-Chang commented 2 years ago

The error info is different from before.

Maybe check the logs for the 'Karmada-webhook' pod to see what's going on.

wenhuwang commented 2 years ago

Yes, of course i did, the logs are as follows:

kubectl config use-context karmada-apiserver
Switched to context "karmada-apiserver".

kubectl karmada join host --cluster-kubeconfig=host-config 
W0216 14:18:22.478715    8192 cluster.go:106] failed to create cluster(host). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource
W0216 14:18:22.480569    8192 cluster.go:50] failed to create cluster(host). error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource
Error: failed to create cluster(host) object. error: Internal error occurred: failed calling webhook "cluster.karmada.io": the server could not find the requested resource

Maybe the reason is validating-config, as mentioned above i've deployed and undeployed karmada many times, but it seems validating-config keeps unchanged, i will try it again tomorrow

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io validating-config
NAME                WEBHOOKS   AGE
validating-config   6          5d18h

@YuxiJin-tobeyjin I encountered the same problem，it can be solved by doing the following:

  1. delete the following configuration in validating-config validatingwebhookconfigurations resource： 
    - admissionReviewVersions:
- v1
clientConfig:
caBundle: *****
url: https://karmada-webhook.karmada-system.svc:443/validate-cluster
failurePolicy: Fail
matchPolicy: Equivalent
name: cluster.karmada.io
namespaceSelector: {}
objectSelector: {}
rules:
- apiGroups:
- cluster.karmada.io
apiVersions:
- '*'
operations:
- CREATE
- UPDATE
resources:
- clusters
scope: Cluster
sideEffects: None
timeoutSeconds: 3
  2. Clean up the secret resource corresponding to the cluster 
    # kubectl -n karmada-cluster get secret
NAME                        TYPE     DATA   AGE
cluster1                    Opaque   2      20h
cluster1-impersonator       Opaque   1      20h
  3. join cluster to karmada.
YuxiJin-tobeyjin commented 2 years ago

Thanks a lot @wenhuwang , problem solves. @XiShanYongYe-Chang , so the cluster.karmada.io webhook-configuration needs to be deleted from the deploy files?

XiShanYongYe-Chang commented 2 years ago

We remove cluster.karmada.io webhook-configuration with PR #1152, If you use the master branch, it is needed. But with release v1.0.1, we haven't merged it.

Is there something wrong?

XiShanYongYe-Chang commented 2 years ago

In conclusion, two causes are found to cause the failure to join the cluster:

XiShanYongYe-Chang commented 2 years ago

Close first, we can reopen it if anyone occurs this problem again.

/close

karmada-bot commented 2 years ago

@XiShanYongYe-Chang: Closing this issue.

In response to [this](https://github.com/karmada-io/karmada/issues/1291#issuecomment-1058877156): >Close first, we can reopen it if anyone occurs this problem again. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.