search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.43k stars 877 forks source link

member cluster is unknown,push pull mode both #3973

Open 13567436138 opened 1 year ago

13567436138 commented 1 year ago

Please provide an in-depth description of the question you have:

What do you think about this question?:

[root@localhost karmada]# kubectl get cluster --kubeconfig ~/kube-karmada 
NAME       VERSION   MODE   READY     AGE
cluster1             Pull   Unknown   89m
cluster2             Pull   Unknown   23m

Environment:

chaosi-zju commented 1 year ago

could you please try:

  1. excute kubectl describe cluster cluster1
  2. how did you install your raw k8s cluster? Can you show me your kubeconfig of karmada cluster and cluster1?
  3. does the network between karmada cluster and cluster1 is ok? Or the public network is ok but you specified a local ip in some steps?
chaosi-zju commented 1 year ago

Hello, is this problem still exist? did you find the reason?

SeekerRook commented 1 year ago

Hello. I have the same problem as @13567436138 . 

># kubectl get clusters --kubeconfig=karmada-config                                                                  
NAME       VERSION   MODE   READY     AGE
cluster1             Pull   Unknown   17d
cluster2             Pull   Unknown   121m

As an answer to @chaosi-zju Here are the specs for my personal case:

- Cluster2 description:

Name: cluster2 Namespace:
Labels: Annotations: API Version: cluster.karmada.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-09-01T07:58:40Z Generation: 1 Resource Version: 41556 UID: 19159559-0ab6-4f2f-8166-acb545d4cf70 Spec: Id: 2b06157f-a7b7-4905-bf33-5b38582fc74e Impersonator Secret Ref: Name: cluster2-impersonator Namespace: karmada-cluster Resource Models: Grade: 0 Ranges: Max: 1 Min: 0 Name: cpu Max: 4Gi Min: 0 Name: memory Grade: 1 Ranges: Max: 2 Min: 1 Name: cpu Max: 16Gi Min: 4Gi Name: memory Grade: 2 Ranges: Max: 4 Min: 2 Name: cpu Max: 32Gi Min: 16Gi Name: memory Grade: 3 Ranges: Max: 8 Min: 4 Name: cpu Max: 64Gi Min: 32Gi Name: memory Grade: 4 Ranges: Max: 16 Min: 8 Name: cpu Max: 128Gi Min: 64Gi Name: memory Grade: 5 Ranges: Max: 32 Min: 16 Name: cpu Max: 256Gi Min: 128Gi Name: memory Grade: 6 Ranges: Max: 64 Min: 32 Name: cpu Max: 512Gi Min: 256Gi Name: memory Grade: 7 Ranges: Max: 128 Min: 64 Name: cpu Max: 1Ti Min: 512Gi Name: memory Grade: 8 Ranges: Max: 9223372036854775807 Min: 128 Name: cpu Max: 9223372036854775807 Min: 1Ti Name: memory Secret Ref: Name: cluster2 Namespace: karmada-cluster Sync Mode: Pull Status: Events: 

I installed kubernetes using [K3s](https://docs.k3s.io/) and karmada using Helm.  Here are the kubeconfigs
-  Cluster-Manager

apiVersion: v1 clusters:


-  Cluster 1

apiVersion: v1 clusters:



As for the network connection all 3 clusters are in VMs on the same host connected using a Host-Only Adapter and a NAT adapter to connect to the Internet (the IPs on the kubeconfigs are the IPs of the Host-Only Adater).

Thanks in advance!
chaosi-zju commented 1 year ago

@SeekerRook Hello, latest two days I have thought about your problem, but haven't figured it out. I want more information from you~

  1. Find out which kubeconfig the karmada-agent in member cluster is using to connetc karmada-apiserver, you can try by kubectl describe command.
  2. Can you enter the VM of member cluster? find out whether kubectl command in member cluster VM is executed successful with above kubeconfig?

I mean, your problem may related to the network, the train of thought is try to make sure whether karmada-agent can connect to the karmada-apiserver with provided kubeconfig.

SeekerRook commented 1 year ago

Hello @chaosi-zju and thanks for the reply!

  1. There is no resource named karmada-agent in the member cluster: 
    
#>kubectl get all --kubeconfig cluster2 -n karmada-cluster              
No resources found in karmada-cluster namespace.
#> kubectl get all --kubeconfig cluster2 -n karmada-system              
NAME                           READY   STATUS    RESTARTS        AGE
pod/karmada-866648bb56-pxkxj   1/1     Running   15 (4m7s ago)   2d22h

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/karmada 1/1 1 1 2d22h

NAME DESIRED CURRENT READY AGE replicaset.apps/karmada-866648bb56 1 1 1 2d22h


2. As for the connection I can reach karmada cluster from inside the member VM using the kubeconfig.

As an update since my previous comment the problem also appears when registering the member clusters in push mode.

 Keep in mind that I am new to Karmada so perhaps I have done something wrong during the setup.

Thanks again for the help.
chaosi-zju commented 1 year ago

@SeekerRook Glad to help you the newly try. Indeed, our documentation for helm installation method is not very detailed, I will improve the doc.

Let's review the installation steps：

  1. let's firstly deploy karmada control plane components in host cluster, you can refer to following commands: 
    
git clone https://github.com/karmada-io/karmada.git
cd karmada

you should update the ~/.kube/karmada.config to your real karmada-control-plane cluster kubeconfig

export KUBECONFIG=~/.kube/karmada.config

(optional) pull images in advance and avoid duplicate downloads when each installation

for img in cat charts/karmada/values.yaml | grep -C 1 'repository:' | sed 's/*karmadaImageVersion/latest/g' | awk -F ':' '{print $2}' | sed 's/\"//g' | xargs -n3 | awk '{print $1"/"$2":"$3}'; do docker pull $img; done

deply karmada

helm install karmada -n karmada-system --create-namespace --dependency-update --set apiServer.hostNetwork=true ./charts/karmada --debug


tell me whether this step is ok after your try, you can paste a picture about the result of `kubectl get po -A -o wide` ~

2. deploy `karmada-agent` in member cluster.
you can refer to [demo](https://github.com/karmada-io/karmada/tree/master/charts/karmada#1-install-agent) to deploy `agent` in member cluster. The agent is necessary for member cluster in `pull` mode.
tell me any problem in following the linked demo.
SeekerRook commented 1 year ago

@chaosi-zju I tried the installation process you specified and it gave me the following error

helm install karmada -n karmada-system --create-namespace --dependency-update --set apiServer.hostNetwork=true --kubeconfig ../remote/cluster-manager ./charts/karmada --debug

install.go:200: [debug] Original chart version: ""
install.go:217: [debug] CHART PATH: /home/chris/Code/Thesis/karmada/charts/karmada

client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-pre-job" ServiceAccount
client.go:482: [debug] Ignoring delete failure for "karmada-pre-job" /v1, Kind=ServiceAccount: serviceaccounts "karmada-pre-job" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-pre-job" ClusterRole
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-pre-job" ClusterRoleBinding
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-config" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-config" /v1, Kind=ConfigMap: configmaps "karmada-config" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-autoscaling-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-autoscaling-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-autoscaling-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-config-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-config-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-config-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-kustomization" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-kustomization" /v1, Kind=ConfigMap: configmaps "karmada-crds-kustomization" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-multicluster-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-multicluster-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-multicluster-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-networking-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-networking-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-networking-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-policy-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-policy-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-policy-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "karmada-crds-work-bases" ConfigMap
client.go:482: [debug] Ignoring delete failure for "karmada-crds-work-bases" /v1, Kind=ConfigMap: configmaps "karmada-crds-work-bases" not found
client.go:134: [debug] creating 1 resource(s)
client.go:134: [debug] creating 1 resource(s)
client.go:706: [debug] Watching for changes to Job karmada-pre-install with timeout of 5m0s
client.go:734: [debug] Add/Modify event for karmada-pre-install: ADDED
client.go:773: [debug] karmada-pre-install: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-pre-install: MODIFIED
client.go:773: [debug] karmada-pre-install: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-pre-install: MODIFIED
client.go:773: [debug] karmada-pre-install: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-pre-install: MODIFIED
client.go:478: [debug] Starting delete for "karmada-pre-install" Job
client.go:134: [debug] creating 12 resource(s)
client.go:134: [debug] creating 1 resource(s)
client.go:706: [debug] Watching for changes to Job karmada-post-install with timeout of 5m0s
client.go:734: [debug] Add/Modify event for karmada-post-install: ADDED
client.go:773: [debug] karmada-post-install: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-post-install: MODIFIED
client.go:773: [debug] karmada-post-install: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-post-install: MODIFIED
client.go:773: [debug] karmada-post-install: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-post-install: MODIFIED
client.go:773: [debug] karmada-post-install: Jobs active: 0, jobs failed: 1, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-post-install: MODIFIED
client.go:773: [debug] karmada-post-install: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

helm.go:84: [debug] failed post-install: 1 error occurred:
        * timed out waiting for the condition

INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:147
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598

A similar error has occured to me before when installing from the github repo and i found a solution by using the helm repo karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts.

and this command

helm install karmada karmada-charts/karmada --kubeconfig=cluster-manager  --create-namespace --namespace karmada-system  --version=1.6.0  --set apiServer.hostNetwork=false  --set apiServer.serviceType=NodePort  --set apiServer.nodePort=32443  --set certs.auto.hosts[0]="kubernetes.default.svc"  --set certs.auto.hosts[1]="*.etcd.karmada-system.svc.cluster.local"  --set certs.auto.hosts[2]="*.karmada-system.svc.cluster.local"  --set certs.auto.hosts[3]="*.karmada-system.svc"  --set certs.auto.hosts[4]="localhost"  --set certs.auto.hosts[5]="127.0.0.1"  --set certs.auto.hosts[6]="192.168.56.110"

Is it possible that by adding --set apiServer.hostNetwork=true (the only addition between the default and my installation) that it will do the same?

Another question as it is not clear in the documentation: are there any steps required for push mode cluster beyond kubectl karmada join cluster1 --kubeconfig=<karmada-config> --cluster-kubeconfig<member-config> --cluster-context=<context-name> ? Also the kubeconfig of the --kubeconfig flag should be the karmada-config not the cluster-manager config if i understand correctly.

chaosi-zju commented 1 year ago

Problem 1:

timed out waiting for the condition

You should execute kubectl get po -o wide -A to find which pod failed, and then kubectl describe po xxx -n yyy to figure why it failed.

Mostly because Init:ImagePullBackOff (if so, I advise you to pull image before helm install, I provided you a command to pull image in advance in above comment, but it may be useless because I forgot your cluster is in VM, so, can you refer to that command to pull image in VM?).

If the failed pod is not karmada-apiserver, is has nothing to do with --set apiServer.hostNetwork=true.

Besides, you can refer to #3905

Problem 2:

are there any steps required for push mode cluster beyond

I have guessed what bothered you.

Firstly, you can exec kubectl get po -A -o wide, you will find two apiserver, one is your raw kube-apiserver, another is a karmada-apiserver installed by Karmada, so most karmada-config in document means the kubeconfig of karmada-apiserver.

After understanding the information above, you will known you should do two things:

1）get the kubeconfig of karmada-apiserver

following doc, the is one line:

kubectl get secret -n karmada-system karmada-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d

this command can print kubeconfig of karmada-apiserver, you can save to your local path like ~/.kube/karmada-apiserver.config

2）exec kubectl karmada join or karmadactl join

I accustomed to using karmadactl like:

# install latest verison karmadactl
curl -s https://raw.githubusercontent.com/karmada-io/karmada/master/hack/install-cli.sh | sudo bash

# join
karmadactl join member1 --kubeconfig ~/.kube/karmada-apiserver.config --karmada-context karmada-apiserver --cluster-kubeconfig ~/.kube/members.config --cluster-context member1

in side, member1 is the member cluster name, ~/.kube/karmada-apiserver.config is the kubeconfig of karmada-apiserver, members.config is the kubeconfig of member1 cluster.

however, the kubeconfig of karmada-apiserver generated by this helm installation may give you a bad experience, if you occurred problem, you can refer to #4013

chaosi-zju commented 1 year ago

@SeekerRook @13567436138 Hello, didn't you succeed later?

If not, you can refer to my new PR #4045, here is a install script for helm install, you need just execute:

hack/local-up-karmada-helm.sh

Note:

Feel free to contact me if you have any more questions~

SeekerRook commented 1 year ago

Hello again @chaosi-zju . I suspect I haave spotted the issue. after runnin kubectl get pods -o wide -A on the karmada host I get the following rersult:

NAMESPACE        NAME                                               READY   STATUS      RESTARTS         AGE   IP           NODE      NOMINATED NODE   READINESS GATES
karmada-system   karmada-post-install-9qxkf                         0/1     Completed   0                28d   <none>       karmada   <none>           <none>
kube-system      helm-install-traefik-87lkg                         0/1     Completed   1                28d   <none>       karmada   <none>           <none>
karmada-system   karmada-post-install-zhtsd                         0/1     Error       0                28d   <none>       karmada   <none>           <none>
kube-system      helm-install-traefik-crd-6q75w                     0/1     Completed   0                28d   <none>       karmada   <none>           <none>
karmada-system   karmada-post-install-6m5wb                         0/1     Error       0                28d   <none>       karmada   <none>           <none>
karmada-system   karmada-post-install-dwrzq                         0/1     Error       0                28d   <none>       karmada   <none>           <none>
kube-system      svclb-traefik-52d4b4ab-6ktch                       2/2     Running     14 (5m9s ago)    28d   10.42.0.92   karmada   <none>           <none>
karmada-system   karmada-scheduler-7c95d56449-qsp5f                 1/1     Running     6 (5d7h ago)     28d   10.42.0.97   karmada   <none>           <none>
kube-system      traefik-64f55bb67d-54j46                           1/1     Running     7 (5m10s ago)    28d   10.42.0.90   karmada   <none>           <none>
karmada-system   etcd-0                                             1/1     Running     7 (5m10s ago)    28d   10.42.0.95   karmada   <none>           <none>
kube-system      coredns-77ccd57875-jr422                           1/1     Running     7 (5m10s ago)    28d   10.42.0.88   karmada   <none>           <none>
kube-system      local-path-provisioner-957fdf8bc-j6jz2             1/1     Running     9 (4m28s ago)    28d   10.42.0.89   karmada   <none>           <none>
kube-system      metrics-server-648b5df564-8jfz5                    1/1     Running     9 (4m29s ago)    28d   10.42.0.87   karmada   <none>           <none>
karmada-system   karmada-apiserver-54d5f8f4df-9dh2j                 1/1     Running     10 (4m32s ago)   28d   10.42.0.91   karmada   <none>           <none>
karmada-system   karmada-aggregated-apiserver-86db44f594-6dtqc      1/1     Running     19 (4m14s ago)   28d   10.42.0.94   karmada   <none>           <none>
karmada-system   karmada-webhook-755fb4d6bb-8xmm7                   1/1     Running     18 (4m14s ago)   28d   10.42.0.93   karmada   <none>           <none>
karmada-system   karmada-kube-controller-manager-64f77857c4-snfnl   1/1     Running     22 (4m1s ago)    28d   10.42.0.96   karmada   <none>           <none>

The IPs of the Pods are the internal IPs of the cluster and not the external IP (as specified in #4013). Any ideas on how i can resolve it?

chaosi-zju commented 1 year ago

The IPs of the Pods are the internal IPs of the cluster and not the external IP (as specified in #4013). Any ideas on how i can resolve it?

Yes, default is internal IP, you can resolve by two methods:

  1. just like helm install karmada -n karmada-system --create-namespace --dependency-update --set apiServer.hostNetwork=true --kubeconfig ../remote/cluster-manager ./charts/karmada --debug, add parameter --set apiServer.hostNetwork=true

  2. edit charts/karmada/values.yaml, find apiserver.hostNetwork this line, update it to true, just like: image

two methods are of the same principle