Why can't Karmada apiserver start

[Schwarao]

log： describe： ![Uploading image.png…]()

[Schwarao]

describe：

[Schwarao]

Cluster created using kubeadm

[Schwarao]

This is my initialization command：

sudo KUBECONFIG=/home/rongqigang/.kube/config kubectl karmada init --kubeconfig=/home/rongqigang/.kube/config \ --etcd-init-image=alpine:3.19.2 \ --karmada-aggregated-apiserver-image=karmada/karmada-aggregated-apiserver:v1.10.1 \ --karmada-controller-manager-image=karmada/karmada-controller-manager:v1.10.1 \ --karmada-scheduler-image=karmada/karmada-scheduler:v1.10.1 \ --karmada-webhook-image=karmada/karmada-webhook:v1.10.1 \ --kube-image-registry registry.cn-hangzhou.aliyuncs.com/google_containers \ --crds crds.tar.gz

[RainbowMango]

cc @chaosi-zju

[chaosi-zju]

hi @Schwarao, could you please provided full karmada-apiserver logs?

if current pod logs is not complete, maybe you can use kubectl logs -p karmada-apiserver-xxx -n karmada-system to fetch previous containter logs~

[chaosi-zju]

besides, can you check the version of kubectl karmada~

[Schwarao]

This is the complete log：

[Schwarao]

此外，你能检查一下kubectl karmada~ May I ask what command to use

[chaosi-zju]

May I ask what command to use

kubectl karmada version

[chaosi-zju]

if current pod logs is not complete, maybe you can use kubectl logs -p karmada-apiserver-xxx -n karmada-system to fetch previous containter logs~

can you use -p parameter to print logs? just like: kubectl logs -p karmada-apiserver-xxx -n karmada-system

[chaosi-zju]

run.go:74] "command failed" err="context deadline exceeded"

This error is most likely because karmada-apiserver cannot connect to etcd-0, we may need more infomation:

1. use kubectl describe po karmada-apiserver-xxx -n karmada-system to get the Containers.karmada-apiserver.Command field
2. check what is etcd-server address and whether/why etcd-server is not connected.

[Schwarao]

here

[Schwarao]

kubectl describe po karmada-apiserver-8dfd9bcd7-8k5g4 -n karmada-system：

Name: karmada-apiserver-8dfd9bcd7-8k5g4 Namespace: karmada-system Priority: 0 Service Account: default Node: k8s-master01/10.240.10.70 Start Time: Thu, 27 Jun 2024 10:47:16 +0800 Labels: app=karmada-apiserver pod-template-hash=8dfd9bcd7 Annotations: cni.projectcalico.org/containerID: acbea277dc4fd1c791b852e83e97d63319e3460fdefd85cad27410af86322645 cni.projectcalico.org/podIP: 10.0.32.135/32 cni.projectcalico.org/podIPs: 10.0.32.135/32 Status: Running IP: 10.0.32.135 IPs: IP: 10.0.32.135 Controlled By: ReplicaSet/karmada-apiserver-8dfd9bcd7 Containers: karmada-apiserver: Container ID: docker://8e9833a5899d34b2107e426ab17dbba78f971d894476999479bab35972b2bf29 Image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.11 Image ID: docker-pullable://registry.aliyuncs.com/google_containers/kube-apiserver@sha256:24262f37a760fdee1970f6b7f496ee5189774af38b0812a289e67518aaf32243 Port: 5443/TCP Host Port: 0/TCP Command: kube-apiserver --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/karmada/pki/ca.crt --enable-bootstrap-token-auth=true --etcd-cafile=/etc/karmada/pki/etcd-ca.crt --etcd-certfile=/etc/karmada/pki/etcd-client.crt --etcd-keyfile=/etc/karmada/pki/etcd-client.key --etcd-servers=https://etcd-0.etcd.karmada-system.svc.cluster.local:2379 --bind-address=0.0.0.0 --kubelet-client-certificate=/etc/karmada/pki/karmada.crt --kubelet-client-key=/etc/karmada/pki/karmada.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --disable-admission-plugins=StorageObjectInUseProtection,ServiceAccount --runtime-config= --apiserver-count=1 --secure-port=5443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/karmada/pki/karmada.key --service-account-signing-key-file=/etc/karmada/pki/karmada.key --service-cluster-ip-range=10.96.0.0/12 --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client.crt --proxy-client-key-file=/etc/karmada/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --tls-cert-file=/etc/karmada/pki/apiserver.crt --tls-private-key-file=/etc/karmada/pki/apiserver.key --tls-min-version=VersionTLS13 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 27 Jun 2024 18:19:50 +0800 Finished: Thu, 27 Jun 2024 18:20:10 +0800 Ready: False Restart Count: 93 Liveness: http-get https://:5443/livez delay=15s timeout=5s period=30s #success=1 #failure=3 Readiness: http-get https://:5443/readyz delay=0s timeout=5s period=30s #success=1 #failure=3 Environment: Mounts: /etc/karmada/pki from karmada-cert (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: karmada-cert: Type: Secret (a volume populated by a Secret) SecretName: karmada-cert Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: :NoExecute op=Exists Events: Type Reason Age From Message

---

Warning Unhealthy 28m (x208 over 7h33m) kubelet Readiness probe failed: Get "https://10.0.32.135:5443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning BackOff 3m4s (x1964 over 7h32m) kubelet Back-off restarting failed container karmada-apiserver in pod karmada-apiserver-8dfd9bcd7-8k5g4_karmada-system(5469299d-9754-410e-9b62-1b471b150b20) [rongqigang@k8s-master01:~/calico_image]$

[Schwarao]

More information:

endpoints ：

etcd svc：

[Schwarao]

May I ask if Kubernetes 1.30.2 is supported @chaosi-zju

[chaosi-zju]

May I ask if Kubernetes 1.30.2 is supported

I tested replace karmada-apiserver image to v1.30.2, it runs ok~

$ kubectl get deploy karmada-apiserver -n karmada-system -o yaml | grep -C 3 image:
        - --tls-cert-file=/etc/karmada/pki/apiserver.crt
        - --tls-private-key-file=/etc/karmada/pki/apiserver.key
        - --tls-min-version=VersionTLS13
        image: registry.k8s.io/kube-apiserver:v1.30.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
$ kubectl get pods -o wide -n karmada-system | grep karmada-apiserver     
karmada-apiserver-6d698678df-qfbsz                     1/1     Running   0          2m35s   172.18.0.4    karmada-host-control-plane   <none>           <none>

[Schwarao]

May I ask why I reported an error here?

[Schwarao]

Do we just need to change this to 1.30.2

[chaosi-zju]

May I ask why I reported an error here?

This should be a problem with your own kubernetes container network. There is a service named etcd in your environment, but it cannot do dns resolution on serviceName in the container.

[chaosi-zju]

Do we just need to change this to 1.30.2

You can, but it will not make any difference~

You karmada-apiserver installed failed is because it cann't connect to etcd by serviceName.

However, the reason for can not connect to etcd by serviceName is because there may be some problem with your own kubernetes container network, wihch result in dns resolution failure.

---

Maybe you can a test to check whether your kubernetes container network has problem:

• deploy a simple nginx deployment and a service
• deploy another simple ubuntu deployment and enter the container of ubuntu pod and check whether you can connect to nginx service from ubuntu container by nginx service name~