openyurtio / openyurt

OpenYurt - Extending your native Kubernetes to edge(project under CNCF)
https://openyurt.io
Apache License 2.0
1.72k stars 404 forks source link

[BUG] dial tcp 127.0.0.1:10261: connect: connection refused #2129

Open TaibiaoGuo opened 3 months ago

TaibiaoGuo commented 3 months ago

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

# kubectl get pod -A
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-74dbdc644f-rp754    1/1     Running   0          10h
kube-system   calico-node-tlsx7                           1/1     Running   0          10h
kube-system   coredns-b7c47bcdc-ckvmr                     1/1     Running   0          10h
kube-system   coredns-b7c47bcdc-pnzkf                     1/1     Running   0          10h
kube-system   kube-apiserver-k8ace-gateway-sgp            1/1     Running   0          10h
kube-system   kube-controller-manager-k8ace-gateway-sgp   1/1     Running   0          10h
kube-system   kube-proxy-zktm6                            1/1     Running   0          10h
kube-system   kube-scheduler-k8ace-gateway-sgp            1/1     Running   0          10h
kube-system   nodelocaldns-cpqn7                          1/1     Running   0          10h
kube-system   raven-agent-ds-gqdp7                        1/1     Running   0          9h
kube-system   yurt-manager-58d55fb8b5-vrclk               1/1     Running   0          10h

Environment:

others

root@*****:/home/x/openyurt# _output/local/bin/linux/amd64/yurtadm  join *****:6443 --token=***** --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5
I0818 04:28:23.263614   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:23.263650   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc000040980] map[] map[token-bootstrap-client@kubernetes:0xc0002deff0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:23.473401   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
I0818 04:28:23.667918   20546 kubernetes.go:369] kubernetes version: v1.23.10
I0818 04:28:23.667933   20546 join.go:396] node join data info: join.joinData{cfgPath:"", joinNodeData:(*joindata.NodeRegistration)(0xc0002defa0), apiServerEndpoint:"*****:6443", token:"*****", tlsBootstrapCfg:(*api.Config)(0xc0001c72c0), clientSet:(*kubernetes.Clientset)(0xc0001b4d00), ignorePreflightErrors:sets.Set[string]{"DirAvailable--etc-kubernetes-manifests":sets.Empty{}, "FileAvailable--etc-kubernetes-kubelet.conf":sets.Empty{}}, organizations:"", pauseImage:"registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2", yurthubImage:"registry.cn-hangzhou.aliyuncs.com/openyurt/yurthub:latest", yurthubTemplate:"", yurthubManifest:"", kubernetesVersion:"v1.23.10", caCertHashes:[]string{}, nodeLabels:map[string]string{}, kubernetesResourceServer:"dl.k8s.io", yurthubServer:"127.0.0.1", reuseCNIBin:false, namespace:"kube-system", staticPodTemplateList:[]string(nil), staticPodManifestList:[]string(nil)}
I0818 04:28:23.722511   20546 system.go:43] Setting ipv4 forward
I0818 04:28:23.722545   20546 system.go:52] Setting bridge settings for kubernetes.
I0818 04:28:23.722632   20546 system.go:74] Disabling SELinux.
I0818 04:28:23.722637   20546 kubernetes.go:117] Check and install kubelet v1.23.10
I0818 04:28:23.757815   20546 kubernetes.go:126] kubelet --version: v1.23.10
I0818 04:28:23.757845   20546 kubernetes.go:138] Kubelet v1.23.10 already exist, skip install.
I0818 04:28:23.757853   20546 kubernetes.go:208] Check and install kubeadm v1.23.10
I0818 04:28:23.773324   20546 kubernetes.go:215] "kubeadm" version=<
        {
          "clientVersion": {
            "major": "1",
            "minor": "23",
            "gitVersion": "v1.23.10",
            "gitCommit": "7e54d50d3012cf3389e43b096ba35300f36e0817",
            "gitTreeState": "clean",
            "buildDate": "2022-08-17T18:31:47Z",
            "goVersion": "go1.17.13",
            "compiler": "gc",
            "platform": "linux/amd64"
          }
        }
 >
I0818 04:28:23.773379   20546 kubernetes.go:232] Kubeadm v1.23.10 already exist, skip install.
I0818 04:28:23.773392   20546 kubernetes.go:183] Skip download cni, use already exist file: /tmp/cni-plugins-linux-amd64-v0.8.0.tgz
I0818 04:28:24.150263   20546 kubernetes.go:258] Setting kubelet service.
I0818 04:28:24.153453   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:24.153466   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc0004fe080] map[] map[token-bootstrap-client@kubernetes:0xc0000fc0f0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:24.212160   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
I0818 04:28:24.212511   20546 yurthub.go:45] [join-node] Adding edge hub static yaml
I0818 04:28:24.212560   20546 yurthub.go:89] yurthub template: /etc/kubernetes/manifests/yurthub.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    openyurt.io/static-pod-hash: 74c4554785
  creationTimestamp: null
  labels:
    k8s-app: yurt-hub
  name: yurt-hub
  namespace: kube-system
spec:
  containers:
  - command:
    - yurthub
    - --v=2
    - --bind-address=127.0.0.1
    - --server-addr=https://*****:6443
    - --node-name=$(NODE_NAME)
    - --bootstrap-file=/var/lib/yurthub/bootstrap-hub.conf
    - --working-mode=edge
    - --namespace=kube-system
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: openyurt/yurthub:v1.4.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        host: 127.0.0.1
        path: /v1/healthz
        port: 10267
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    name: yurt-hub
    resources:
      limits:
        memory: 300Mi
      requests:
        cpu: 150m
        memory: 150Mi
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/yurthub
      name: hub-dir
    - mountPath: /etc/kubernetes
      name: kubernetes
  dnsPolicy: ClusterFirst
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 30
  volumes:
  - hostPath:
      path: /var/lib/yurthub
      type: DirectoryOrCreate
    name: hub-dir
  - hostPath:
      path: /etc/kubernetes
      type: Directory
    name: kubernetes
status: {}
I0818 04:28:24.212598   20546 yurthub.go:94] [join-node] Add hub agent static yaml is ok
I0818 04:28:24.212609   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:24.212615   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc0004fe900] map[] map[token-bootstrap-client@kubernetes:0xc0004fc5f0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:24.269986   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
[preflight] Running pre-flight checks
        [WARNING FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0818 04:28:30.333229   20582 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [*****]
W0818 04:28:30.335509   20582 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
error execution phase kubelet-start: error uploading crisocket: Get "http://127.0.0.1:10261/api/v1/nodes/*****?timeout=10s": dial tcp 127.0.0.1:10261: connect: connection refused
To see the stack trace of this error execute with --v=5 or higher
Error: exit status 1

/kind bug

rambohe-ch commented 3 months ago

@TaibiaoGuo Thank you for raising the issue. The address 127.0.0.1:10261 is listened by Yurthub component, please check the logs of Yurthub and find why yurthub can not startup correctly?

stale[bot] commented 2 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.