search

easzlab / kubeasz

使用Ansible脚本安装K8S集群，介绍组件交互原理，方便直接，不受国内网络环境影响
https://github.com/easzlab/kubeasz
10.51k stars 3.52k forks source link

kubeasz3.6.2 安装kubernetes 1.25.13报错 #1321

Closed Rabbit-st closed 1 year ago

Rabbit-st commented 1 year ago

What happened? 发生了什么问题？

kube-apiserver报错：

# journalctl -xe -u kube-apiserver
......
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.747833   42714 healthz.go:257] informer-sync,poststarthook/priority-and-fairness-config-producer check failed: readyz
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]informer-sync failed: 2 informers not started yet: [*v1beta2.PriorityLevelConfiguration *v1beta2.FlowSchema]
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]poststarthook/priority-and-fairness-config-producer failed: not finished
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.847461   42714 shared_informer.go:278] stop requested
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.847531   42714 shared_informer.go:278] stop requested
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.847593   42714 healthz.go:257] informer-sync,poststarthook/priority-and-fairness-config-producer check failed: readyz
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]informer-sync failed: 2 informers not started yet: [*v1beta2.FlowSchema *v1beta2.PriorityLevelConfiguration]
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]poststarthook/priority-and-fairness-config-producer failed: not finished
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: W1010 17:33:30.848003   42714 reflector.go:424] storage/cacher.go:/prioritylevelconfigurations: failed to list *flowcontrol.PriorityLevelConfiguration: no kind "PriorityLevelConfiguration" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: E1010 17:33:30.848026   42714 cacher.go:440] cacher (*flowcontrol.PriorityLevelConfiguration): unexpected ListAndWatch error: failed to list *flowcontrol.PriorityLevelConfiguration: no kind "PriorityLevelConfiguration" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"; reinitializing...
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: W1010 17:33:30.848317   42714 reflector.go:424] storage/cacher.go:/flowschemas: failed to list *flowcontrol.FlowSchema: no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: E1010 17:33:30.848333   42714 cacher.go:440] cacher (*flowcontrol.FlowSchema): unexpected ListAndWatch error: failed to list *flowcontrol.FlowSchema: no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"; reinitializing...
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: W1010 17:33:30.848732   42714 reflector.go:424] storage/cacher.go:/prioritylevelconfigurations: failed to list *flowcontrol.PriorityLevelConfiguration: no kind "PriorityLevelConfiguration" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: E1010 17:33:30.848750   42714 cacher.go:440] cacher (*flowcontrol.PriorityLevelConfiguration): unexpected ListAndWatch error: failed to list *flowcontrol.PriorityLevelConfiguration: no kind "PriorityLevelConfiguration" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"; reinitializing...
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: W1010 17:33:30.849072   42714 reflector.go:424] storage/cacher.go:/flowschemas: failed to list *flowcontrol.FlowSchema: no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: E1010 17:33:30.849087   42714 cacher.go:440] cacher (*flowcontrol.FlowSchema): unexpected ListAndWatch error: failed to list *flowcontrol.FlowSchema: no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta3" in scheme "pkg/api/legacyscheme/scheme.go:30"; reinitializing...
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.947412   42714 shared_informer.go:278] stop requested
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.947428   42714 shared_informer.go:278] stop requested
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: I1010 17:33:30.947467   42714 healthz.go:257] informer-sync,poststarthook/priority-and-fairness-config-producer check failed: readyz
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]informer-sync failed: 2 informers not started yet: [*v1beta2.PriorityLevelConfiguration *v1beta2.FlowSchema]
Oct 10 17:33:30 kubernetes-master-0001 kube-apiserver[42714]: [-]poststarthook/priority-and-fairness-config-producer failed: not finished

kubelet的报错：

Oct 10 17:30:42 kubernetes-master-0001 kubelet[34242]: E1010 17:30:42.108619   34242 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 10 17:30:47 kubernetes-master-0001 kubelet[34242]: E1010 17:30:47.109038   34242 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259158   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259187   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259201   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259302   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259312   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259475   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259591   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259714   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259822   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.259931   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.260041   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738416   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738452   34242 reflector.go:347] vendor/k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: vendor/k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738477   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738489   34242 reflector.go:347] vendor/k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: vendor/k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738504   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738516   34242 reflector.go:347] object-"kube-system"/"calico-etcd-secrets": watch of *v1.Secret ended with: very short watch: object-"kube-system"/"calico-etcd-secrets": Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738529   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738539   34242 reflector.go:347] pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738555   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738566   34242 reflector.go:347] object-"kube-system"/"kube-dns": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"kube-dns": Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738577   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738587   34242 reflector.go:347] object-"kube-system"/"node-local-dns": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"node-local-dns": Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738602   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738614   34242 reflector.go:347] vendor/k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: vendor/k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738626   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738636   34242 reflector.go:347] object-"kube-system"/"kube-root-ca.crt": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"kube-root-ca.crt": Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738648   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738660   34242 reflector.go:347] vendor/k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: vendor/k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738671   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738682   34242 reflector.go:347] object-"kube-system"/"calico-config": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"calico-config": Unexpected watch close - watch lasted less than a second and no items received
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: I1010 17:30:50.738693   34242 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 10 17:30:50 kubernetes-master-0001 kubelet[34242]: W1010 17:30:50.738705   34242 reflector.go:347] object-"kube-system"/"kubernetes-services-endpoint": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"kubernetes-services-endpoint": Unexpected watch close - watch lasted less than a second and no items received

What did you expect to happen? 期望的结果是什么？

正常

How can we reproduce it (as minimally and precisely as possible)? 尽可能最小化、精确地描述如何复现问题

kubeasz3.6.2安装kubernetes 1.25.13 kube-apiserver异常。 复现方法： 下载1.25.13版本：

./ezdown -D -k v1.25.13

创建集群：

docker exec -it kubeasz ezctl new k8s-cluster-test

config.yml里k8s版本改为1.25.13

K8S_VER: "1.25.13"

只需安装前面四步：

$ dk ezctl setup k8s-cluster-test 01
$ dk ezctl setup k8s-cluster-test 02
$ dk ezctl setup k8s-cluster-test 03
$ dk ezctl setup k8s-cluster-test 04

查看节点状态：

kubectl get nodes

可以发现集群异常。

Anything else we need to know? 其他需要说明的情况

安装kubernetes 1.28.1版本都正常

Kubernetes version k8s 版本

Kubernetes v1.25.13

Kubeasz version

3.6.2

OS version 操作系统版本

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy $ uname -a Linux kubernetes-master-0001 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ```

Related plugins (CNI, CSI, ...) and versions (if applicable) 其他网络插件等需要说明的情况

Rabbit-st commented 1 year ago

找到原因了，重装系统后，数据盘数据还保留导致的