Closed Ssspade closed 4 months ago
@Ssspade Thanks for raising the issue.
I0319 09:28:14.179044 17976 upgrade.go:88] Create upgrade space success I0319 09:28:14.179247 17976 upgrade.go:106] Auto prepare upgrade manifest success I0319 09:28:14.179395 17976 upgrade.go:112] Auto upgrade backupManifest success I0319 09:28:14.179672 17976 upgrade.go:118] Auto upgrade replaceManifest success I0319 09:28:14.179721 17976 util.go:75] WaitForPodRunning namespace is kube-system, name is yurt-hub-node2 F0319 09:30:14.652184 17976 upgrade.go:48] could not upgrade static pod, timeout waiting for static pod kube-system/yurt-hub-node2 to be running
As the above logs, manifest file of YurtHub static pod has been replaced, but YurtHub static pod failed to restart. so would you like to check the following details:
check the YurtHub manifest file: /etc/kubernetes/manifests/yurthub.yaml, the contents of this file is upgraded or not?
dump the logs of kubelet component(journalctl -u kubelet &> kubelet.log), and check the reason why YurtHub static pod failed to restart.
Thanks for your attention. I would like to ask, what is the purpose of "yss-upgrade-worker-yurt-hub-xxx"?
When I first modified the yurthub.yaml file and placed it in the directory /etc/kubernetes/manifests/, it started to create and run. However, due to incorrect modifications in my yurthub.yaml file, yurt-hub-node2 could not run properly. Therefore, I deleted the yurthub.yaml file, re-modified it, and placed it back in the directory /etc/kubernetes/manifests/. After doing so, yurt-hub-node2 ran successfully, but "yss-upgrade-worker-" encountered an error.
I manually deleted the pod "yss-upgrade-worker-" to restart it, but upon checking the kubelet logs, I did not find any relevant logs, neither on the node nor on the master.
Sorry, my previous reproduction method was a bit flawed. I deleted yurthub.yaml and recopied it to the directory /etc/kubernetes/manifests/. The logs for yurt-hub-xxx are as follows:
I0320 09:32:53.103470 1 util.go:289] start proxying: get /apis/apps.openyurt.io/v1beta1/nodepools?limit=500&resourceVersion=0, in flight requests: 1
I0320 09:32:53.103765 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca-bundle::/var/lib/yurthub/pki/ca.crt" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-03-12 08:43:04 +0000 UTC to 2034-03-10 08:43:04 +0000 UTC (now=2024-03-20 09:32:53.10367724 +0000 UTC))"
I0320 09:32:53.104143 1 util.go:289] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 2
I0320 09:32:53.104455 1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/var/lib/yurthub/pki/yurthub-server-current.pem::/var/lib/yurthub/pki/yurthub-server-current.pem" certDetail="\"system:node:node1\" [serving] groups=[system:nodes] validServingFor=[169.254.2.1,127.0.0.1] issuer=\"kubernetes\" (2024-03-20 07:46:26 +0000 UTC to 2025-03-20 07:46:26 +0000 UTC (now=2024-03-20 09:32:53.10435406 +0000 UTC))"
I0320 09:32:53.104524 1 secure_serving.go:200] Serving securely on 169.254.2.1:10268
I0320 09:32:53.104601 1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/var/lib/yurthub/pki/yurthub-server-current.pem::/var/lib/yurthub/pki/yurthub-server-current.pem"
I0320 09:32:53.104678 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0320 09:32:53.104699 1 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/yurthub/pki/ca.crt"
E0320 09:32:53.105635 1 gc.go:181] could not list keys for kubelet events, specified key is not found
E0320 09:32:53.105669 1 gc.go:181] could not list keys for kube-proxy events, specified key is not found
I0320 09:32:53.106652 1 util.go:289] start proxying: get /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dyurt-hub-cfg&limit=500&resourceVersion=0, in flight requests: 3
I0320 09:32:53.108302 1 util.go:248] yurthub list configmaps: /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dyurt-hub-cfg&limit=500&resourceVersion=0 with status code 200, spent 1.50208ms
I0320 09:32:53.110921 1 approver.go:191] current filter setting: map[coredns/endpoints/list:map[servicetopology:{}] coredns/endpoints/watch:map[servicetopology:{}] coredns/endpointslices/list:map[servicetopology:{}] coredns/endpointslices/watch:map[servicetopology:{}] kube-proxy/endpoints/list:map[servicetopology:{}] kube-proxy/endpoints/watch:map[servicetopology:{}] kube-proxy/endpointslices/list:map[servicetopology:{}] kube-proxy/endpointslices/watch:map[servicetopology:{}] kube-proxy/services/list:map[discardcloudservice:{} nodeportisolation:{}] kube-proxy/services/watch:map[discardcloudservice:{} nodeportisolation:{}] kubelet/configmaps/get:map[inclusterconfig:{}] kubelet/configmaps/list:map[inclusterconfig:{}] kubelet/configmaps/watch:map[inclusterconfig:{}] kubelet/pods/list:map[hostnetworkpropagation:{}] kubelet/pods/watch:map[hostnetworkpropagation:{}] kubelet/services/list:map[masterservice:{}] kubelet/services/watch:map[masterservice:{}] nginx-ingress-controller/endpoints/list:map[servicetopology:{}] nginx-ingress-controller/endpoints/watch:map[servicetopology:{}] nginx-ingress-controller/endpointslices/list:map[servicetopology:{}] nginx-ingress-controller/endpointslices/watch:map[servicetopology:{}]] after add
I0320 09:32:53.111522 1 util.go:289] start proxying: get /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dyurt-hub-cfg&resourceVersion=476411&timeout=5m32s&timeoutSeconds=332&watch=true, in flight requests: 3
I0320 09:32:53.206474 1 util.go:248] yurthub list nodepools: /apis/apps.openyurt.io/v1beta1/nodepools?limit=500&resourceVersion=0 with status code 200, spent 102.901ms
I0320 09:32:53.206547 1 serializer.go:200] schema.GroupVersionResource{Group:"apps.openyurt.io", Version:"v1beta1", Resource:"nodepools"} is not found in client-go runtime scheme
I0320 09:32:53.207393 1 util.go:289] start proxying: get /apis/apps.openyurt.io/v1beta1/nodepools?allowWatchBookmarks=true&resourceVersion=476486&timeoutSeconds=578&watch=true, in flight requests: 3
I0320 09:32:53.208631 1 util.go:248] yurthub list services: /api/v1/services?limit=500&resourceVersion=0 with status code 200, spent 104.39578ms
I0320 09:32:53.212136 1 util.go:289] start proxying: get /api/v1/services?allowWatchBookmarks=true&resourceVersion=476878&timeout=9m56s&timeoutSeconds=596&watch=true, in flight requests: 3
E0320 09:32:53.538807 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:32:53.538887 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:32:58.194780 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:32:58.194836 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:03.757531 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:03.757566 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:08.020620 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:08.020678 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:13.837857 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:13.837916 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:18.661483 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:18.661539 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:23.089416 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:23.089471 1 ota.go:66] Get pod list failed, specified key is not found
E0320 09:33:28.518469 1 storage_wrapper.go:141] could not list objects for kubelet/pods.v1.core, specified key is not found
E0320 09:33:28.518503 1 ota.go:66] Get pod list failed, specified key is not found
The Edge node kubelet logs are as follows:
Mar 20 09:33:25 node1 kubelet[393]: E0320 09:33:25.510726 393 controller.go:187] failed to update lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "node1": the object has been modified; please apply your changes to the latest version and try again
Mar 20 09:33:31 node1 kubelet[393]: E0320 09:33:31.201961 393 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 fdae:7c79:ff46::1"
Mar 20 09:33:35 node1 kubelet[393]: E0320 09:33:35.859105 393 controller.go:187] failed to update lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "node1": the object has been modified; please apply your changes to the latest version and try again
Mar 20 09:33:45 node1 kubelet[393]: E0320 09:33:45.374661 393 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 fdae:7c79:ff46::1"
Mar 20 09:33:47 node1 kubelet[393]: E0320 09:33:47.522655 393 controller.go:187] failed to update lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "node1": the object has been modified; please apply your changes to the latest version and try again
Mar 20 09:33:57 node1 kubelet[393]: E0320 09:33:57.623901 393 controller.go:187] failed to update lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "node1": the object has been modified; please apply your changes to the latest version and try again
Mar 20 09:34:07 node1 kubelet[393]: E0320 09:34:07.951674 393 controller.go:187] failed to update lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "node1": the object has been modified; please apply your changes to the latest version and try again
The normally running yurt-hub is re-created through kubelet after the failure of yss-upgrade-worker.
@Ssspade It seems that Yurthub component have not worked correctly. would you like to show me the fully Yurthub logs?
During the execution of yss-upgrade, the logs for yurt-hub are recorded in the file named "yurt-hub-during-yss.log".
yurt-hub-during-yss.log After 2 minutes, if yss-upgrade fails, yurt-hub will restart and its logs will be written to the file named "yurt-hub.log".
During the execution of yss-upgrade, the logs for yurt-hub are recorded in the file named "yurt-hub-during-yss.log".
yurt-hub-during-yss.log After 2 minutes, if yss-upgrade fails, yurt-hub will restart and its logs will be written to the file named "yurt-hub.log".
@Ssspade It seems that yurt-manager component is not installed correctly, because NodePool CRD is not found in your cluster.
In your logs file, there are some errors about NodePool resource as following:
I0326 02:14:43.771606 1 serializer.go:200] schema.GroupVersionResource{Group:"apps.openyurt.io", Version:"v1beta1", Resource:"nodepools"} is not found in client-go runtime scheme
@Ssspade Would you like to upload the detail metrics about yurthub component?
you can get the metrics on the node by execute the command: curl http://127.0.0.1:10267/metrics
Of course, I'm very willing to. The installation process, I referred to the official websitev1.4。 curl http://127.0.0.1:10267/metrics,log: curl_metric.log
During the execution of yss-upgrade, the logs for yurt-hub are recorded in the file named "yurt-hub-during-yss.log".
yurt-hub-during-yss.log After 2 minutes, if yss-upgrade fails, yurt-hub will restart and its logs will be written to the file named "yurt-hub.log".
yurt-hub.log
@Ssspade It seems that yurt-manager component is not installed correctly, because NodePool CRD is not found in your cluster.
In your logs file, there are some errors about NodePool resource as following:
I0326 02:14:43.771606 1 serializer.go:200] schema.GroupVersionResource{Group:"apps.openyurt.io", Version:"v1beta1", Resource:"nodepools"} is not found in client-go runtime scheme
I use the command to view it:
root@master:~# kubectl get crd
NAME CREATED AT
gateways.raven.openyurt.io 2024-03-21T09:29:27Z
nodepools.apps.openyurt.io 2024-03-21T09:29:26Z
platformadmins.iot.openyurt.io 2024-03-21T09:29:26Z
yurtappdaemons.apps.openyurt.io 2024-03-21T09:29:26Z
yurtappoverriders.apps.openyurt.io 2024-03-21T09:29:26Z
yurtappsets.apps.openyurt.io 2024-03-21T09:29:26Z
yurtstaticsets.apps.openyurt.io 2024-03-21T09:29:26Z
Of course, I'm very willing to. The installation process, I referred to the official websitev1.4。 curl http://127.0.0.1:10267/metrics,log: curl_metric.log
@Ssspade From the metrics of yurthub, only yurthub itself send reqeusts to kube-apiserver through yurthub, and other components have not sent requests to kube-apiserver through yurthub.
node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="watch"} 1
Have you used yurtadm join
to add node into this cluster? or use kubeadm join
command to add node?
During the execution of yss-upgrade, the logs for yurt-hub are recorded in the file named "yurt-hub-during-yss.log".
yurt-hub-during-yss.log After 2 minutes, if yss-upgrade fails, yurt-hub will restart and its logs will be written to the file named "yurt-hub.log".
yurt-hub.log
@Ssspade It seems that yurt-manager component is not installed correctly, because NodePool CRD is not found in your cluster. In your logs file, there are some errors about NodePool resource as following:
I0326 02:14:43.771606 1 serializer.go:200] schema.GroupVersionResource{Group:"apps.openyurt.io", Version:"v1beta1", Resource:"nodepools"} is not found in client-go runtime scheme
I use the command to view it:
root@master:~# kubectl get crd NAME CREATED AT gateways.raven.openyurt.io 2024-03-21T09:29:27Z nodepools.apps.openyurt.io 2024-03-21T09:29:26Z platformadmins.iot.openyurt.io 2024-03-21T09:29:26Z yurtappdaemons.apps.openyurt.io 2024-03-21T09:29:26Z yurtappoverriders.apps.openyurt.io 2024-03-21T09:29:26Z yurtappsets.apps.openyurt.io 2024-03-21T09:29:26Z yurtstaticsets.apps.openyurt.io 2024-03-21T09:29:26Z
@Ssspade yes, from the metrics of yurthub, it seems that you installed yurthub before yurt-manager installation. would you like to tell me how do you add your node into cluster?
Of course, I'm very willing to. The installation process, I referred to the official websitev1.4。 curl http://127.0.0.1:10267/metrics,log: curl_metric.log
@Ssspade From the metrics of yurthub, only yurthub itself send reqeusts to kube-apiserver through yurthub, and other components have not sent requests to kube-apiserver through yurthub.
node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="watch"} 1
Have you used
yurtadm join
to add node into this cluster? or usekubeadm join
command to add node?
I have used kubeadm join
to add node. The way to install openyurt is on Kubernetes. I haven't completed the "Join Nodes-Configure Kubelet (2.3)" step yet.
@rambohe-ch My OpenYurt was installed on top of an existing Kubernetes cluster and node. Firstly, I followed the installation guide to install the master node, and then, according to the node joining guide(2. Install OpenYurt node components), I installed the OpenYurt Node component on the existing Kubernetes nodes.
If I disregard errors from the yss-upgrade
component and proceed with the "Join Nodes" step. curl http://127.0.0.1:10267/metrics log is curl-metric.log.
Additionally, my physical environment consists of an x86 master node and ARM64 edge nodes. I'm unsure if this might have any impact.
I made the following modifications, and the static pod upgrade function is running normally: (1) Downgraded Kubernetes version: v1.23.0 to v1.22.11 (2) Downgraded Flannel version: latest to v0.18.1 Thank you for your assistance.
All other Pods are running, with only yss-upgrade-worker-yurt-hub.
default nginx-node2-7cb6c5fdbc-7zzq6 1/1 Running 0 32m
kube-flannel kube-flannel-ds-cj8z2 1/1 Running 0 32m
kube-system coredns-wbpwh 1/1 Running 0 32m
kube-system kube-proxy-86g8d 1/1 Running 0 32m
kube-system raven-agent-ds-5qhxk 1/1 Running 2 (52m ago) 98m
kube-system yss-upgrade-worker-yurt-hub-node2-65d4cdf959 0/1 Error 0
kube-system yurt-hub-node2 1/1 Running 0 31m
kube-system yss-upgrade-worker-yurt-hub-node2-65d4cdf959 0/1 Error 0
logs: I0319 09:28:14.178337 17976 upgrade.go:35] FLAG: --hash="65d4cdf959" I0319 09:28:14.178637 17976 upgrade.go:35] FLAG: --help="false" I0319 09:28:14.178662 17976 upgrade.go:35] FLAG: --kubeconfig="" I0319 09:28:14.178673 17976 upgrade.go:35] FLAG: --manifest="yurthub" I0319 09:28:14.178681 17976 upgrade.go:35] FLAG: --mode="AdvancedRollingUpdate" I0319 09:28:14.178690 17976 upgrade.go:35] FLAG: --name="yurt-hub-node2" I0319 09:28:14.178698 17976 upgrade.go:35] FLAG: --namespace="kube-system" I0319 09:28:14.178706 17976 upgrade.go:35] FLAG: --timeout="2m0s" I0319 09:28:14.179044 17976 upgrade.go:88] Create upgrade space success I0319 09:28:14.179247 17976 upgrade.go:106] Auto prepare upgrade manifest success I0319 09:28:14.179395 17976 upgrade.go:112] Auto upgrade backupManifest success I0319 09:28:14.179672 17976 upgrade.go:118] Auto upgrade replaceManifest success I0319 09:28:14.179721 17976 util.go:75] WaitForPodRunning namespace is kube-system, name is yurt-hub-node2 F0319 09:30:14.652184 17976 upgrade.go:48] could not upgrade static pod, timeout waiting for static pod kube-system/yurt-hub-node2 to be running goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:1026 +0x90 k8s.io/klog/v2.(loggingT).output(0x1be27e0, 0x3, {0x0, 0x0}, 0x400024c930, 0x0, {0x14bebdd?, 0x40001140f0?}, 0x0?, 0x0) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:975 +0x5f4 k8s.io/klog/v2.(loggingT).printf(0x400024fb80?, 0x109bbf0?, {0x0, 0x0}, {0x0, 0x0}, {0xfe1316, 0x20}, {0x40001140f0, 0x1, ...}) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:753 +0x184 k8s.io/klog/v2.Fatalf(...) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:1514 github.com/openyurtio/openyurt/cmd/yurt-node-servant/static-pod-upgrade.NewUpgradeCmd.func1(0x400007ef00?, {0x4000516690?, 0x5?, 0x5?}) /build/cmd/yurt-node-servant/static-pod-upgrade/upgrade.go:48 +0x258 github.com/spf13/cobra.(Command).execute(0x400007ef00, {0x4000516640, 0x5, 0x5}) /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:944 +0x5f4 github.com/spf13/cobra.(Command).ExecuteC(0x400007e000) /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x368 github.com/spf13/cobra.(*Command).Execute(...) /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 main.main() /build/cmd/yurt-node-servant/node-servant.go:53 +0x2dc
goroutine 34 [chan receive, 2 minutes]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0?) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:1169 +0x60 created by k8s.io/klog/v2.init.0 /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:420 +0x150
Environment:
OpenYurt version: helm search repo openyurt NAME CHART VERSION APP VERSION DESCRIPTION
openyurt/openyurt 1.2.1 1.2.1 A Helm chart for openyurt
openyurt/pool-coordinator 1.3.1 1.3.0 A Helm chart for OpenYurt pool-coordinator comp... openyurt/raven-agent 0.4.1 0.4.1 A Helm chart for Kubernetes
openyurt/yurt-app-manager 0.1.3 0.6.0 The controller manager that contains the nodepo... openyurt/yurt-coordinator 1.4.0 1.4.0 A Helm chart for OpenYurt yurt-coordinator comp... openyurt/yurt-dashboard 0.1.1 0.7.0 Dashboard for openyurt
openyurt/yurt-manager 1.4.3 1.4.0 A Helm chart for OpenYurt yurt-manager component. openyurt/yurthub 1.4.1 1.4.0 A Helm chart for OpenYurt YurtHub Component
Kubernetes version (use
kubectl version
): 1.23.0OS (e.g:
cat /etc/os-release
): ubuntuKernel (e.g.
uname -a
): 5.10others
/kind bug