Closed transparency066 closed 1 year ago
rambohe-ch
commented
1 year ago @transparency066 It seems that yurthub can not connect to kube-apiserver, and you can check the server health status in the metrics of yurthub by the command: curl http://127.0.0.1:10267/metrics.
and you should make sure that yurthub can connect to kube-apiserver when you re-configures yurthub.
transparency066
commented
1 year ago go_gc_duration_seconds{quantile="0"} 2.8984e-05 go_gc_duration_seconds{quantile="0.25"} 7.3124e-05 go_gc_duration_seconds{quantile="0.5"} 0.000138523 go_gc_duration_seconds{quantile="0.75"} 0.000240081 go_gc_duration_seconds{quantile="1"} 0.000700298 go_gc_duration_seconds_sum 0.096005456 go_gc_duration_seconds_count 547
go_goroutines 80
go_info{version="go1.18.10"} 1
go_memstats_alloc_bytes 1.1164184e+07
go_memstats_alloc_bytes_total 1.555998208e+09
go_memstats_buck_hash_sys_bytes 1.61637e+06
go_memstats_frees_total 1.8244247e+07
go_memstats_gc_sys_bytes 5.657504e+06
go_memstats_heap_alloc_bytes 1.1164184e+07
go_memstats_heap_idle_bytes 3.178496e+06
go_memstats_heap_inuse_bytes 1.2222464e+07
go_memstats_heap_objects 95191
go_memstats_heap_released_bytes 2.654208e+06
go_memstats_heap_sys_bytes 1.540096e+07
go_memstats_last_gc_time_seconds 1.6872285868090148e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 1.8339438e+07
go_memstats_mcache_inuse_bytes 4800
go_memstats_mcache_sys_bytes 15600
go_memstats_mspan_inuse_bytes 192528
go_memstats_mspan_sys_bytes 195264
go_memstats_next_gc_bytes 1.1477712e+07
go_memstats_other_sys_bytes 724678
go_memstats_stack_inuse_bytes 1.376256e+06
go_memstats_stack_sys_bytes 1.376256e+06
go_memstats_sys_bytes 2.4986632e+07
go_threads 12
node_yurthub_closable_conns_collector{server="100.80.241.43:6443"} 0
node_yurthub_in_flight_requests_collector{client="go-http-client",resource="csidrivers",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="csinodes",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="leases",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="create"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="pods",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="runtimeclasses",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="create"} 0 node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="patch"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_total 3
node_yurthub_proxy_latency_collector{client="go-http-client",resource="csidrivers",subresources="",type="apiserver_latency",verb="list"} 118189 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csidrivers",subresources="",type="full_latency",verb="list"} 186552 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csinodes",subresources="",type="apiserver_latency",verb="get"} 234323 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csinodes",subresources="",type="full_latency",verb="get"} 441823 node_yurthub_proxy_latency_collector{client="go-http-client",resource="leases",subresources="",type="apiserver_latency",verb="get"} 99815 node_yurthub_proxy_latency_collector{client="go-http-client",resource="leases",subresources="",type="full_latency",verb="get"} 159074 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="create"} 22646 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="get"} 86148 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="list"} 327299 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="create"} 36670 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="get"} 104780 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="list"} 402995 node_yurthub_proxy_latency_collector{client="go-http-client",resource="pods",subresources="",type="apiserver_latency",verb="get"} 118223 node_yurthub_proxy_latency_collector{client="go-http-client",resource="pods",subresources="",type="full_latency",verb="get"} 139521 node_yurthub_proxy_latency_collector{client="go-http-client",resource="runtimeclasses",subresources="",type="apiserver_latency",verb="list"} 86546 node_yurthub_proxy_latency_collector{client="go-http-client",resource="runtimeclasses",subresources="",type="full_latency",verb="list"} 133640 node_yurthub_proxy_latency_collector{client="go-http-client",resource="services",subresources="",type="apiserver_latency",verb="list"} 329371 node_yurthub_proxy_latency_collector{client="go-http-client",resource="services",subresources="",type="full_latency",verb="list"} 461140 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="apiserver_latency",verb="create"} 70005 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="apiserver_latency",verb="patch"} 461076 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="full_latency",verb="create"} 114781 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="full_latency",verb="patch"} 523876 node_yurthub_proxy_latency_collector{client="yurthub",resource="configmaps",subresources="",type="apiserver_latency",verb="list"} 726094 node_yurthub_proxy_latency_collector{client="yurthub",resource="configmaps",subresources="",type="full_latency",verb="list"} 744325 node_yurthub_proxy_latency_collector{client="yurthub",resource="nodepools",subresources="",type="apiserver_latency",verb="list"} 102808 node_yurthub_proxy_latency_collector{client="yurthub",resource="nodepools",subresources="",type="full_latency",verb="list"} 134168 node_yurthub_proxy_latency_collector{client="yurthub",resource="services",subresources="",type="apiserver_latency",verb="list"} 3.371083e+06 node_yurthub_proxy_latency_collector{client="yurthub",resource="services",subresources="",type="full_latency",verb="list"} 3.390693e+06
node_yurthub_rejected_requests_counter 0
process_cpu_seconds_total 278.76
process_max_fds 1.048576e+06
process_open_fds 20
process_resident_memory_bytes 3.8285312e+07
process_start_time_seconds 1.68716349157e+09
process_virtual_memory_bytes 7.701504e+08
process_virtual_memory_max_bytes 1.8446744073709552e+19
promhttp_metric_handler_requests_in_flight 1
promhttp_metric_handler_requests_total{code="200"} 0 promhttp_metric_handler_requests_total{code="500"} 0 promhttp_metric_handler_requests_total{code="503"} 0
I execute the command and get the output above. i followed the documentation in openyurt.io. when i cancel the yurthub and use kubelet directly. It works. So it looks like a yurthub configuration mistake. By the way, this edge node is a virtual machine which runs on the master node machine. it there any affect? And how to solve it?
transparency066
commented
1 year ago And should i install yurt-manager and configure yurthub in master node first?
rambohe-ch
commented
1 year ago And should i install yurt-manager and configure yurthub in master node first?
@transparency066 It seems that yurthub can not work as expected, so would you be able to upload the detailed logs of yurthub?
yes, you should install yurt-manager component and yurthub artifacts at first. then use yurtadm join command to join node into cluster.
transparency066
commented
1 year ago I'm trying to manually configure the edge node that has joined in the cluster before. So should i delete the node from the cluster and use yurtadm? or just keep configuring the component in the edge node?
rambohe-ch
commented
1 year ago I'm trying to manually configure the edge node that has joined in the cluster before. So should i delete the node from the cluster and use yurtadm? or just keep configuring the component in the edge node?
@transparency066 It is recommended to use yurtadm join command to join node into cluster and yurtadm reset command to cleanup node.
transparency066
commented
1 year ago What about the nodes that have joined k8s
rambohe-ch
commented
1 year ago What about the nodes that have joined k8s
@transparency066 for native nodes, you can reference the tutorial: https://openyurt.io/zh/docs/installation/yurtadm-join/#2-%E5%9C%A8%E5%AD%98%E9%87%8F%E7%9A%84k8s%E8%8A%82%E7%82%B9%E4%B8%8A%E5%AE%89%E8%A3%85openyurt-node%E7%BB%84%E4%BB%B6
transparency066
commented
1 year ago that's what i'm doing right now. i wanna try to tnstall openyurt in two ways. anyway, thank you very much. i will install the components in master node and check out whether the previous error disappear or not.
transparency066
commented
1 year ago i'm sorry about another question. I got the master node prepared. unfortunately i found the yurt-manager is not running after i install yurthub. these are all the pods.
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel kube-flannel-ds-fl4r2 1/1 Running 1 (42s ago) 18m 10.0.2.6 k8s-node1 <none> <none>
kube-flannel kube-flannel-ds-jqkzs 1/1 Running 1 (3m33s ago) 18m 100.80.241.43 k8s-master <none> <none>
kube-system coredns-6d8c4cb4d-8d2fk 1/1 Running 1 (42s ago) 27m 10.244.1.3 k8s-node1 <none> <none>
kube-system coredns-6d8c4cb4d-997j2 1/1 Running 1 (3m33s ago) 27m 10.244.0.4 k8s-master <none> <none>
kube-system etcd-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-apiserver-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-controller-manager-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-proxy-mp5ll 1/1 Running 1 (42s ago) 19m 10.0.2.6 k8s-node1 <none> <none>
kube-system kube-proxy-q8mlx 1/1 Running 2 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-scheduler-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system raven-agent-ds-cz55t 1/1 Running 1 (42s ago) 16m 10.0.2.6 k8s-node1 <none> <none>
kube-system raven-agent-ds-k5vqf 1/1 Running 1 (3m33s ago) 16m 100.80.241.43 k8s-master <none> <none>
kube-system yurt-hub-k8s-node1 1/1 Running 1 (42s ago) 12m 10.0.2.6 k8s-node1 <none> <none>
kube-system yurt-manager-6bfdfdb4b8-cj4nx 0/1 CrashLoopBackOff 10 (43s ago) 17m 10.244.0.5 k8s-master <none> <none>i use the describe command and get events like:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m29s default-scheduler Successfully assigned kube-system/yurt-manager-6bfdfdb4b8-zmclf to k8s-master
Warning Unhealthy 65s (x2 over 2m28s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Normal Pulled 26s (x4 over 2m29s) kubelet Container image "openyurt/yurt-manager:v1.3.0" already present on machine
Normal Created 26s (x4 over 2m29s) kubelet Created container yurt-manager
Normal Started 26s (x4 over 2m29s) kubelet Started container yurt-manager
Warning BackOff 8s (x5 over 85s) kubelet Back-off restarting failed containerhere is the yurt-hub.yaml on edge node
apiVersion: v1
kind: Pod
metadata:
labels:
k8s-app: yurt-hub
name: yurt-hub
namespace: kube-system
spec:
volumes:
- name: hub-dir
hostPath:
path: /var/lib/yurthub
type: DirectoryOrCreate
- name: kubernetes
hostPath:
path: /etc/kubernetes
type: Directory
- name: pem-dir
hostPath:
path: /var/lib/kubelet/pki
type: Directory
containers:
- name: yurt-hub
image: openyurt/yurthub:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: hub-dir
mountPath: /var/lib/yurthub
- name: kubernetes
mountPath: /etc/kubernetes
- name: pem-dir
mountPath: /var/lib/kubelet/pki
command:
- yurthub
- --v=2
- --server-addr=https://100.80.241.43:6443
- --node-name=$(NODE_NAME)
- --join-token=m2loma.4gtv6xtof8hkqcqi
livenessProbe:
httpGet:
host: 127.0.0.1
path: /v1/healthz
port: 10267
initialDelaySeconds: 300
periodSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 150m
memory: 150Mi
limits:
memory: 300Mi
securityContext:
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
hostNetwork: true
priorityClassName: system-node-critical
priority: 2000001000how can i solve it?
transparency066
commented
1 year ago this is the log of yurt-manager
{"log":"yurt-manager version: projectinfo.Info{GitVersion:\"v1.3.0\", GitCommit:\"ac6ce54\", BuildDate:\"2023-05-15T02:25:41Z\", GoVersion:\"go1.18.10\", Compiler:\"gc\", Platform:\"linux/amd64\", AllVersions:[]string{\"v1.3.0\"}}\n","stream":"stdout","time":"2023-06-20T09:37:55.765332378Z"}
{"log":"I0620 09:37:55.765233 1 manager.go:143] FLAG: --add_dir_header=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765348437Z"}
{"log":"I0620 09:37:55.765251 1 manager.go:143] FLAG: --alsologtostderr=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765366287Z"}
{"log":"I0620 09:37:55.765252 1 manager.go:143] FLAG: --controllers=\"[*]\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765368032Z"}
{"log":"I0620 09:37:55.765255 1 manager.go:143] FLAG: --create-default-pool=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765369423Z"}
{"log":"I0620 09:37:55.765257 1 manager.go:143] FLAG: --disable-independent-webhooks=\"[]\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765370665Z"}
{"log":"I0620 09:37:55.765258 1 manager.go:143] FLAG: --enable-leader-election=\"true\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765371877Z"}
{"log":"I0620 09:37:55.765260 1 manager.go:143] FLAG: --feature-gates=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76537305Z"}
{"log":"I0620 09:37:55.765262 1 manager.go:143] FLAG: --health-probe-addr=\":10272\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765374259Z"}
{"log":"I0620 09:37:55.765263 1 manager.go:143] FLAG: --help=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765375414Z"}
{"log":"I0620 09:37:55.765264 1 manager.go:143] FLAG: --log-flush-frequency=\"5s\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76537657Z"}
{"log":"I0620 09:37:55.765266 1 manager.go:143] FLAG: --log_backtrace_at=\":0\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765377738Z"}
{"log":"I0620 09:37:55.765268 1 manager.go:143] FLAG: --log_dir=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765378911Z"}
{"log":"I0620 09:37:55.765269 1 manager.go:143] FLAG: --log_file=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765380048Z"}
{"log":"I0620 09:37:55.765270 1 manager.go:143] FLAG: --log_file_max_size=\"1800\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765381167Z"}
{"log":"I0620 09:37:55.765272 1 manager.go:143] FLAG: --logtostderr=\"true\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765382303Z"}
{"log":"I0620 09:37:55.765273 1 manager.go:143] FLAG: --metrics-addr=\":10271\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765383451Z"}
{"log":"I0620 09:37:55.765274 1 manager.go:143] FLAG: --node-servant-image=\"openyurt/node-servant:latest\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765384592Z"}
{"log":"I0620 09:37:55.765276 1 manager.go:143] FLAG: --one_output=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76538579Z"}
{"log":"I0620 09:37:55.765277 1 manager.go:143] FLAG: --rest-config-burst=\"50\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765386914Z"}
{"log":"I0620 09:37:55.765281 1 manager.go:143] FLAG: --rest-config-qps=\"30\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765388073Z"}
{"log":"I0620 09:37:55.765283 1 manager.go:143] FLAG: --skip_headers=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765389186Z"}
{"log":"I0620 09:37:55.765284 1 manager.go:143] FLAG: --skip_log_headers=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765390315Z"}
{"log":"I0620 09:37:55.765285 1 manager.go:143] FLAG: --stderrthreshold=\"2\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765391449Z"}
{"log":"I0620 09:37:55.765288 1 manager.go:143] FLAG: --v=\"4\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765392612Z"}
{"log":"I0620 09:37:55.765289 1 manager.go:143] FLAG: --version=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76539374Z"}
{"log":"I0620 09:37:55.765291 1 manager.go:143] FLAG: --vmodule=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765394882Z"}
{"log":"I0620 09:37:55.765292 1 manager.go:143] FLAG: --webhook-port=\"10273\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765400645Z"}
{"log":"I0620 09:37:55.765294 1 manager.go:143] FLAG: --working-namespace=\"kube-system\"\n","stream":"stderr","time":"2023-06-20T09:37:55.7654023Z"}
{"log":"I0620 09:37:55.765473 1 deleg.go:130] setup \"msg\"=\"new clientset registry\" \n","stream":"stderr","time":"2023-06-20T09:37:55.765491142Z"}
{"log":"I0620 09:37:55.779008 1 deleg.go:130] controller-runtime/metrics \"msg\"=\"Metrics server is starting to listen\" \"addr\"=\":10271\"\n","stream":"stderr","time":"2023-06-20T09:37:55.779089862Z"}
{"log":"I0620 09:37:55.779150 1 deleg.go:130] setup \"msg\"=\"setup controllers\" \n","stream":"stderr","time":"2023-06-20T09:37:55.779187482Z"}
{"log":"I0620 09:37:55.779168 1 controller.go:73] \"SetupWithManager\" len=11\n","stream":"stderr","time":"2023-06-20T09:37:55.779205946Z"}
{"log":"I0620 09:37:55.779704 1 yurtappdaemon_controller.go:82] yurtappdaemon-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtAppDaemon\n","stream":"stderr","time":"2023-06-20T09:37:55.779729733Z"}
{"log":"I0620 09:37:55.780200 1 csrapprover_controller.go:144] certificates.k8s.io/v1, Kind=CertificateSigningRequest is supported\n","stream":"stderr","time":"2023-06-20T09:37:55.780225897Z"}
{"log":"I0620 09:37:55.780690 1 daemon_pod_updater_controller.go:109] daemonupdater-controller add controller apps/v1, Kind=DaemonSet\n","stream":"stderr","time":"2023-06-20T09:37:55.78072405Z"}
{"log":"I0620 09:37:55.781619 1 gateway_controller.go:69] ravenl3-gateway-controller add controller raven.openyurt.io/v1alpha1, Kind=Gateway\n","stream":"stderr","time":"2023-06-20T09:37:55.781648181Z"}
{"log":"I0620 09:37:55.782344 1 service_controller.go:64] ravenl3-service-controller add controller /v1, Kind=Service\n","stream":"stderr","time":"2023-06-20T09:37:55.782373423Z"}
{"log":"I0620 09:37:55.783045 1 endpoints_controller.go:66] servicetopology-endpoints-controller add controller /v1, Kind=Endpoints\n","stream":"stderr","time":"2023-06-20T09:37:55.78308457Z"}
{"log":"I0620 09:37:55.784153 1 endpointslice_controller.go:117] discovery.k8s.io/v1, Kind=EndpointSlice is supported\n","stream":"stderr","time":"2023-06-20T09:37:55.784191506Z"}
{"log":"I0620 09:37:55.784170 1 endpointslice_controller.go:88] servicetopology-endpointslice controller is added\n","stream":"stderr","time":"2023-06-20T09:37:55.784203306Z"}
{"log":"I0620 09:37:55.785004 1 yurtstaticset_controller.go:135] yurtstaticset-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtStaticSet\n","stream":"stderr","time":"2023-06-20T09:37:55.785060659Z"}
{"log":"I0620 09:37:55.787103 1 nodepool_controller.go:76] nodepool-controller add controller apps.openyurt.io/v1beta1, Kind=NodePool\n","stream":"stderr","time":"2023-06-20T09:37:55.787134199Z"}
{"log":"I0620 09:37:55.790518 1 poolcoordinatorcert_controller.go:413] poolcoordinatorcert: CA already exist in secret, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.790565791Z"}
{"log":"I0620 09:37:55.790529 1 poolcoordinatorcert_controller.go:338] poolcoordinatorcert: init poolcoordinator started\n","stream":"stderr","time":"2023-06-20T09:37:55.790579566Z"}
{"log":"I0620 09:37:55.792611 1 poolcoordinatorcert_controller.go:383] poolcoordinatorcert: cert apiserver-etcd-client not change, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.792676427Z"}
{"log":"I0620 09:37:55.795017 1 poolcoordinatorcert_controller.go:383] poolcoordinatorcert: cert pool-coordinator-yurthub-client not change, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.795106766Z"}
{"log":"I0620 09:37:55.810003 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-static-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:55.810111323Z"}
{"log":"I0620 09:37:55.814369 1 certificate.go:390] poolcoordinatorcert: successfully write ca cert/key into pool-coordinator-static-certs\n","stream":"stderr","time":"2023-06-20T09:37:55.814450239Z"}
{"log":"I0620 09:37:55.821675 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-yurthub-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:55.821751802Z"}
{"log":"I0620 09:37:55.825150 1 certificate.go:390] poolcoordinatorcert: successfully write ca cert/key into pool-coordinator-yurthub-certs\n","stream":"stderr","time":"2023-06-20T09:37:55.825226532Z"}
{"log":"I0620 09:37:56.059999 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-static-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:56.060082508Z"}
{"log":"I0620 09:37:56.067451 1 certificate.go:435] poolcoordinatorcert: successfully write key pair into secret pool-coordinator-static-certs\n","stream":"stderr","time":"2023-06-20T09:37:56.067511246Z"}
{"log":"I0620 09:37:56.067860 1 yurtappset_controller.go:83] yurtappset-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtAppSet\n","stream":"stderr","time":"2023-06-20T09:37:56.067884489Z"}
{"log":"I0620 09:37:56.067866 1 yurtappset_controller.go:89] yurtappset-controller newReconciler apps.openyurt.io/v1alpha1, Kind=YurtAppSet\n","stream":"stderr","time":"2023-06-20T09:37:56.067893327Z"}
{"log":"I0620 09:37:56.067891 1 deleg.go:130] setup \"msg\"=\"setup webhook\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06790836Z"}
{"log":"I0620 09:37:56.067898 1 webhook.go:139] skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't calledGVK/v1, Kind=Pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067916453Z"}
{"log":"I0620 09:37:56.067910 1 webhook.go:164] Registering a validating webhookGVK/v1, Kind=Podpath/validate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.06792062Z"}
{"log":"I0620 09:37:56.067946 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-core-openyurt-io-v1-pod\" \n","stream":"stderr","time":"2023-06-20T09:37:56.067963605Z"}
{"log":"I0620 09:37:56.067974 1 server.go:96] Add webhook mutate path /mutate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067982785Z"}
{"log":"I0620 09:37:56.067976 1 server.go:101] Add webhook validate path /validate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067984462Z"}
{"log":"I0620 09:37:56.067993 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"raven.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"Gateway\"} \"path\"=\"/mutate-raven-openyurt-io-v1alpha1-gateway\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068000995Z"}
{"log":"I0620 09:37:56.068008 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-raven-openyurt-io-v1alpha1-gateway\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068015325Z"}
{"log":"I0620 09:37:56.068020 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"raven.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"Gateway\"} \"path\"=\"/validate-raven-openyurt-io-v1alpha1-gateway\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068028917Z"}
{"log":"I0620 09:37:56.068033 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-raven-openyurt-io-v1alpha1-gateway\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068040289Z"}
{"log":"I0620 09:37:56.068051 1 server.go:96] Add webhook mutate path /mutate-raven-openyurt-io-v1alpha1-gateway\n","stream":"stderr","time":"2023-06-20T09:37:56.068058321Z"}
{"log":"I0620 09:37:56.068053 1 server.go:101] Add webhook validate path /validate-raven-openyurt-io-v1alpha1-gateway\n","stream":"stderr","time":"2023-06-20T09:37:56.06805996Z"}
{"log":"I0620 09:37:56.068059 1 deleg.go:130] controller-runtime/builder \"msg\"=\"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068066803Z"}
{"log":"I0620 09:37:56.068065 1 deleg.go:130] controller-runtime/builder \"msg\"=\"skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068074674Z"}
{"log":"I0620 09:37:56.068087 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/convert\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068093766Z"}
{"log":"I0620 09:37:56.068098 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Conversion webhook enabled\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068107036Z"}
{"log":"I0620 09:37:56.068100 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068108893Z"}
{"log":"I0620 09:37:56.068101 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068110053Z"}
{"log":"I0620 09:37:56.068108 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"} \"path\"=\"/mutate-apps-openyurt-io-v1beta1-nodepool\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068115075Z"}
{"log":"I0620 09:37:56.068122 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1beta1-nodepool\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068128885Z"}
{"log":"I0620 09:37:56.068133 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"} \"path\"=\"/validate-apps-openyurt-io-v1beta1-nodepool\"\n","stream":"stderr","time":"2023-06-20T09:37:56.06813993Z"}
{"log":"I0620 09:37:56.068145 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1beta1-nodepool\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06815212Z"}
{"log":"I0620 09:37:56.068161 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Conversion webhook enabled\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068168586Z"}
{"log":"I0620 09:37:56.068163 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1beta1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068170337Z"}
{"log":"I0620 09:37:56.068164 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1beta1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068171455Z"}
{"log":"I0620 09:37:56.068171 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtStaticSet\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtstaticset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068177968Z"}
{"log":"I0620 09:37:56.068184 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtstaticset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068191418Z"}
{"log":"I0620 09:37:56.068194 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtStaticSet\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtstaticset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.06820145Z"}
{"log":"I0620 09:37:56.068210 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtstaticset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068217292Z"}
{"log":"I0620 09:37:56.068226 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtstaticset\n","stream":"stderr","time":"2023-06-20T09:37:56.0682336Z"}
{"log":"I0620 09:37:56.068228 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtstaticset\n","stream":"stderr","time":"2023-06-20T09:37:56.068235404Z"}
{"log":"I0620 09:37:56.068235 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppSet\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068241789Z"}
{"log":"I0620 09:37:56.068260 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06826784Z"}
{"log":"I0620 09:37:56.068270 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppSet\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068276754Z"}
{"log":"I0620 09:37:56.068282 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068289069Z"}
{"log":"I0620 09:37:56.068298 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtappset\n","stream":"stderr","time":"2023-06-20T09:37:56.068305527Z"}
{"log":"I0620 09:37:56.068300 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtappset\n","stream":"stderr","time":"2023-06-20T09:37:56.068307279Z"}
{"log":"I0620 09:37:56.068306 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppDaemon\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068313125Z"}
{"log":"I0620 09:37:56.068319 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068326251Z"}
{"log":"I0620 09:37:56.068328 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppDaemon\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappdaemon\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068335327Z"}
{"log":"I0620 09:37:56.068340 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappdaemon\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068354569Z"}
{"log":"I0620 09:37:56.068361 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\n","stream":"stderr","time":"2023-06-20T09:37:56.068368767Z"}
{"log":"I0620 09:37:56.068364 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtappdaemon\n","stream":"stderr","time":"2023-06-20T09:37:56.06837035Z"}
{"log":"I0620 09:37:56.068366 1 deleg.go:130] setup \"msg\"=\"initialize webhook\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068373204Z"}
{"log":"I0620 09:37:56.068656 1 webhook_controller.go:151] Starting webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.068678265Z"}
{"log":"I0620 09:37:56.068663 1 shared_informer.go:240] Waiting for caches to sync for webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.068685978Z"}
{"log":"I0620 09:37:56.068693 1 reflector.go:219] Starting reflector *v1.MutatingWebhookConfiguration (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068721497Z"}
{"log":"I0620 09:37:56.068695 1 reflector.go:255] Listing and watching *v1.MutatingWebhookConfiguration from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068729093Z"}
{"log":"I0620 09:37:56.068700 1 reflector.go:219] Starting reflector *v1.Secret (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068731057Z"}
{"log":"I0620 09:37:56.068705 1 reflector.go:255] Listing and watching *v1.Secret from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068732774Z"}
{"log":"I0620 09:37:56.068709 1 reflector.go:219] Starting reflector *v1.ValidatingWebhookConfiguration (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068734332Z"}
{"log":"I0620 09:37:56.068719 1 reflector.go:255] Listing and watching *v1.ValidatingWebhookConfiguration from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068735552Z"}
{"log":"I0620 09:37:56.069237 1 webhook_controller.go:108] MutatingWebhookConfiguration yurt-manager-mutating-webhook-configuration added\n","stream":"stderr","time":"2023-06-20T09:37:56.069253843Z"}
{"log":"I0620 09:37:56.069293 1 webhook_controller.go:125] ValidatingWebhookConfiguration yurt-manager-validating-webhook-configuration added\n","stream":"stderr","time":"2023-06-20T09:37:56.069311235Z"}
{"log":"I0620 09:37:56.071606 1 webhook_controller.go:91] Secret yurt-manager-webhook-certs added\n","stream":"stderr","time":"2023-06-20T09:37:56.071652185Z"}
{"log":"I0620 09:37:56.168975 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:37:56.169169946Z"}
{"log":"I0620 09:37:56.169018 1 shared_informer.go:247] Caches are synced for webhook-controller \n","stream":"stderr","time":"2023-06-20T09:37:56.169206027Z"}
{"log":"I0620 09:37:56.169032 1 webhook_controller.go:164] Started webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.169212852Z"}
{"log":"I0620 09:37:56.173027 1 fs.go:124] cert directory doesn't exist, creatingdirectory/tmp/yurt-manager-webhook-certs\n","stream":"stderr","time":"2023-06-20T09:37:56.173121294Z"}
{"log":"I0620 09:37:56.173700 1 atomic_writer.go:177] performed write of new data to ts data directoryts directory/tmp/yurt-manager-webhook-certs/..2023_06_20_09_37_56.3041147742\n","stream":"stderr","time":"2023-06-20T09:37:56.173861941Z"}
{"log":"I0620 09:37:56.180066 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/\"\n","stream":"stderr","time":"2023-06-20T09:37:56.18017204Z"}
{"log":"I0620 09:37:56.180097 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/profile\"\n","stream":"stderr","time":"2023-06-20T09:37:56.18020313Z"}
{"log":"I0620 09:37:56.180113 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/threadcreate\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180209977Z"}
{"log":"I0620 09:37:56.180133 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/block\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180216376Z"}
{"log":"I0620 09:37:56.180168 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/allocs\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180224324Z"}
{"log":"I0620 09:37:56.180185 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/cmdline\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180229648Z"}
{"log":"I0620 09:37:56.180199 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/symbol\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180254583Z"}
{"log":"I0620 09:37:56.180213 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/goroutine\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180261549Z"}
{"log":"I0620 09:37:56.180230 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/heap\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180266627Z"}
{"log":"I0620 09:37:56.180251 1 deleg.go:130] setup \"msg\"=\"starting manager\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180287598Z"}
{"log":"I0620 09:37:56.180397 1 leaderelection.go:248] attempting to acquire leader lease kube-system/yurt-manager...\n","stream":"stderr","time":"2023-06-20T09:37:56.180442922Z"}
{"log":"I0620 09:37:56.180434 1 server.go:213] controller-runtime/webhook/webhooks \"msg\"=\"Starting webhook server\" \n","stream":"stderr","time":"2023-06-20T09:37:56.18047928Z"}
{"log":"I0620 09:37:56.180450 1 deleg.go:130] setup \"msg\"=\"Starting metrics server\" \"path\"=\"/metrics\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180486036Z"}
{"log":"I0620 09:37:56.180482 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:37:56.180552282Z"}
{"log":"I0620 09:37:56.180746 1 deleg.go:130] controller-runtime/certwatcher \"msg\"=\"Updated current TLS certificate\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180816995Z"}
{"log":"I0620 09:37:56.180854 1 deleg.go:130] controller-runtime/webhook \"msg\"=\"Serving webhook server\" \"host\"=\"0.0.0.0\" \"port\"=10273\n","stream":"stderr","time":"2023-06-20T09:37:56.180893764Z"}
{"log":"I0620 09:37:56.180923 1 deleg.go:130] controller-runtime/certwatcher \"msg\"=\"Starting certificate watcher\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180954996Z"}
{"log":"I0620 09:37:56.184193 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:37:56.184289363Z"}
{"log":"I0620 09:37:56.184215 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:37:56.184317583Z"}
{"log":"I0620 09:37:59.116711 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:37:59.116873789Z"}
{"log":"I0620 09:37:59.116735 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:37:59.116936012Z"}
{"log":"I0620 09:38:01.989579 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:01.989724115Z"}
{"log":"I0620 09:38:01.989600 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:01.989771396Z"}
{"log":"I0620 09:38:04.571200 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:04.571328769Z"}
{"log":"I0620 09:38:04.571207 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:04.571349243Z"}
{"log":"I0620 09:38:07.233029 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:07.233231732Z"}
{"log":"I0620 09:38:07.233055 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:07.233283387Z"}
{"log":"I0620 09:38:10.893567 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:10.893704254Z"}
{"log":"I0620 09:38:10.893590 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:10.893756806Z"}
{"log":"I0620 09:38:14.816789 1 leaderelection.go:258] successfully acquired lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:14.816962856Z"}
{"log":"I0620 09:38:14.817048 1 recorder.go:103] setup/events \"msg\"=\"Normal\" \"message\"=\"yurt-manager-6bfdfdb4b8-zmclf_588ee144-ebe3-409a-8513-06ec78e0af9c became leader\" \"object\"={\"kind\":\"Lease\",\"namespace\":\"kube-system\",\"name\":\"yurt-manager\",\"uid\":\"d55bc2fc-d3a3-414a-a2b5-79b348ff5c0d\",\"apiVersion\":\"coordination.k8s.io/v1\",\"resourceVersion\":\"2526\"} \"reason\"=\"LeaderElection\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817222628Z"}
{"log":"I0620 09:38:14.817092 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtAppSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817275197Z"}
{"log":"I0620 09:38:14.817135 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.StatefulSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817289856Z"}
{"log":"I0620 09:38:14.817162 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Deployment\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817299257Z"}
{"log":"I0620 09:38:14.817178 1 controller.go:186] setup/controller/yurtappset \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817307408Z"}
{"log":"I0620 09:38:14.817190 1 controller.go:178] setup/controller/servicetopology-endpoints \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817334622Z"}
{"log":"I0620 09:38:14.817231 1 controller.go:178] setup/controller/servicetopology-endpoints \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817346118Z"}
{"log":"I0620 09:38:14.817246 1 controller.go:178] setup/controller/yurtappdaemon \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtAppDaemon\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817355086Z"}
{"log":"I0620 09:38:14.817255 1 controller.go:186] setup/controller/servicetopology-endpoints \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817362859Z"}
{"log":"I0620 09:38:14.817292 1 controller.go:178] setup/controller/yurtappdaemon \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.8174502Z"}
{"log":"I0620 09:38:14.817319 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817503128Z"}
{"log":"I0620 09:38:14.817327 1 controller.go:186] setup/controller/yurtappdaemon \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817517036Z"}
{"log":"I0620 09:38:14.817366 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.Gateway\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817526853Z"}
{"log":"I0620 09:38:14.817417 1 controller.go:178] setup/controller/nodepool \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817545878Z"}
{"log":"I0620 09:38:14.817396 1 controller.go:178] setup/controller/ravenl3-gateway \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.Gateway\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81755523Z"}
{"log":"I0620 09:38:14.817451 1 controller.go:178] setup/controller/nodepool \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817595379Z"}
{"log":"I0620 09:38:14.817481 1 controller.go:186] setup/controller/nodepool \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817604439Z"}
{"log":"I0620 09:38:14.817499 1 controller.go:178] setup/controller/ravenl3-gateway \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817612189Z"}
{"log":"I0620 09:38:14.817531 1 controller.go:186] setup/controller/ravenl3-gateway \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817623325Z"}
{"log":"I0620 09:38:14.817620 1 controller.go:178] setup/controller/daemonpodupdater \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.DaemonSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817733039Z"}
{"log":"I0620 09:38:14.817625 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Endpoints\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817770262Z"}
{"log":"I0620 09:38:14.817643 1 controller.go:178] setup/controller/delegatelease \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Lease\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817779153Z"}
{"log":"I0620 09:38:14.817660 1 controller.go:186] setup/controller/delegatelease \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817785455Z"}
{"log":"I0620 09:38:14.817664 1 controller.go:186] setup/controller/ravenl3-service \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817791278Z"}
{"log":"I0620 09:38:14.817668 1 controller.go:178] setup/controller/daemonpodupdater \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817796637Z"}
{"log":"I0620 09:38:14.817680 1 reflector.go:219] Starting reflector *v1alpha1.YurtAppSet (10h14m37.153933484s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.817802632Z"}
{"log":"I0620 09:38:14.817707 1 controller.go:186] setup/controller/daemonpodupdater \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817807981Z"}
{"log":"I0620 09:38:14.817709 1 reflector.go:255] Listing and watching *v1alpha1.YurtAppSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.817813407Z"}
{"log":"I0620 09:38:14.817874 1 reflector.go:219] Starting reflector *v1alpha1.NodePool (9h45m5.513141062s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818013201Z"}
{"log":"I0620 09:38:14.817904 1 reflector.go:255] Listing and watching *v1alpha1.NodePool from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818029962Z"}
{"log":"I0620 09:38:14.817917 1 controller.go:178] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818038934Z"}
{"log":"I0620 09:38:14.817945 1 controller.go:178] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818048123Z"}
{"log":"I0620 09:38:14.817977 1 controller.go:186] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818059512Z"}
{"log":"I0620 09:38:14.818083 1 controller.go:178] setup/controller/csrapprover \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.CertificateSigningRequest\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81820225Z"}
{"log":"I0620 09:38:14.818131 1 controller.go:186] setup/controller/csrapprover \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818230779Z"}
{"log":"I0620 09:38:14.818132 1 controller.go:178] setup/controller/podbinding \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818239649Z"}
{"log":"I0620 09:38:14.818152 1 controller.go:178] setup/controller/poolcoordinatorcert \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818249884Z"}
{"log":"I0620 09:38:14.818168 1 reflector.go:219] Starting reflector *v1.DaemonSet (9h44m12.954201241s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818259666Z"}
{"log":"I0620 09:38:14.818179 1 controller.go:186] setup/controller/podbinding \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818269246Z"}
{"log":"I0620 09:38:14.818193 1 controller.go:186] setup/controller/poolcoordinatorcert \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818280687Z"}
{"log":"I0620 09:38:14.818176 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtStaticSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81828988Z"}
{"log":"I0620 09:38:14.818197 1 reflector.go:255] Listing and watching *v1.DaemonSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818300176Z"}
{"log":"I0620 09:38:14.818254 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818311904Z"}
{"log":"I0620 09:38:14.818297 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818387788Z"}
{"log":"I0620 09:38:14.818316 1 reflector.go:219] Starting reflector *v1beta1.NodePool (10h35m43.937429363s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818397953Z"}
{"log":"I0620 09:38:14.818330 1 reflector.go:255] Listing and watching *v1beta1.NodePool from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818407789Z"}
{"log":"I0620 09:38:14.818330 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818417094Z"}
{"log":"I0620 09:38:14.818398 1 controller.go:186] setup/controller/yurtstaticset \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818485004Z"}
{"log":"I0620 09:38:14.818655 1 reflector.go:219] Starting reflector *v1.CertificateSigningRequest (10h18m40.649508617s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818825535Z"}
{"log":"I0620 09:38:14.818683 1 reflector.go:255] Listing and watching *v1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818863017Z"}
{"log":"I0620 09:38:14.819000 1 reflector.go:219] Starting reflector *v1.Pod (10h40m56.107884968s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819111678Z"}
{"log":"I0620 09:38:14.819017 1 reflector.go:255] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819176208Z"}
{"log":"I0620 09:38:14.819031 1 reflector.go:219] Starting reflector *v1alpha1.Gateway (10h13m43.424381342s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819209451Z"}
{"log":"I0620 09:38:14.819048 1 reflector.go:255] Listing and watching *v1alpha1.Gateway from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819215415Z"}
{"log":"I0620 09:38:14.819308 1 reflector.go:219] Starting reflector *v1.Node (9h57m37.876902086s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819367953Z"}
{"log":"I0620 09:38:14.819318 1 reflector.go:255] Listing and watching *v1.Node from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819383071Z"}
{"log":"I0620 09:38:14.819593 1 reflector.go:219] Starting reflector *v1.Service (10h22m2.577113809s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819725675Z"}
{"log":"I0620 09:38:14.819618 1 reflector.go:255] Listing and watching *v1.Service from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819750836Z"}
{"log":"I0620 09:38:14.819708 1 reflector.go:219] Starting reflector *v1.Deployment (9h28m17.742796923s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819759726Z"}
{"log":"I0620 09:38:14.819718 1 reflector.go:255] Listing and watching *v1.Deployment from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.81976517Z"}
{"log":"I0620 09:38:14.819970 1 reflector.go:219] Starting reflector *v1alpha1.YurtAppDaemon (9h45m57.370062593s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820021069Z"}
{"log":"I0620 09:38:14.819980 1 reflector.go:255] Listing and watching *v1alpha1.YurtAppDaemon from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820032354Z"}
{"log":"I0620 09:38:14.820119 1 reflector.go:219] Starting reflector *v1.StatefulSet (10h43m3.156628121s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820178756Z"}
{"log":"I0620 09:38:14.820147 1 reflector.go:255] Listing and watching *v1.StatefulSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820189445Z"}
{"log":"I0620 09:38:14.820182 1 reflector.go:219] Starting reflector *v1.Endpoints (10h57m46.652276057s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820223982Z"}
{"log":"I0620 09:38:14.820190 1 reflector.go:255] Listing and watching *v1.Endpoints from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820235639Z"}
{"log":"I0620 09:38:14.820544 1 reflector.go:219] Starting reflector *v1.Lease (10h47m35.307689498s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820601307Z"}
{"log":"I0620 09:38:14.820566 1 reflector.go:255] Listing and watching *v1.Lease from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820609966Z"}
{"log":"I0620 09:38:14.820743 1 reflector.go:219] Starting reflector *v1alpha1.YurtStaticSet (9h43m54.542719979s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820830809Z"}
{"log":"I0620 09:38:14.820769 1 reflector.go:255] Listing and watching *v1alpha1.YurtStaticSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820841711Z"}
{"log":"I0620 09:38:14.834451 1 warning_handler.go:65] KubeAPIWarningLogger \"msg\"=\"apps.openyurt.io/v1alpha1 NodePool is deprecated in v1.0.0+, unavailable in v1.2.0+; use apps.openyurt.io/v1beta1 NodePool\" \n","stream":"stderr","time":"2023-06-20T09:38:14.834521328Z"}
{"log":"I0620 09:38:14.917982 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918272743Z"}
{"log":"I0620 09:38:14.918066 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918321488Z"}
{"log":"I0620 09:38:14.918123 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918329879Z"}
{"log":"I0620 09:38:14.918207 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918350927Z"}
{"log":"I0620 09:38:14.918332 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918371417Z"}
{"log":"I0620 09:38:14.918352 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.91837771Z"}
{"log":"I0620 09:38:14.918336 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918391087Z"}
{"log":"I0620 09:38:14.918404 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918655857Z"}
{"log":"I0620 09:38:14.919389 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919624581Z"}
{"log":"I0620 09:38:14.919478 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919662367Z"}
{"log":"I0620 09:38:14.919506 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919671475Z"}
{"log":"I0620 09:38:14.919512 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.91967784Z"}
{"log":"I0620 09:38:14.919573 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919726147Z"}
{"log":"I0620 09:38:14.919592 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919781951Z"}
{"log":"I0620 09:38:14.919595 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919792029Z"}
{"log":"I0620 09:38:14.919642 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919798118Z"}
{"log":"I0620 09:38:14.919655 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919805125Z"}
{"log":"I0620 09:38:14.919714 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919815321Z"}
{"log":"I0620 09:38:14.919723 1 nodepool_enqueue_handlers.go:46] nodepool: node(k8s-master) does not belong to any nodepool\n","stream":"stderr","time":"2023-06-20T09:38:14.919821538Z"}
{"log":"I0620 09:38:14.919737 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919827743Z"}
{"log":"I0620 09:38:14.919747 1 nodepool_enqueue_handlers.go:46] nodepool: node(k8s-node1) does not belong to any nodepool\n","stream":"stderr","time":"2023-06-20T09:38:14.919834143Z"}
{"log":"I0620 09:38:14.919908 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92012736Z"}
{"log":"I0620 09:38:14.920144 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92044843Z"}
{"log":"I0620 09:38:14.920230 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920533891Z"}
{"log":"I0620 09:38:14.920324 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920548461Z"}
{"log":"I0620 09:38:14.920379 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92055515Z"}
{"log":"I0620 09:38:14.920544 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920646022Z"}
{"log":"I0620 09:38:14.920705 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920774373Z"}
{"log":"I0620 09:38:14.920413 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92087281Z"}
{"log":"I0620 09:38:14.920706 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920893714Z"}
{"log":"I0620 09:38:14.920852 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920958413Z"}
{"log":"I0620 09:38:14.920902 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920970578Z"}
{"log":"I0620 09:38:14.920911 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920980574Z"}
{"log":"I0620 09:38:14.920974 1 controller.go:220] setup/controller/csrapprover \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.921274176Z"}
{"log":"I0620 09:38:14.921080 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921322931Z"}
{"log":"I0620 09:38:14.921089 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92133656Z"}
{"log":"I0620 09:38:14.921240 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921349969Z"}
{"log":"I0620 09:38:14.921249 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921412107Z"}
{"log":"I0620 09:38:14.921356 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92143728Z"}
{"log":"I0620 09:38:14.921364 1 gateway_enqueue_handlers.go:42] ravenl3-gateway: node(k8s-node1) does not belong to any gateway\n","stream":"stderr","time":"2023-06-20T09:38:14.921448116Z"}
{"log":"I0620 09:38:14.921404 1 gateway_enqueue_handlers.go:42] ravenl3-gateway: node(k8s-master) does not belong to any gateway\n","stream":"stderr","time":"2023-06-20T09:38:14.92146422Z"}
{"log":"I0620 09:38:14.921429 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921475308Z"}
{"log":"I0620 09:38:14.921444 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921821866Z"}
{"log":"I0620 09:38:14.921467 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921837694Z"}
{"log":"I0620 09:38:14.921473 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921848036Z"}
{"log":"I0620 09:38:14.921497 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921857894Z"}
{"log":"I0620 09:38:14.921524 1 controller.go:220] setup/controller/poolcoordinatorcert \"msg\"=\"Starting workers\" \"worker count\"=1\n","stream":"stderr","time":"2023-06-20T09:38:14.921868175Z"}
{"log":"I0620 09:38:14.921490 1 controller.go:220] setup/controller/nodepool \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.921880883Z"}
{"log":"I0620 09:38:14.921530 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921892587Z"}
{"log":"I0620 09:38:14.921561 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921902099Z"}
{"log":"I0620 09:38:14.921567 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921911532Z"}
{"log":"I0620 09:38:14.921473 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921920909Z"}
{"log":"I0620 09:38:14.921576 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921930377Z"}
{"log":"I0620 09:38:14.921565 1 controller.go:220] setup/controller/servicetopology-endpoints \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.92197419Z"}
{"log":"I0620 09:38:14.921625 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921985679Z"}
{"log":"I0620 09:38:14.921657 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921996083Z"}
{"log":"I0620 09:38:14.921664 1 controller.go:220] setup/controller/podbinding \"msg\"=\"Starting workers\" \"worker count\"=5\n","stream":"stderr","time":"2023-06-20T09:38:14.922005961Z"}
{"log":"I0620 09:38:14.921669 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922017087Z"}
{"log":"I0620 09:38:14.921682 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922026842Z"}
{"log":"I0620 09:38:14.921680 1 controller.go:220] setup/controller/daemonpodupdater \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922038457Z"}
{"log":"I0620 09:38:14.921667 1 controller.go:220] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922049126Z"}
{"log":"I0620 09:38:14.921718 1 csrapprover_controller.go:190] csr(csr-7m2zn) is approved\n","stream":"stderr","time":"2023-06-20T09:38:14.92206021Z"}
{"log":"I0620 09:38:14.921697 1 controller.go:220] setup/controller/yurtappset \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922071021Z"}
{"log":"I0620 09:38:14.921739 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922082349Z"}
{"log":"I0620 09:38:14.921748 1 controller.go:220] setup/controller/ravenl3-gateway \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922094138Z"}
{"log":"I0620 09:38:14.921725 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922105454Z"}
{"log":"I0620 09:38:14.921807 1 controller.go:220] setup/controller/yurtstaticset \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922115887Z"}
{"log":"I0620 09:38:14.921810 1 controller.go:220] setup/controller/delegatelease \"msg\"=\"Starting workers\" \"worker count\"=5\n","stream":"stderr","time":"2023-06-20T09:38:14.922129729Z"}
{"log":"I0620 09:38:14.921842 1 csrapprover_controller.go:190] csr(csr-5jhbk) is approved\n","stream":"stderr","time":"2023-06-20T09:38:14.922139975Z"}
{"log":"I0620 09:38:14.921882 1 controller.go:220] setup/controller/yurtappdaemon \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.92215051Z"}
{"log":"I0620 09:38:14.921885 1 controller.go:220] setup/controller/ravenl3-service \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922160841Z"}
{"log":"I0620 09:38:14.921936 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet etcd\n","stream":"stderr","time":"2023-06-20T09:38:14.922172103Z"}
{"log":"I0620 09:38:14.921971 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-scheduler\n","stream":"stderr","time":"2023-06-20T09:38:14.922182245Z"}
{"log":"I0620 09:38:14.922002 1 podbinding_controller.go:92] node request: k8s-master\n","stream":"stderr","time":"2023-06-20T09:38:14.922192952Z"}
{"log":"E0620 09:38:14.922034 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/etcd, YurtStaticSet.apps.openyurt.io \"etcd\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922310021Z"}
{"log":"I0620 09:38:14.921981 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-apiserver\n","stream":"stderr","time":"2023-06-20T09:38:14.9223587Z"}
{"log":"E0620 09:38:14.922109 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-scheduler, YurtStaticSet.apps.openyurt.io \"kube-scheduler\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922404573Z"}
{"log":"I0620 09:38:14.921980 1 podbinding_controller.go:92] node request: k8s-node1\n","stream":"stderr","time":"2023-06-20T09:38:14.922418774Z"}
{"log":"E0620 09:38:14.922210 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-apiserver, YurtStaticSet.apps.openyurt.io \"kube-apiserver\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922430098Z"}
{"log":"I0620 09:38:14.922259 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet yurt-hub\n","stream":"stderr","time":"2023-06-20T09:38:14.922442099Z"}
{"log":"I0620 09:38:14.922302 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet yurt-hub-cloud\n","stream":"stderr","time":"2023-06-20T09:38:14.922456356Z"}
{"log":"I0620 09:38:14.922216 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-controller-manager\n","stream":"stderr","time":"2023-06-20T09:38:14.922535403Z"}
{"log":"E0620 09:38:14.922770 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-controller-manager, YurtStaticSet.apps.openyurt.io \"kube-controller-manager\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922931382Z"}
{"log":"I0620 09:38:14.930081 1 reflector.go:219] Starting reflector *v1.ConfigMap (9h43m30.684598835s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.930240236Z"}
{"log":"I0620 09:38:14.930113 1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.930290381Z"}
{"log":"I0620 09:38:14.932774 1 delegatelease_controller.go:160] detaint k8s-master: no key node.openyurt.io/unschedulable exists, nothing to do\n","stream":"stderr","time":"2023-06-20T09:38:14.93289449Z"}
{"log":"I0620 09:38:14.932791 1 delegatelease_controller.go:160] detaint k8s-node1: no key node.openyurt.io/unschedulable exists, nothing to do\n","stream":"stderr","time":"2023-06-20T09:38:14.932939712Z"}
{"log":"panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2023-06-20T09:38:14.948547409Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x163d13b]\n","stream":"stderr","time":"2023-06-20T09:38:14.948582511Z"}
{"log":"\n","stream":"stderr","time":"2023-06-20T09:38:14.948590955Z"}
{"log":"goroutine 656 [running]:\n","stream":"stderr","time":"2023-06-20T09:38:14.948596233Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).configureTolerationForPod(0xc00069dd40, 0xc0013f43e8, 0x0)\n","stream":"stderr","time":"2023-06-20T09:38:14.94860166Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:142 +0x21b\n","stream":"stderr","time":"2023-06-20T09:38:14.948607739Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).processNode(0xc0012102d0?, {0x1de4268?, 0xc001216480?}, 0xc001224000)\n","stream":"stderr","time":"2023-06-20T09:38:14.948613823Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:117 +0x33d\n","stream":"stderr","time":"2023-06-20T09:38:14.948619256Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).Reconcile(0xc00069dd40, {0x1de4268, 0xc001216480}, {{{0x0, 0x0}, {0xc0000dc240, 0x9}}})\n","stream":"stderr","time":"2023-06-20T09:38:14.948630304Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:93 +0x367\n","stream":"stderr","time":"2023-06-20T09:38:14.948636175Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000ca3360, {0x1de4268, 0xc001216450}, {{{0x0?, 0x1aa05c0?}, {0xc0000dc240?, 0x1?}}})\n","stream":"stderr","time":"2023-06-20T09:38:14.948641315Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114 +0x222\n","stream":"stderr","time":"2023-06-20T09:38:14.948668673Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000ca3360, {0x1de41c0, 0xc0003b0a80}, {0x1966b00?, 0xc001330020?})\n","stream":"stderr","time":"2023-06-20T09:38:14.948677056Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311 +0x2e9\n","stream":"stderr","time":"2023-06-20T09:38:14.948682395Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000ca3360, {0x1de41c0, 0xc0003b0a80})\n","stream":"stderr","time":"2023-06-20T09:38:14.948687587Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266 +0x1d9\n","stream":"stderr","time":"2023-06-20T09:38:14.948692882Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n","stream":"stderr","time":"2023-06-20T09:38:14.948698034Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227 +0x85\n","stream":"stderr","time":"2023-06-20T09:38:14.948704943Z"}
{"log":"created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n","stream":"stderr","time":"2023-06-20T09:38:14.948710168Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:223 +0x309\n","stream":"stderr","time":"2023-06-20T09:38:14.948715195Z"}@transparency066 The above bug is known to yurt-manager component, and has been fixed in the pull request: #1538 . you can use the latest image of yurt-manager or set the log level below 4 to skip this bug.
rambohe-ch
commented
1 year ago @transparency066 By the way, i have backported the above pull request into tag: v1.3.2 just now, and you also use yurt-manager:v1.3.2 after image released.
transparency066
commented
1 year ago i will wait for the next version. now i use yurtadm instead. but there is a error when i executed yurtadm join 100.80.241.43:6443 --token=3j0skf.urcezt4uj4vtmvwi --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5 command.
container runtime is not running: output: time="2023-06-21T14:05:11+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeServiceand the log of kubelet is like
Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml>but if i use kubeadm it joins into clusters successfully. it seems something in yurtadm is wrong. how do i fix it?
transparency066
commented
1 year ago and my k8s version is 1.23.0. is it too high to have compatibility problems?
transparency066
commented
1 year ago i will wait for the next version. now i use yurtadm instead. but there is a error when i executed
yurtadm join 100.80.241.43:6443 --token=3j0skf.urcezt4uj4vtmvwi --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5command.container runtime is not running: output: time="2023-06-21T14:05:11+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeServiceand the log of kubelet is like
Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml>but if i use kubeadm it joins into clusters successfully. it seems something in yurtadm is wrong. how do i fix it?
i'm sorry. i should use docker-runtime command. it works.
rambohe-ch
commented
1 year ago @transparency066 yurt-manager:v1.3.2 has been released, and the openyurt helm charts have been updated.
transparency066
commented
1 year ago i install the yurt-manager again and still get the v1.3.0. any solution?
rambohe-ch
commented
1 year ago i install the yurt-manager again and still get the v1.3.0. any solution?
@transparency066 I think you need to update helm repo, and you can reference the helm tutorial here: https://helm.sh/docs/helm/helm_repo_update/
transparency066
commented
1 year ago u are right! thank you very much.
rambohe-ch
commented
1 year ago @transparency066 It seems that your problem had been solved, and would you like to close this issue?
{"log":"I0619 08:31:11.977819 1 util.go:289] start proxying: get /apis/storage.k8s.io/v1/csinodes/k8s-node1, in flight requests: 4\n","stream":"stderr","time":"2023-06-19T08:31:11.978141501Z"} {"log":"E0619 08:31:11.977982 1 local.go:218] can not cache for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1\n","stream":"stderr","time":"2023-06-19T08:31:11.97822749Z"} {"log":"E0619 08:31:11.978030 1 local.go:88] could not proxy local for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1, can not cache for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1\n","stream":"stderr","time":"2023-06-19T08:31:11.978236642Z"} {"log":"I0619 08:31:11.978115 1 util.go:248] go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1 with status code 400, spent 169.116µs\n","stream":"stderr","time":"2023-06-19T08:31:11.978247369Z"}
when i reconfigure the kubelet drop-in and yurt-hub, the edge not seems to be not ready. And the logs in the yurthub in edge node is like above. I don't understand what the mistake is. But in the master node, i get the all pods including yurthub running.