Closed transparency066 closed 1 year ago
@transparency066 It seems that yurthub can not connect to kube-apiserver, and you can check the server health status in the metrics of yurthub by the command: curl http://127.0.0.1:10267/metrics
.
and you should make sure that yurthub can connect to kube-apiserver when you re-configures yurthub.
go_gc_duration_seconds{quantile="0"} 2.8984e-05 go_gc_duration_seconds{quantile="0.25"} 7.3124e-05 go_gc_duration_seconds{quantile="0.5"} 0.000138523 go_gc_duration_seconds{quantile="0.75"} 0.000240081 go_gc_duration_seconds{quantile="1"} 0.000700298 go_gc_duration_seconds_sum 0.096005456 go_gc_duration_seconds_count 547
go_goroutines 80
go_info{version="go1.18.10"} 1
go_memstats_alloc_bytes 1.1164184e+07
go_memstats_alloc_bytes_total 1.555998208e+09
go_memstats_buck_hash_sys_bytes 1.61637e+06
go_memstats_frees_total 1.8244247e+07
go_memstats_gc_sys_bytes 5.657504e+06
go_memstats_heap_alloc_bytes 1.1164184e+07
go_memstats_heap_idle_bytes 3.178496e+06
go_memstats_heap_inuse_bytes 1.2222464e+07
go_memstats_heap_objects 95191
go_memstats_heap_released_bytes 2.654208e+06
go_memstats_heap_sys_bytes 1.540096e+07
go_memstats_last_gc_time_seconds 1.6872285868090148e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 1.8339438e+07
go_memstats_mcache_inuse_bytes 4800
go_memstats_mcache_sys_bytes 15600
go_memstats_mspan_inuse_bytes 192528
go_memstats_mspan_sys_bytes 195264
go_memstats_next_gc_bytes 1.1477712e+07
go_memstats_other_sys_bytes 724678
go_memstats_stack_inuse_bytes 1.376256e+06
go_memstats_stack_sys_bytes 1.376256e+06
go_memstats_sys_bytes 2.4986632e+07
go_threads 12
node_yurthub_closable_conns_collector{server="100.80.241.43:6443"} 0
node_yurthub_in_flight_requests_collector{client="go-http-client",resource="csidrivers",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="csinodes",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="leases",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="create"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="nodes",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="pods",subresources="",verb="get"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="runtimeclasses",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="go-http-client",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="create"} 0 node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="patch"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="configmaps",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="nodepools",subresources="",verb="watch"} 1 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="list"} 0 node_yurthub_in_flight_requests_collector{client="yurthub",resource="services",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_total 3
node_yurthub_proxy_latency_collector{client="go-http-client",resource="csidrivers",subresources="",type="apiserver_latency",verb="list"} 118189 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csidrivers",subresources="",type="full_latency",verb="list"} 186552 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csinodes",subresources="",type="apiserver_latency",verb="get"} 234323 node_yurthub_proxy_latency_collector{client="go-http-client",resource="csinodes",subresources="",type="full_latency",verb="get"} 441823 node_yurthub_proxy_latency_collector{client="go-http-client",resource="leases",subresources="",type="apiserver_latency",verb="get"} 99815 node_yurthub_proxy_latency_collector{client="go-http-client",resource="leases",subresources="",type="full_latency",verb="get"} 159074 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="create"} 22646 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="get"} 86148 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="apiserver_latency",verb="list"} 327299 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="create"} 36670 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="get"} 104780 node_yurthub_proxy_latency_collector{client="go-http-client",resource="nodes",subresources="",type="full_latency",verb="list"} 402995 node_yurthub_proxy_latency_collector{client="go-http-client",resource="pods",subresources="",type="apiserver_latency",verb="get"} 118223 node_yurthub_proxy_latency_collector{client="go-http-client",resource="pods",subresources="",type="full_latency",verb="get"} 139521 node_yurthub_proxy_latency_collector{client="go-http-client",resource="runtimeclasses",subresources="",type="apiserver_latency",verb="list"} 86546 node_yurthub_proxy_latency_collector{client="go-http-client",resource="runtimeclasses",subresources="",type="full_latency",verb="list"} 133640 node_yurthub_proxy_latency_collector{client="go-http-client",resource="services",subresources="",type="apiserver_latency",verb="list"} 329371 node_yurthub_proxy_latency_collector{client="go-http-client",resource="services",subresources="",type="full_latency",verb="list"} 461140 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="apiserver_latency",verb="create"} 70005 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="apiserver_latency",verb="patch"} 461076 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="full_latency",verb="create"} 114781 node_yurthub_proxy_latency_collector{client="kubelet",resource="events",subresources="",type="full_latency",verb="patch"} 523876 node_yurthub_proxy_latency_collector{client="yurthub",resource="configmaps",subresources="",type="apiserver_latency",verb="list"} 726094 node_yurthub_proxy_latency_collector{client="yurthub",resource="configmaps",subresources="",type="full_latency",verb="list"} 744325 node_yurthub_proxy_latency_collector{client="yurthub",resource="nodepools",subresources="",type="apiserver_latency",verb="list"} 102808 node_yurthub_proxy_latency_collector{client="yurthub",resource="nodepools",subresources="",type="full_latency",verb="list"} 134168 node_yurthub_proxy_latency_collector{client="yurthub",resource="services",subresources="",type="apiserver_latency",verb="list"} 3.371083e+06 node_yurthub_proxy_latency_collector{client="yurthub",resource="services",subresources="",type="full_latency",verb="list"} 3.390693e+06
node_yurthub_rejected_requests_counter 0
process_cpu_seconds_total 278.76
process_max_fds 1.048576e+06
process_open_fds 20
process_resident_memory_bytes 3.8285312e+07
process_start_time_seconds 1.68716349157e+09
process_virtual_memory_bytes 7.701504e+08
process_virtual_memory_max_bytes 1.8446744073709552e+19
promhttp_metric_handler_requests_in_flight 1
promhttp_metric_handler_requests_total{code="200"} 0 promhttp_metric_handler_requests_total{code="500"} 0 promhttp_metric_handler_requests_total{code="503"} 0
I execute the command and get the output above. i followed the documentation in openyurt.io. when i cancel the yurthub and use kubelet directly. It works. So it looks like a yurthub configuration mistake. By the way, this edge node is a virtual machine which runs on the master node machine. it there any affect? And how to solve it?
And should i install yurt-manager and configure yurthub in master node first?
And should i install yurt-manager and configure yurthub in master node first?
@transparency066 It seems that yurthub can not work as expected, so would you be able to upload the detailed logs of yurthub?
yes, you should install yurt-manager component and yurthub artifacts at first. then use yurtadm join
command to join node into cluster.
I'm trying to manually configure the edge node that has joined in the cluster before. So should i delete the node from the cluster and use yurtadm? or just keep configuring the component in the edge node?
I'm trying to manually configure the edge node that has joined in the cluster before. So should i delete the node from the cluster and use yurtadm? or just keep configuring the component in the edge node?
@transparency066 It is recommended to use yurtadm join
command to join node into cluster and yurtadm reset
command to cleanup node.
What about the nodes that have joined k8s
What about the nodes that have joined k8s
@transparency066 for native nodes, you can reference the tutorial: https://openyurt.io/zh/docs/installation/yurtadm-join/#2-%E5%9C%A8%E5%AD%98%E9%87%8F%E7%9A%84k8s%E8%8A%82%E7%82%B9%E4%B8%8A%E5%AE%89%E8%A3%85openyurt-node%E7%BB%84%E4%BB%B6
that's what i'm doing right now. i wanna try to tnstall openyurt in two ways. anyway, thank you very much. i will install the components in master node and check out whether the previous error disappear or not.
i'm sorry about another question. I got the master node prepared. unfortunately i found the yurt-manager is not running after i install yurthub. these are all the pods.
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel kube-flannel-ds-fl4r2 1/1 Running 1 (42s ago) 18m 10.0.2.6 k8s-node1 <none> <none>
kube-flannel kube-flannel-ds-jqkzs 1/1 Running 1 (3m33s ago) 18m 100.80.241.43 k8s-master <none> <none>
kube-system coredns-6d8c4cb4d-8d2fk 1/1 Running 1 (42s ago) 27m 10.244.1.3 k8s-node1 <none> <none>
kube-system coredns-6d8c4cb4d-997j2 1/1 Running 1 (3m33s ago) 27m 10.244.0.4 k8s-master <none> <none>
kube-system etcd-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-apiserver-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-controller-manager-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-proxy-mp5ll 1/1 Running 1 (42s ago) 19m 10.0.2.6 k8s-node1 <none> <none>
kube-system kube-proxy-q8mlx 1/1 Running 2 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system kube-scheduler-k8s-master 1/1 Running 8 (3m33s ago) 27m 100.80.241.43 k8s-master <none> <none>
kube-system raven-agent-ds-cz55t 1/1 Running 1 (42s ago) 16m 10.0.2.6 k8s-node1 <none> <none>
kube-system raven-agent-ds-k5vqf 1/1 Running 1 (3m33s ago) 16m 100.80.241.43 k8s-master <none> <none>
kube-system yurt-hub-k8s-node1 1/1 Running 1 (42s ago) 12m 10.0.2.6 k8s-node1 <none> <none>
kube-system yurt-manager-6bfdfdb4b8-cj4nx 0/1 CrashLoopBackOff 10 (43s ago) 17m 10.244.0.5 k8s-master <none> <none>
i use the describe command and get events like:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m29s default-scheduler Successfully assigned kube-system/yurt-manager-6bfdfdb4b8-zmclf to k8s-master
Warning Unhealthy 65s (x2 over 2m28s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Normal Pulled 26s (x4 over 2m29s) kubelet Container image "openyurt/yurt-manager:v1.3.0" already present on machine
Normal Created 26s (x4 over 2m29s) kubelet Created container yurt-manager
Normal Started 26s (x4 over 2m29s) kubelet Started container yurt-manager
Warning BackOff 8s (x5 over 85s) kubelet Back-off restarting failed container
here is the yurt-hub.yaml on edge node
apiVersion: v1
kind: Pod
metadata:
labels:
k8s-app: yurt-hub
name: yurt-hub
namespace: kube-system
spec:
volumes:
- name: hub-dir
hostPath:
path: /var/lib/yurthub
type: DirectoryOrCreate
- name: kubernetes
hostPath:
path: /etc/kubernetes
type: Directory
- name: pem-dir
hostPath:
path: /var/lib/kubelet/pki
type: Directory
containers:
- name: yurt-hub
image: openyurt/yurthub:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: hub-dir
mountPath: /var/lib/yurthub
- name: kubernetes
mountPath: /etc/kubernetes
- name: pem-dir
mountPath: /var/lib/kubelet/pki
command:
- yurthub
- --v=2
- --server-addr=https://100.80.241.43:6443
- --node-name=$(NODE_NAME)
- --join-token=m2loma.4gtv6xtof8hkqcqi
livenessProbe:
httpGet:
host: 127.0.0.1
path: /v1/healthz
port: 10267
initialDelaySeconds: 300
periodSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 150m
memory: 150Mi
limits:
memory: 300Mi
securityContext:
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
hostNetwork: true
priorityClassName: system-node-critical
priority: 2000001000
how can i solve it?
this is the log of yurt-manager
{"log":"yurt-manager version: projectinfo.Info{GitVersion:\"v1.3.0\", GitCommit:\"ac6ce54\", BuildDate:\"2023-05-15T02:25:41Z\", GoVersion:\"go1.18.10\", Compiler:\"gc\", Platform:\"linux/amd64\", AllVersions:[]string{\"v1.3.0\"}}\n","stream":"stdout","time":"2023-06-20T09:37:55.765332378Z"}
{"log":"I0620 09:37:55.765233 1 manager.go:143] FLAG: --add_dir_header=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765348437Z"}
{"log":"I0620 09:37:55.765251 1 manager.go:143] FLAG: --alsologtostderr=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765366287Z"}
{"log":"I0620 09:37:55.765252 1 manager.go:143] FLAG: --controllers=\"[*]\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765368032Z"}
{"log":"I0620 09:37:55.765255 1 manager.go:143] FLAG: --create-default-pool=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765369423Z"}
{"log":"I0620 09:37:55.765257 1 manager.go:143] FLAG: --disable-independent-webhooks=\"[]\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765370665Z"}
{"log":"I0620 09:37:55.765258 1 manager.go:143] FLAG: --enable-leader-election=\"true\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765371877Z"}
{"log":"I0620 09:37:55.765260 1 manager.go:143] FLAG: --feature-gates=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76537305Z"}
{"log":"I0620 09:37:55.765262 1 manager.go:143] FLAG: --health-probe-addr=\":10272\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765374259Z"}
{"log":"I0620 09:37:55.765263 1 manager.go:143] FLAG: --help=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765375414Z"}
{"log":"I0620 09:37:55.765264 1 manager.go:143] FLAG: --log-flush-frequency=\"5s\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76537657Z"}
{"log":"I0620 09:37:55.765266 1 manager.go:143] FLAG: --log_backtrace_at=\":0\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765377738Z"}
{"log":"I0620 09:37:55.765268 1 manager.go:143] FLAG: --log_dir=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765378911Z"}
{"log":"I0620 09:37:55.765269 1 manager.go:143] FLAG: --log_file=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765380048Z"}
{"log":"I0620 09:37:55.765270 1 manager.go:143] FLAG: --log_file_max_size=\"1800\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765381167Z"}
{"log":"I0620 09:37:55.765272 1 manager.go:143] FLAG: --logtostderr=\"true\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765382303Z"}
{"log":"I0620 09:37:55.765273 1 manager.go:143] FLAG: --metrics-addr=\":10271\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765383451Z"}
{"log":"I0620 09:37:55.765274 1 manager.go:143] FLAG: --node-servant-image=\"openyurt/node-servant:latest\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765384592Z"}
{"log":"I0620 09:37:55.765276 1 manager.go:143] FLAG: --one_output=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76538579Z"}
{"log":"I0620 09:37:55.765277 1 manager.go:143] FLAG: --rest-config-burst=\"50\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765386914Z"}
{"log":"I0620 09:37:55.765281 1 manager.go:143] FLAG: --rest-config-qps=\"30\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765388073Z"}
{"log":"I0620 09:37:55.765283 1 manager.go:143] FLAG: --skip_headers=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765389186Z"}
{"log":"I0620 09:37:55.765284 1 manager.go:143] FLAG: --skip_log_headers=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765390315Z"}
{"log":"I0620 09:37:55.765285 1 manager.go:143] FLAG: --stderrthreshold=\"2\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765391449Z"}
{"log":"I0620 09:37:55.765288 1 manager.go:143] FLAG: --v=\"4\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765392612Z"}
{"log":"I0620 09:37:55.765289 1 manager.go:143] FLAG: --version=\"false\"\n","stream":"stderr","time":"2023-06-20T09:37:55.76539374Z"}
{"log":"I0620 09:37:55.765291 1 manager.go:143] FLAG: --vmodule=\"\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765394882Z"}
{"log":"I0620 09:37:55.765292 1 manager.go:143] FLAG: --webhook-port=\"10273\"\n","stream":"stderr","time":"2023-06-20T09:37:55.765400645Z"}
{"log":"I0620 09:37:55.765294 1 manager.go:143] FLAG: --working-namespace=\"kube-system\"\n","stream":"stderr","time":"2023-06-20T09:37:55.7654023Z"}
{"log":"I0620 09:37:55.765473 1 deleg.go:130] setup \"msg\"=\"new clientset registry\" \n","stream":"stderr","time":"2023-06-20T09:37:55.765491142Z"}
{"log":"I0620 09:37:55.779008 1 deleg.go:130] controller-runtime/metrics \"msg\"=\"Metrics server is starting to listen\" \"addr\"=\":10271\"\n","stream":"stderr","time":"2023-06-20T09:37:55.779089862Z"}
{"log":"I0620 09:37:55.779150 1 deleg.go:130] setup \"msg\"=\"setup controllers\" \n","stream":"stderr","time":"2023-06-20T09:37:55.779187482Z"}
{"log":"I0620 09:37:55.779168 1 controller.go:73] \"SetupWithManager\" len=11\n","stream":"stderr","time":"2023-06-20T09:37:55.779205946Z"}
{"log":"I0620 09:37:55.779704 1 yurtappdaemon_controller.go:82] yurtappdaemon-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtAppDaemon\n","stream":"stderr","time":"2023-06-20T09:37:55.779729733Z"}
{"log":"I0620 09:37:55.780200 1 csrapprover_controller.go:144] certificates.k8s.io/v1, Kind=CertificateSigningRequest is supported\n","stream":"stderr","time":"2023-06-20T09:37:55.780225897Z"}
{"log":"I0620 09:37:55.780690 1 daemon_pod_updater_controller.go:109] daemonupdater-controller add controller apps/v1, Kind=DaemonSet\n","stream":"stderr","time":"2023-06-20T09:37:55.78072405Z"}
{"log":"I0620 09:37:55.781619 1 gateway_controller.go:69] ravenl3-gateway-controller add controller raven.openyurt.io/v1alpha1, Kind=Gateway\n","stream":"stderr","time":"2023-06-20T09:37:55.781648181Z"}
{"log":"I0620 09:37:55.782344 1 service_controller.go:64] ravenl3-service-controller add controller /v1, Kind=Service\n","stream":"stderr","time":"2023-06-20T09:37:55.782373423Z"}
{"log":"I0620 09:37:55.783045 1 endpoints_controller.go:66] servicetopology-endpoints-controller add controller /v1, Kind=Endpoints\n","stream":"stderr","time":"2023-06-20T09:37:55.78308457Z"}
{"log":"I0620 09:37:55.784153 1 endpointslice_controller.go:117] discovery.k8s.io/v1, Kind=EndpointSlice is supported\n","stream":"stderr","time":"2023-06-20T09:37:55.784191506Z"}
{"log":"I0620 09:37:55.784170 1 endpointslice_controller.go:88] servicetopology-endpointslice controller is added\n","stream":"stderr","time":"2023-06-20T09:37:55.784203306Z"}
{"log":"I0620 09:37:55.785004 1 yurtstaticset_controller.go:135] yurtstaticset-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtStaticSet\n","stream":"stderr","time":"2023-06-20T09:37:55.785060659Z"}
{"log":"I0620 09:37:55.787103 1 nodepool_controller.go:76] nodepool-controller add controller apps.openyurt.io/v1beta1, Kind=NodePool\n","stream":"stderr","time":"2023-06-20T09:37:55.787134199Z"}
{"log":"I0620 09:37:55.790518 1 poolcoordinatorcert_controller.go:413] poolcoordinatorcert: CA already exist in secret, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.790565791Z"}
{"log":"I0620 09:37:55.790529 1 poolcoordinatorcert_controller.go:338] poolcoordinatorcert: init poolcoordinator started\n","stream":"stderr","time":"2023-06-20T09:37:55.790579566Z"}
{"log":"I0620 09:37:55.792611 1 poolcoordinatorcert_controller.go:383] poolcoordinatorcert: cert apiserver-etcd-client not change, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.792676427Z"}
{"log":"I0620 09:37:55.795017 1 poolcoordinatorcert_controller.go:383] poolcoordinatorcert: cert pool-coordinator-yurthub-client not change, reuse it\n","stream":"stderr","time":"2023-06-20T09:37:55.795106766Z"}
{"log":"I0620 09:37:55.810003 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-static-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:55.810111323Z"}
{"log":"I0620 09:37:55.814369 1 certificate.go:390] poolcoordinatorcert: successfully write ca cert/key into pool-coordinator-static-certs\n","stream":"stderr","time":"2023-06-20T09:37:55.814450239Z"}
{"log":"I0620 09:37:55.821675 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-yurthub-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:55.821751802Z"}
{"log":"I0620 09:37:55.825150 1 certificate.go:390] poolcoordinatorcert: successfully write ca cert/key into pool-coordinator-yurthub-certs\n","stream":"stderr","time":"2023-06-20T09:37:55.825226532Z"}
{"log":"I0620 09:37:56.059999 1 secret.go:57] poolcoordinatorcert: secret pool-coordinator-static-certs already exists\n","stream":"stderr","time":"2023-06-20T09:37:56.060082508Z"}
{"log":"I0620 09:37:56.067451 1 certificate.go:435] poolcoordinatorcert: successfully write key pair into secret pool-coordinator-static-certs\n","stream":"stderr","time":"2023-06-20T09:37:56.067511246Z"}
{"log":"I0620 09:37:56.067860 1 yurtappset_controller.go:83] yurtappset-controller add controller apps.openyurt.io/v1alpha1, Kind=YurtAppSet\n","stream":"stderr","time":"2023-06-20T09:37:56.067884489Z"}
{"log":"I0620 09:37:56.067866 1 yurtappset_controller.go:89] yurtappset-controller newReconciler apps.openyurt.io/v1alpha1, Kind=YurtAppSet\n","stream":"stderr","time":"2023-06-20T09:37:56.067893327Z"}
{"log":"I0620 09:37:56.067891 1 deleg.go:130] setup \"msg\"=\"setup webhook\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06790836Z"}
{"log":"I0620 09:37:56.067898 1 webhook.go:139] skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't calledGVK/v1, Kind=Pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067916453Z"}
{"log":"I0620 09:37:56.067910 1 webhook.go:164] Registering a validating webhookGVK/v1, Kind=Podpath/validate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.06792062Z"}
{"log":"I0620 09:37:56.067946 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-core-openyurt-io-v1-pod\" \n","stream":"stderr","time":"2023-06-20T09:37:56.067963605Z"}
{"log":"I0620 09:37:56.067974 1 server.go:96] Add webhook mutate path /mutate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067982785Z"}
{"log":"I0620 09:37:56.067976 1 server.go:101] Add webhook validate path /validate-core-openyurt-io-v1-pod\n","stream":"stderr","time":"2023-06-20T09:37:56.067984462Z"}
{"log":"I0620 09:37:56.067993 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"raven.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"Gateway\"} \"path\"=\"/mutate-raven-openyurt-io-v1alpha1-gateway\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068000995Z"}
{"log":"I0620 09:37:56.068008 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-raven-openyurt-io-v1alpha1-gateway\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068015325Z"}
{"log":"I0620 09:37:56.068020 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"raven.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"Gateway\"} \"path\"=\"/validate-raven-openyurt-io-v1alpha1-gateway\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068028917Z"}
{"log":"I0620 09:37:56.068033 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-raven-openyurt-io-v1alpha1-gateway\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068040289Z"}
{"log":"I0620 09:37:56.068051 1 server.go:96] Add webhook mutate path /mutate-raven-openyurt-io-v1alpha1-gateway\n","stream":"stderr","time":"2023-06-20T09:37:56.068058321Z"}
{"log":"I0620 09:37:56.068053 1 server.go:101] Add webhook validate path /validate-raven-openyurt-io-v1alpha1-gateway\n","stream":"stderr","time":"2023-06-20T09:37:56.06805996Z"}
{"log":"I0620 09:37:56.068059 1 deleg.go:130] controller-runtime/builder \"msg\"=\"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068066803Z"}
{"log":"I0620 09:37:56.068065 1 deleg.go:130] controller-runtime/builder \"msg\"=\"skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068074674Z"}
{"log":"I0620 09:37:56.068087 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/convert\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068093766Z"}
{"log":"I0620 09:37:56.068098 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Conversion webhook enabled\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068107036Z"}
{"log":"I0620 09:37:56.068100 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068108893Z"}
{"log":"I0620 09:37:56.068101 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068110053Z"}
{"log":"I0620 09:37:56.068108 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"} \"path\"=\"/mutate-apps-openyurt-io-v1beta1-nodepool\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068115075Z"}
{"log":"I0620 09:37:56.068122 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1beta1-nodepool\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068128885Z"}
{"log":"I0620 09:37:56.068133 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"} \"path\"=\"/validate-apps-openyurt-io-v1beta1-nodepool\"\n","stream":"stderr","time":"2023-06-20T09:37:56.06813993Z"}
{"log":"I0620 09:37:56.068145 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1beta1-nodepool\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06815212Z"}
{"log":"I0620 09:37:56.068161 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Conversion webhook enabled\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1beta1\",\"Kind\":\"NodePool\"}\n","stream":"stderr","time":"2023-06-20T09:37:56.068168586Z"}
{"log":"I0620 09:37:56.068163 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1beta1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068170337Z"}
{"log":"I0620 09:37:56.068164 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1beta1-nodepool\n","stream":"stderr","time":"2023-06-20T09:37:56.068171455Z"}
{"log":"I0620 09:37:56.068171 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtStaticSet\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtstaticset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068177968Z"}
{"log":"I0620 09:37:56.068184 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtstaticset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068191418Z"}
{"log":"I0620 09:37:56.068194 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtStaticSet\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtstaticset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.06820145Z"}
{"log":"I0620 09:37:56.068210 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtstaticset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068217292Z"}
{"log":"I0620 09:37:56.068226 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtstaticset\n","stream":"stderr","time":"2023-06-20T09:37:56.0682336Z"}
{"log":"I0620 09:37:56.068228 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtstaticset\n","stream":"stderr","time":"2023-06-20T09:37:56.068235404Z"}
{"log":"I0620 09:37:56.068235 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppSet\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068241789Z"}
{"log":"I0620 09:37:56.068260 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.06826784Z"}
{"log":"I0620 09:37:56.068270 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppSet\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappset\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068276754Z"}
{"log":"I0620 09:37:56.068282 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappset\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068289069Z"}
{"log":"I0620 09:37:56.068298 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtappset\n","stream":"stderr","time":"2023-06-20T09:37:56.068305527Z"}
{"log":"I0620 09:37:56.068300 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtappset\n","stream":"stderr","time":"2023-06-20T09:37:56.068307279Z"}
{"log":"I0620 09:37:56.068306 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a mutating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppDaemon\"} \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068313125Z"}
{"log":"I0620 09:37:56.068319 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068326251Z"}
{"log":"I0620 09:37:56.068328 1 deleg.go:130] controller-runtime/builder \"msg\"=\"Registering a validating webhook\" \"GVK\"={\"Group\":\"apps.openyurt.io\",\"Version\":\"v1alpha1\",\"Kind\":\"YurtAppDaemon\"} \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappdaemon\"\n","stream":"stderr","time":"2023-06-20T09:37:56.068335327Z"}
{"log":"I0620 09:37:56.068340 1 server.go:145] controller-runtime/webhook \"msg\"=\"Registering webhook\" \"path\"=\"/validate-apps-openyurt-io-v1alpha1-yurtappdaemon\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068354569Z"}
{"log":"I0620 09:37:56.068361 1 server.go:96] Add webhook mutate path /mutate-apps-openyurt-io-v1alpha1-yurtappdaemon\n","stream":"stderr","time":"2023-06-20T09:37:56.068368767Z"}
{"log":"I0620 09:37:56.068364 1 server.go:101] Add webhook validate path /validate-apps-openyurt-io-v1alpha1-yurtappdaemon\n","stream":"stderr","time":"2023-06-20T09:37:56.06837035Z"}
{"log":"I0620 09:37:56.068366 1 deleg.go:130] setup \"msg\"=\"initialize webhook\" \n","stream":"stderr","time":"2023-06-20T09:37:56.068373204Z"}
{"log":"I0620 09:37:56.068656 1 webhook_controller.go:151] Starting webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.068678265Z"}
{"log":"I0620 09:37:56.068663 1 shared_informer.go:240] Waiting for caches to sync for webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.068685978Z"}
{"log":"I0620 09:37:56.068693 1 reflector.go:219] Starting reflector *v1.MutatingWebhookConfiguration (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068721497Z"}
{"log":"I0620 09:37:56.068695 1 reflector.go:255] Listing and watching *v1.MutatingWebhookConfiguration from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068729093Z"}
{"log":"I0620 09:37:56.068700 1 reflector.go:219] Starting reflector *v1.Secret (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068731057Z"}
{"log":"I0620 09:37:56.068705 1 reflector.go:255] Listing and watching *v1.Secret from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068732774Z"}
{"log":"I0620 09:37:56.068709 1 reflector.go:219] Starting reflector *v1.ValidatingWebhookConfiguration (0s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068734332Z"}
{"log":"I0620 09:37:56.068719 1 reflector.go:255] Listing and watching *v1.ValidatingWebhookConfiguration from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:37:56.068735552Z"}
{"log":"I0620 09:37:56.069237 1 webhook_controller.go:108] MutatingWebhookConfiguration yurt-manager-mutating-webhook-configuration added\n","stream":"stderr","time":"2023-06-20T09:37:56.069253843Z"}
{"log":"I0620 09:37:56.069293 1 webhook_controller.go:125] ValidatingWebhookConfiguration yurt-manager-validating-webhook-configuration added\n","stream":"stderr","time":"2023-06-20T09:37:56.069311235Z"}
{"log":"I0620 09:37:56.071606 1 webhook_controller.go:91] Secret yurt-manager-webhook-certs added\n","stream":"stderr","time":"2023-06-20T09:37:56.071652185Z"}
{"log":"I0620 09:37:56.168975 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:37:56.169169946Z"}
{"log":"I0620 09:37:56.169018 1 shared_informer.go:247] Caches are synced for webhook-controller \n","stream":"stderr","time":"2023-06-20T09:37:56.169206027Z"}
{"log":"I0620 09:37:56.169032 1 webhook_controller.go:164] Started webhook-controller\n","stream":"stderr","time":"2023-06-20T09:37:56.169212852Z"}
{"log":"I0620 09:37:56.173027 1 fs.go:124] cert directory doesn't exist, creatingdirectory/tmp/yurt-manager-webhook-certs\n","stream":"stderr","time":"2023-06-20T09:37:56.173121294Z"}
{"log":"I0620 09:37:56.173700 1 atomic_writer.go:177] performed write of new data to ts data directoryts directory/tmp/yurt-manager-webhook-certs/..2023_06_20_09_37_56.3041147742\n","stream":"stderr","time":"2023-06-20T09:37:56.173861941Z"}
{"log":"I0620 09:37:56.180066 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/\"\n","stream":"stderr","time":"2023-06-20T09:37:56.18017204Z"}
{"log":"I0620 09:37:56.180097 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/profile\"\n","stream":"stderr","time":"2023-06-20T09:37:56.18020313Z"}
{"log":"I0620 09:37:56.180113 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/threadcreate\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180209977Z"}
{"log":"I0620 09:37:56.180133 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/block\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180216376Z"}
{"log":"I0620 09:37:56.180168 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/allocs\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180224324Z"}
{"log":"I0620 09:37:56.180185 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/cmdline\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180229648Z"}
{"log":"I0620 09:37:56.180199 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/symbol\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180254583Z"}
{"log":"I0620 09:37:56.180213 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/goroutine\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180261549Z"}
{"log":"I0620 09:37:56.180230 1 internal.go:259] setup \"msg\"=\"Registering metrics http server extra handler\" \"path\"=\"/debug/pprof/heap\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180266627Z"}
{"log":"I0620 09:37:56.180251 1 deleg.go:130] setup \"msg\"=\"starting manager\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180287598Z"}
{"log":"I0620 09:37:56.180397 1 leaderelection.go:248] attempting to acquire leader lease kube-system/yurt-manager...\n","stream":"stderr","time":"2023-06-20T09:37:56.180442922Z"}
{"log":"I0620 09:37:56.180434 1 server.go:213] controller-runtime/webhook/webhooks \"msg\"=\"Starting webhook server\" \n","stream":"stderr","time":"2023-06-20T09:37:56.18047928Z"}
{"log":"I0620 09:37:56.180450 1 deleg.go:130] setup \"msg\"=\"Starting metrics server\" \"path\"=\"/metrics\"\n","stream":"stderr","time":"2023-06-20T09:37:56.180486036Z"}
{"log":"I0620 09:37:56.180482 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:37:56.180552282Z"}
{"log":"I0620 09:37:56.180746 1 deleg.go:130] controller-runtime/certwatcher \"msg\"=\"Updated current TLS certificate\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180816995Z"}
{"log":"I0620 09:37:56.180854 1 deleg.go:130] controller-runtime/webhook \"msg\"=\"Serving webhook server\" \"host\"=\"0.0.0.0\" \"port\"=10273\n","stream":"stderr","time":"2023-06-20T09:37:56.180893764Z"}
{"log":"I0620 09:37:56.180923 1 deleg.go:130] controller-runtime/certwatcher \"msg\"=\"Starting certificate watcher\" \n","stream":"stderr","time":"2023-06-20T09:37:56.180954996Z"}
{"log":"I0620 09:37:56.184193 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:37:56.184289363Z"}
{"log":"I0620 09:37:56.184215 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:37:56.184317583Z"}
{"log":"I0620 09:37:59.116711 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:37:59.116873789Z"}
{"log":"I0620 09:37:59.116735 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:37:59.116936012Z"}
{"log":"I0620 09:38:01.989579 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:01.989724115Z"}
{"log":"I0620 09:38:01.989600 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:01.989771396Z"}
{"log":"I0620 09:38:04.571200 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:04.571328769Z"}
{"log":"I0620 09:38:04.571207 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:04.571349243Z"}
{"log":"I0620 09:38:07.233029 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:07.233231732Z"}
{"log":"I0620 09:38:07.233055 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:07.233283387Z"}
{"log":"I0620 09:38:10.893567 1 leaderelection.go:352] lock is held by yurt-manager-6bfdfdb4b8-zmclf_564af51e-2495-4a77-ad92-b43e0d528223 and has not yet expired\n","stream":"stderr","time":"2023-06-20T09:38:10.893704254Z"}
{"log":"I0620 09:38:10.893590 1 leaderelection.go:253] failed to acquire lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:10.893756806Z"}
{"log":"I0620 09:38:14.816789 1 leaderelection.go:258] successfully acquired lease kube-system/yurt-manager\n","stream":"stderr","time":"2023-06-20T09:38:14.816962856Z"}
{"log":"I0620 09:38:14.817048 1 recorder.go:103] setup/events \"msg\"=\"Normal\" \"message\"=\"yurt-manager-6bfdfdb4b8-zmclf_588ee144-ebe3-409a-8513-06ec78e0af9c became leader\" \"object\"={\"kind\":\"Lease\",\"namespace\":\"kube-system\",\"name\":\"yurt-manager\",\"uid\":\"d55bc2fc-d3a3-414a-a2b5-79b348ff5c0d\",\"apiVersion\":\"coordination.k8s.io/v1\",\"resourceVersion\":\"2526\"} \"reason\"=\"LeaderElection\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817222628Z"}
{"log":"I0620 09:38:14.817092 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtAppSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817275197Z"}
{"log":"I0620 09:38:14.817135 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.StatefulSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817289856Z"}
{"log":"I0620 09:38:14.817162 1 controller.go:178] setup/controller/yurtappset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Deployment\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817299257Z"}
{"log":"I0620 09:38:14.817178 1 controller.go:186] setup/controller/yurtappset \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817307408Z"}
{"log":"I0620 09:38:14.817190 1 controller.go:178] setup/controller/servicetopology-endpoints \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817334622Z"}
{"log":"I0620 09:38:14.817231 1 controller.go:178] setup/controller/servicetopology-endpoints \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817346118Z"}
{"log":"I0620 09:38:14.817246 1 controller.go:178] setup/controller/yurtappdaemon \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtAppDaemon\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817355086Z"}
{"log":"I0620 09:38:14.817255 1 controller.go:186] setup/controller/servicetopology-endpoints \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817362859Z"}
{"log":"I0620 09:38:14.817292 1 controller.go:178] setup/controller/yurtappdaemon \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.8174502Z"}
{"log":"I0620 09:38:14.817319 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817503128Z"}
{"log":"I0620 09:38:14.817327 1 controller.go:186] setup/controller/yurtappdaemon \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817517036Z"}
{"log":"I0620 09:38:14.817366 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.Gateway\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817526853Z"}
{"log":"I0620 09:38:14.817417 1 controller.go:178] setup/controller/nodepool \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817545878Z"}
{"log":"I0620 09:38:14.817396 1 controller.go:178] setup/controller/ravenl3-gateway \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.Gateway\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81755523Z"}
{"log":"I0620 09:38:14.817451 1 controller.go:178] setup/controller/nodepool \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817595379Z"}
{"log":"I0620 09:38:14.817481 1 controller.go:186] setup/controller/nodepool \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817604439Z"}
{"log":"I0620 09:38:14.817499 1 controller.go:178] setup/controller/ravenl3-gateway \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817612189Z"}
{"log":"I0620 09:38:14.817531 1 controller.go:186] setup/controller/ravenl3-gateway \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817623325Z"}
{"log":"I0620 09:38:14.817620 1 controller.go:178] setup/controller/daemonpodupdater \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.DaemonSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817733039Z"}
{"log":"I0620 09:38:14.817625 1 controller.go:178] setup/controller/ravenl3-service \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Endpoints\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817770262Z"}
{"log":"I0620 09:38:14.817643 1 controller.go:178] setup/controller/delegatelease \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Lease\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817779153Z"}
{"log":"I0620 09:38:14.817660 1 controller.go:186] setup/controller/delegatelease \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817785455Z"}
{"log":"I0620 09:38:14.817664 1 controller.go:186] setup/controller/ravenl3-service \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817791278Z"}
{"log":"I0620 09:38:14.817668 1 controller.go:178] setup/controller/daemonpodupdater \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.817796637Z"}
{"log":"I0620 09:38:14.817680 1 reflector.go:219] Starting reflector *v1alpha1.YurtAppSet (10h14m37.153933484s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.817802632Z"}
{"log":"I0620 09:38:14.817707 1 controller.go:186] setup/controller/daemonpodupdater \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.817807981Z"}
{"log":"I0620 09:38:14.817709 1 reflector.go:255] Listing and watching *v1alpha1.YurtAppSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.817813407Z"}
{"log":"I0620 09:38:14.817874 1 reflector.go:219] Starting reflector *v1alpha1.NodePool (9h45m5.513141062s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818013201Z"}
{"log":"I0620 09:38:14.817904 1 reflector.go:255] Listing and watching *v1alpha1.NodePool from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818029962Z"}
{"log":"I0620 09:38:14.817917 1 controller.go:178] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818038934Z"}
{"log":"I0620 09:38:14.817945 1 controller.go:178] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1beta1.NodePool\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818048123Z"}
{"log":"I0620 09:38:14.817977 1 controller.go:186] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818059512Z"}
{"log":"I0620 09:38:14.818083 1 controller.go:178] setup/controller/csrapprover \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.CertificateSigningRequest\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81820225Z"}
{"log":"I0620 09:38:14.818131 1 controller.go:186] setup/controller/csrapprover \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818230779Z"}
{"log":"I0620 09:38:14.818132 1 controller.go:178] setup/controller/podbinding \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818239649Z"}
{"log":"I0620 09:38:14.818152 1 controller.go:178] setup/controller/poolcoordinatorcert \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Service\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818249884Z"}
{"log":"I0620 09:38:14.818168 1 reflector.go:219] Starting reflector *v1.DaemonSet (9h44m12.954201241s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818259666Z"}
{"log":"I0620 09:38:14.818179 1 controller.go:186] setup/controller/podbinding \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818269246Z"}
{"log":"I0620 09:38:14.818193 1 controller.go:186] setup/controller/poolcoordinatorcert \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818280687Z"}
{"log":"I0620 09:38:14.818176 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1alpha1.YurtStaticSet\"\n","stream":"stderr","time":"2023-06-20T09:38:14.81828988Z"}
{"log":"I0620 09:38:14.818197 1 reflector.go:255] Listing and watching *v1.DaemonSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818300176Z"}
{"log":"I0620 09:38:14.818254 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Node\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818311904Z"}
{"log":"I0620 09:38:14.818297 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818387788Z"}
{"log":"I0620 09:38:14.818316 1 reflector.go:219] Starting reflector *v1beta1.NodePool (10h35m43.937429363s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818397953Z"}
{"log":"I0620 09:38:14.818330 1 reflector.go:255] Listing and watching *v1beta1.NodePool from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818407789Z"}
{"log":"I0620 09:38:14.818330 1 controller.go:178] setup/controller/yurtstaticset \"msg\"=\"Starting EventSource\" \"source\"=\"kind source: *v1.Pod\"\n","stream":"stderr","time":"2023-06-20T09:38:14.818417094Z"}
{"log":"I0620 09:38:14.818398 1 controller.go:186] setup/controller/yurtstaticset \"msg\"=\"Starting Controller\" \n","stream":"stderr","time":"2023-06-20T09:38:14.818485004Z"}
{"log":"I0620 09:38:14.818655 1 reflector.go:219] Starting reflector *v1.CertificateSigningRequest (10h18m40.649508617s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818825535Z"}
{"log":"I0620 09:38:14.818683 1 reflector.go:255] Listing and watching *v1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.818863017Z"}
{"log":"I0620 09:38:14.819000 1 reflector.go:219] Starting reflector *v1.Pod (10h40m56.107884968s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819111678Z"}
{"log":"I0620 09:38:14.819017 1 reflector.go:255] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819176208Z"}
{"log":"I0620 09:38:14.819031 1 reflector.go:219] Starting reflector *v1alpha1.Gateway (10h13m43.424381342s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819209451Z"}
{"log":"I0620 09:38:14.819048 1 reflector.go:255] Listing and watching *v1alpha1.Gateway from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819215415Z"}
{"log":"I0620 09:38:14.819308 1 reflector.go:219] Starting reflector *v1.Node (9h57m37.876902086s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819367953Z"}
{"log":"I0620 09:38:14.819318 1 reflector.go:255] Listing and watching *v1.Node from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819383071Z"}
{"log":"I0620 09:38:14.819593 1 reflector.go:219] Starting reflector *v1.Service (10h22m2.577113809s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819725675Z"}
{"log":"I0620 09:38:14.819618 1 reflector.go:255] Listing and watching *v1.Service from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819750836Z"}
{"log":"I0620 09:38:14.819708 1 reflector.go:219] Starting reflector *v1.Deployment (9h28m17.742796923s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.819759726Z"}
{"log":"I0620 09:38:14.819718 1 reflector.go:255] Listing and watching *v1.Deployment from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.81976517Z"}
{"log":"I0620 09:38:14.819970 1 reflector.go:219] Starting reflector *v1alpha1.YurtAppDaemon (9h45m57.370062593s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820021069Z"}
{"log":"I0620 09:38:14.819980 1 reflector.go:255] Listing and watching *v1alpha1.YurtAppDaemon from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820032354Z"}
{"log":"I0620 09:38:14.820119 1 reflector.go:219] Starting reflector *v1.StatefulSet (10h43m3.156628121s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820178756Z"}
{"log":"I0620 09:38:14.820147 1 reflector.go:255] Listing and watching *v1.StatefulSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820189445Z"}
{"log":"I0620 09:38:14.820182 1 reflector.go:219] Starting reflector *v1.Endpoints (10h57m46.652276057s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820223982Z"}
{"log":"I0620 09:38:14.820190 1 reflector.go:255] Listing and watching *v1.Endpoints from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820235639Z"}
{"log":"I0620 09:38:14.820544 1 reflector.go:219] Starting reflector *v1.Lease (10h47m35.307689498s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820601307Z"}
{"log":"I0620 09:38:14.820566 1 reflector.go:255] Listing and watching *v1.Lease from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820609966Z"}
{"log":"I0620 09:38:14.820743 1 reflector.go:219] Starting reflector *v1alpha1.YurtStaticSet (9h43m54.542719979s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820830809Z"}
{"log":"I0620 09:38:14.820769 1 reflector.go:255] Listing and watching *v1alpha1.YurtStaticSet from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.820841711Z"}
{"log":"I0620 09:38:14.834451 1 warning_handler.go:65] KubeAPIWarningLogger \"msg\"=\"apps.openyurt.io/v1alpha1 NodePool is deprecated in v1.0.0+, unavailable in v1.2.0+; use apps.openyurt.io/v1beta1 NodePool\" \n","stream":"stderr","time":"2023-06-20T09:38:14.834521328Z"}
{"log":"I0620 09:38:14.917982 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918272743Z"}
{"log":"I0620 09:38:14.918066 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918321488Z"}
{"log":"I0620 09:38:14.918123 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918329879Z"}
{"log":"I0620 09:38:14.918207 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918350927Z"}
{"log":"I0620 09:38:14.918332 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918371417Z"}
{"log":"I0620 09:38:14.918352 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.91837771Z"}
{"log":"I0620 09:38:14.918336 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918391087Z"}
{"log":"I0620 09:38:14.918404 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.918655857Z"}
{"log":"I0620 09:38:14.919389 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919624581Z"}
{"log":"I0620 09:38:14.919478 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919662367Z"}
{"log":"I0620 09:38:14.919506 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919671475Z"}
{"log":"I0620 09:38:14.919512 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.91967784Z"}
{"log":"I0620 09:38:14.919573 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919726147Z"}
{"log":"I0620 09:38:14.919592 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919781951Z"}
{"log":"I0620 09:38:14.919595 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919792029Z"}
{"log":"I0620 09:38:14.919642 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919798118Z"}
{"log":"I0620 09:38:14.919655 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919805125Z"}
{"log":"I0620 09:38:14.919714 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919815321Z"}
{"log":"I0620 09:38:14.919723 1 nodepool_enqueue_handlers.go:46] nodepool: node(k8s-master) does not belong to any nodepool\n","stream":"stderr","time":"2023-06-20T09:38:14.919821538Z"}
{"log":"I0620 09:38:14.919737 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.919827743Z"}
{"log":"I0620 09:38:14.919747 1 nodepool_enqueue_handlers.go:46] nodepool: node(k8s-node1) does not belong to any nodepool\n","stream":"stderr","time":"2023-06-20T09:38:14.919834143Z"}
{"log":"I0620 09:38:14.919908 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92012736Z"}
{"log":"I0620 09:38:14.920144 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92044843Z"}
{"log":"I0620 09:38:14.920230 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920533891Z"}
{"log":"I0620 09:38:14.920324 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920548461Z"}
{"log":"I0620 09:38:14.920379 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92055515Z"}
{"log":"I0620 09:38:14.920544 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920646022Z"}
{"log":"I0620 09:38:14.920705 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920774373Z"}
{"log":"I0620 09:38:14.920413 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92087281Z"}
{"log":"I0620 09:38:14.920706 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920893714Z"}
{"log":"I0620 09:38:14.920852 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920958413Z"}
{"log":"I0620 09:38:14.920902 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920970578Z"}
{"log":"I0620 09:38:14.920911 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.920980574Z"}
{"log":"I0620 09:38:14.920974 1 controller.go:220] setup/controller/csrapprover \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.921274176Z"}
{"log":"I0620 09:38:14.921080 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921322931Z"}
{"log":"I0620 09:38:14.921089 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92133656Z"}
{"log":"I0620 09:38:14.921240 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921349969Z"}
{"log":"I0620 09:38:14.921249 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921412107Z"}
{"log":"I0620 09:38:14.921356 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.92143728Z"}
{"log":"I0620 09:38:14.921364 1 gateway_enqueue_handlers.go:42] ravenl3-gateway: node(k8s-node1) does not belong to any gateway\n","stream":"stderr","time":"2023-06-20T09:38:14.921448116Z"}
{"log":"I0620 09:38:14.921404 1 gateway_enqueue_handlers.go:42] ravenl3-gateway: node(k8s-master) does not belong to any gateway\n","stream":"stderr","time":"2023-06-20T09:38:14.92146422Z"}
{"log":"I0620 09:38:14.921429 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921475308Z"}
{"log":"I0620 09:38:14.921444 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921821866Z"}
{"log":"I0620 09:38:14.921467 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921837694Z"}
{"log":"I0620 09:38:14.921473 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921848036Z"}
{"log":"I0620 09:38:14.921497 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921857894Z"}
{"log":"I0620 09:38:14.921524 1 controller.go:220] setup/controller/poolcoordinatorcert \"msg\"=\"Starting workers\" \"worker count\"=1\n","stream":"stderr","time":"2023-06-20T09:38:14.921868175Z"}
{"log":"I0620 09:38:14.921490 1 controller.go:220] setup/controller/nodepool \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.921880883Z"}
{"log":"I0620 09:38:14.921530 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921892587Z"}
{"log":"I0620 09:38:14.921561 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921902099Z"}
{"log":"I0620 09:38:14.921567 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921911532Z"}
{"log":"I0620 09:38:14.921473 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921920909Z"}
{"log":"I0620 09:38:14.921576 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921930377Z"}
{"log":"I0620 09:38:14.921565 1 controller.go:220] setup/controller/servicetopology-endpoints \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.92197419Z"}
{"log":"I0620 09:38:14.921625 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921985679Z"}
{"log":"I0620 09:38:14.921657 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.921996083Z"}
{"log":"I0620 09:38:14.921664 1 controller.go:220] setup/controller/podbinding \"msg\"=\"Starting workers\" \"worker count\"=5\n","stream":"stderr","time":"2023-06-20T09:38:14.922005961Z"}
{"log":"I0620 09:38:14.921669 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922017087Z"}
{"log":"I0620 09:38:14.921682 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922026842Z"}
{"log":"I0620 09:38:14.921680 1 controller.go:220] setup/controller/daemonpodupdater \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922038457Z"}
{"log":"I0620 09:38:14.921667 1 controller.go:220] setup/controller/servicetopology-endpointslice \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922049126Z"}
{"log":"I0620 09:38:14.921718 1 csrapprover_controller.go:190] csr(csr-7m2zn) is approved\n","stream":"stderr","time":"2023-06-20T09:38:14.92206021Z"}
{"log":"I0620 09:38:14.921697 1 controller.go:220] setup/controller/yurtappset \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922071021Z"}
{"log":"I0620 09:38:14.921739 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922082349Z"}
{"log":"I0620 09:38:14.921748 1 controller.go:220] setup/controller/ravenl3-gateway \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922094138Z"}
{"log":"I0620 09:38:14.921725 1 shared_informer.go:270] caches populated\n","stream":"stderr","time":"2023-06-20T09:38:14.922105454Z"}
{"log":"I0620 09:38:14.921807 1 controller.go:220] setup/controller/yurtstaticset \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922115887Z"}
{"log":"I0620 09:38:14.921810 1 controller.go:220] setup/controller/delegatelease \"msg\"=\"Starting workers\" \"worker count\"=5\n","stream":"stderr","time":"2023-06-20T09:38:14.922129729Z"}
{"log":"I0620 09:38:14.921842 1 csrapprover_controller.go:190] csr(csr-5jhbk) is approved\n","stream":"stderr","time":"2023-06-20T09:38:14.922139975Z"}
{"log":"I0620 09:38:14.921882 1 controller.go:220] setup/controller/yurtappdaemon \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.92215051Z"}
{"log":"I0620 09:38:14.921885 1 controller.go:220] setup/controller/ravenl3-service \"msg\"=\"Starting workers\" \"worker count\"=3\n","stream":"stderr","time":"2023-06-20T09:38:14.922160841Z"}
{"log":"I0620 09:38:14.921936 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet etcd\n","stream":"stderr","time":"2023-06-20T09:38:14.922172103Z"}
{"log":"I0620 09:38:14.921971 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-scheduler\n","stream":"stderr","time":"2023-06-20T09:38:14.922182245Z"}
{"log":"I0620 09:38:14.922002 1 podbinding_controller.go:92] node request: k8s-master\n","stream":"stderr","time":"2023-06-20T09:38:14.922192952Z"}
{"log":"E0620 09:38:14.922034 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/etcd, YurtStaticSet.apps.openyurt.io \"etcd\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922310021Z"}
{"log":"I0620 09:38:14.921981 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-apiserver\n","stream":"stderr","time":"2023-06-20T09:38:14.9223587Z"}
{"log":"E0620 09:38:14.922109 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-scheduler, YurtStaticSet.apps.openyurt.io \"kube-scheduler\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922404573Z"}
{"log":"I0620 09:38:14.921980 1 podbinding_controller.go:92] node request: k8s-node1\n","stream":"stderr","time":"2023-06-20T09:38:14.922418774Z"}
{"log":"E0620 09:38:14.922210 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-apiserver, YurtStaticSet.apps.openyurt.io \"kube-apiserver\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922430098Z"}
{"log":"I0620 09:38:14.922259 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet yurt-hub\n","stream":"stderr","time":"2023-06-20T09:38:14.922442099Z"}
{"log":"I0620 09:38:14.922302 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet yurt-hub-cloud\n","stream":"stderr","time":"2023-06-20T09:38:14.922456356Z"}
{"log":"I0620 09:38:14.922216 1 yurtstaticset_controller.go:281] yurtstaticset: Reconcile YurtStaticSet kube-controller-manager\n","stream":"stderr","time":"2023-06-20T09:38:14.922535403Z"}
{"log":"E0620 09:38:14.922770 1 yurtstaticset_controller.go:286] Fail to get YurtStaticSet kube-system/kube-controller-manager, YurtStaticSet.apps.openyurt.io \"kube-controller-manager\" not found\n","stream":"stderr","time":"2023-06-20T09:38:14.922931382Z"}
{"log":"I0620 09:38:14.930081 1 reflector.go:219] Starting reflector *v1.ConfigMap (9h43m30.684598835s) from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.930240236Z"}
{"log":"I0620 09:38:14.930113 1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167\n","stream":"stderr","time":"2023-06-20T09:38:14.930290381Z"}
{"log":"I0620 09:38:14.932774 1 delegatelease_controller.go:160] detaint k8s-master: no key node.openyurt.io/unschedulable exists, nothing to do\n","stream":"stderr","time":"2023-06-20T09:38:14.93289449Z"}
{"log":"I0620 09:38:14.932791 1 delegatelease_controller.go:160] detaint k8s-node1: no key node.openyurt.io/unschedulable exists, nothing to do\n","stream":"stderr","time":"2023-06-20T09:38:14.932939712Z"}
{"log":"panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2023-06-20T09:38:14.948547409Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x163d13b]\n","stream":"stderr","time":"2023-06-20T09:38:14.948582511Z"}
{"log":"\n","stream":"stderr","time":"2023-06-20T09:38:14.948590955Z"}
{"log":"goroutine 656 [running]:\n","stream":"stderr","time":"2023-06-20T09:38:14.948596233Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).configureTolerationForPod(0xc00069dd40, 0xc0013f43e8, 0x0)\n","stream":"stderr","time":"2023-06-20T09:38:14.94860166Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:142 +0x21b\n","stream":"stderr","time":"2023-06-20T09:38:14.948607739Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).processNode(0xc0012102d0?, {0x1de4268?, 0xc001216480?}, 0xc001224000)\n","stream":"stderr","time":"2023-06-20T09:38:14.948613823Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:117 +0x33d\n","stream":"stderr","time":"2023-06-20T09:38:14.948619256Z"}
{"log":"github.com/openyurtio/openyurt/pkg/controller/poolcoordinator/podbinding.(*ReconcilePodBinding).Reconcile(0xc00069dd40, {0x1de4268, 0xc001216480}, {{{0x0, 0x0}, {0xc0000dc240, 0x9}}})\n","stream":"stderr","time":"2023-06-20T09:38:14.948630304Z"}
{"log":"\u0009/build/pkg/controller/poolcoordinator/podbinding/podbinding_controller.go:93 +0x367\n","stream":"stderr","time":"2023-06-20T09:38:14.948636175Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000ca3360, {0x1de4268, 0xc001216450}, {{{0x0?, 0x1aa05c0?}, {0xc0000dc240?, 0x1?}}})\n","stream":"stderr","time":"2023-06-20T09:38:14.948641315Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114 +0x222\n","stream":"stderr","time":"2023-06-20T09:38:14.948668673Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000ca3360, {0x1de41c0, 0xc0003b0a80}, {0x1966b00?, 0xc001330020?})\n","stream":"stderr","time":"2023-06-20T09:38:14.948677056Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311 +0x2e9\n","stream":"stderr","time":"2023-06-20T09:38:14.948682395Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000ca3360, {0x1de41c0, 0xc0003b0a80})\n","stream":"stderr","time":"2023-06-20T09:38:14.948687587Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266 +0x1d9\n","stream":"stderr","time":"2023-06-20T09:38:14.948692882Z"}
{"log":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n","stream":"stderr","time":"2023-06-20T09:38:14.948698034Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227 +0x85\n","stream":"stderr","time":"2023-06-20T09:38:14.948704943Z"}
{"log":"created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n","stream":"stderr","time":"2023-06-20T09:38:14.948710168Z"}
{"log":"\u0009/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:223 +0x309\n","stream":"stderr","time":"2023-06-20T09:38:14.948715195Z"}
@transparency066 The above bug is known to yurt-manager component, and has been fixed in the pull request: #1538 . you can use the latest image of yurt-manager or set the log level below 4 to skip this bug.
@transparency066 By the way, i have backported the above pull request into tag: v1.3.2 just now, and you also use yurt-manager:v1.3.2 after image released.
i will wait for the next version. now i use yurtadm instead. but there is a error when i executed yurtadm join 100.80.241.43:6443 --token=3j0skf.urcezt4uj4vtmvwi --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5
command.
container runtime is not running: output: time="2023-06-21T14:05:11+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
and the log of kubelet is like
Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml>
but if i use kubeadm it joins into clusters successfully. it seems something in yurtadm is wrong. how do i fix it?
and my k8s version is 1.23.0. is it too high to have compatibility problems?
i will wait for the next version. now i use yurtadm instead. but there is a error when i executed
yurtadm join 100.80.241.43:6443 --token=3j0skf.urcezt4uj4vtmvwi --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5
command.container runtime is not running: output: time="2023-06-21T14:05:11+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
and the log of kubelet is like
Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml>
but if i use kubeadm it joins into clusters successfully. it seems something in yurtadm is wrong. how do i fix it?
i'm sorry. i should use docker-runtime command. it works.
@transparency066 yurt-manager:v1.3.2 has been released, and the openyurt helm charts have been updated.
i install the yurt-manager again and still get the v1.3.0. any solution?
i install the yurt-manager again and still get the v1.3.0. any solution?
@transparency066 I think you need to update helm repo, and you can reference the helm tutorial here: https://helm.sh/docs/helm/helm_repo_update/
u are right! thank you very much.
@transparency066 It seems that your problem had been solved, and would you like to close this issue?
{"log":"I0619 08:31:11.977819 1 util.go:289] start proxying: get /apis/storage.k8s.io/v1/csinodes/k8s-node1, in flight requests: 4\n","stream":"stderr","time":"2023-06-19T08:31:11.978141501Z"} {"log":"E0619 08:31:11.977982 1 local.go:218] can not cache for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1\n","stream":"stderr","time":"2023-06-19T08:31:11.97822749Z"} {"log":"E0619 08:31:11.978030 1 local.go:88] could not proxy local for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1, can not cache for go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1\n","stream":"stderr","time":"2023-06-19T08:31:11.978236642Z"} {"log":"I0619 08:31:11.978115 1 util.go:248] go-http-client get csinodes: /apis/storage.k8s.io/v1/csinodes/k8s-node1 with status code 400, spent 169.116µs\n","stream":"stderr","time":"2023-06-19T08:31:11.978247369Z"}
when i reconfigure the kubelet drop-in and yurt-hub, the edge not seems to be not ready. And the logs in the yurthub in edge node is like above. I don't understand what the mistake is. But in the master node, i get the all pods including yurthub running.