Pods unable to reach 10.43.0.1:443 even with firewall disabled

[Stolz]

Environmental Info:

K3s Version:

$ k3s --version
k3s version v1.25.4+k3s- ()
go version go1.22.2

Node(s) CPU architecture, OS, and Version:

$ cat /etc/lsb-release
DISTRIB_ID="Gentoo"

$ uname -a
Linux solid 6.0.7-gentoo-solid-stolz #4 SMP Sat Nov 5 19:03:13 HKT 2022 x86_64 AMD Ryzen 7 5700G with Radeon Graphics AuthenticAMD GNU/Linux

$ uptime # Long uptime, hence the old kernel version in use
 17:49:54 up 350 days,  7:16,  4 users,  load average: 0.78, 0.79, 0.55

$ iptables --version
iptables v1.8.10 (legacy)

Cluster Configuration: Single node server.

$ cat /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: "0640"

$ env | grep K3S_ # No output because no K3s env variables have been defined

Describe the bug:

Pods from default addons cannot connect to https://10.43.0.1:443.

Steps To Reproduce:

• Flush iptables firewall and reset default policy to allow all traffic
• Install K3s
• Start K3s server
• Check status of pods in kube-system namespace

Expected behavior:

All default addons from /var/lib/rancher/k3s/server/manifests should be up and running. If any iptables extension is missing it should be catched by check-config.sh script.

Actual behavior:

coredns pod never reaches ready staus. local-path-provisioner and metrics-server pods enter CrashLoopBackOff status. All the failing pods show an error related to unable to connect to https://10.43.0.1:443. Server logs mention some iptables extension as missing.

Additional context / logs:

My system has a lot of iptables rules but for the sake of simplicity I have reproduced the issue with a firewall withot any rule and with a permissive default policy. These are all the steps I followed:

Install K3s from official Getoo repository

emerge -av sys-cluster/k3s

Check if there are any kernel options missing ...

``` $ wget -q https://raw.githubusercontent.com/k3s-io/k3s/master/contrib/util/check-config.sh $ modprobe configs $ sh check-config.sh Verifying binaries in .: - sha256sum: sha256sums unavailable - links: link list unavailable System: - /sbin iptables v1.8.10 (legacy): ok - swap: disabled - routes: ok Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 info: reading kernel config from /proc/config.gz ... Generally Necessary: - cgroup hierarchy: cgroups Hybrid mounted, cpuset|memory controllers status: good - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled (as module) - CONFIG_BRIDGE_NETFILTER: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module) - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: missing - CONFIG_NET_CLS_CGROUP: enabled (as module) - CONFIG_CGROUP_NET_PRIO: enabled - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: enabled - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_SET: enabled (as module) - CONFIG_IP_VS: enabled (as module) - CONFIG_IP_VS_NFCT: enabled - CONFIG_IP_VS_PROTO_TCP: enabled - CONFIG_IP_VS_PROTO_UDP: enabled - CONFIG_IP_VS_RR: enabled (as module) - CONFIG_EXT4_FS: enabled - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: enabled - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled (as module) - CONFIG_CRYPTO_GCM: enabled (as module) - CONFIG_CRYPTO_SEQIV: enabled (as module) - CONFIG_CRYPTO_GHASH: enabled (as module) - CONFIG_XFRM: missing - CONFIG_XFRM_USER: missing - CONFIG_XFRM_ALGO: missing - CONFIG_INET_ESP: missing - CONFIG_INET_XFRM_MODE_TRANSPORT: missing - Storage Drivers: - "overlay": - CONFIG_OVERLAY_FS: enabled (as module) STATUS: pass ```

Disable firewall (default policy allows all traffic)

$ iptables -P INPUT ACCEPT
$ iptables -P FORWARD ACCEPT
$ iptables -P OUTPUT ACCEPT
$ iptables -t nat -F
$ iptables -t mangle -F
$ iptables -F
$ iptables -X

$ iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Start K3s server

$ rm -f /var/log/k3s/k3s.log

$ /etc/init.d/k3s start
* Starting k3s ...

$ sleep 10s && /etc/init.d/k3s status
* status: started

Check iptables rules added by K3s ...

``` $ iptables -nvL Chain INPUT (policy ACCEPT 16833 packets, 3787K bytes) pkts bytes target prot opt in out source destination 1237 104K KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */ 13477 2499K KUBE-NODEPORTS 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes health check service ports */ 1237 104K KUBE-EXTERNAL-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */ 16833 3787K KUBE-ROUTER-INPUT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-router netpol - 4IA2OSFRMVNDXBVV */ 16833 3787K KUBE-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */ 0 0 KUBE-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ 0 0 KUBE-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ 0 0 KUBE-EXTERNAL-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */ 0 0 FLANNEL-FWD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */ 0 0 KUBE-ROUTER-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-router netpol - TEMCG2JMHZYE7H7T */ Chain OUTPUT (policy ACCEPT 17079 packets, 4687K bytes) pkts bytes target prot opt in out source destination 886 70988 KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */ 886 70988 KUBE-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ 17079 4687K KUBE-ROUTER-OUTPUT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-router netpol - VEAAIY32XVBHCSCY */ 17079 4687K KUBE-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 Chain FLANNEL-FWD (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT 0 -- * * 10.42.0.0/16 0.0.0.0/0 /* flanneld forward */ 0 0 ACCEPT 0 -- * * 0.0.0.0/0 10.42.0.0/16 /* flanneld forward */ Chain KUBE-EXTERNAL-SERVICES (2 references) pkts bytes target prot opt in out source destination Chain KUBE-FIREWALL (2 references) pkts bytes target prot opt in out source destination 0 0 DROP 0 -- * * !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT 0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 Chain KUBE-FORWARD (1 references) pkts bytes target prot opt in out source destination Chain KUBE-KUBELET-CANARY (0 references) pkts bytes target prot opt in out source destination Chain KUBE-NODEPORTS (1 references) pkts bytes target prot opt in out source destination Chain KUBE-NWPLCY-DEFAULT (0 references) pkts bytes target prot opt in out source destination 0 0 MARK 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* rule to mark traffic matching a network policy */ MARK or 0x10000 Chain KUBE-PROXY-CANARY (0 references) pkts bytes target prot opt in out source destination Chain KUBE-PROXY-FIREWALL (3 references) pkts bytes target prot opt in out source destination Chain KUBE-ROUTER-FORWARD (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000 Chain KUBE-ROUTER-INPUT (1 references) pkts bytes target prot opt in out source destination 0 0 RETURN 0 -- * * 0.0.0.0/0 10.43.0.0/16 /* allow traffic to primary cluster IP range - TZZOAXOCHPHEHX7M */ 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 0 0 RETURN 17 -- * * 0.0.0.0/0 0.0.0.0/0 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */ ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000 Chain KUBE-ROUTER-OUTPUT (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000 Chain KUBE-SERVICES (2 references) pkts bytes target prot opt in out source destination ```

Check pod status

$ kubectl get pods -n kube-system
NAME                                      READY   STATUS             RESTARTS      AGE
coredns-597584b69b-pwlmm                  0/1     Running            0             4m24s
helm-install-traefik-bskvm                1/1     Running            1 (73s ago)   4m23s
helm-install-traefik-crd-t7q8d            1/1     Running            1 (73s ago)   4m23s
local-path-provisioner-79f67d76f8-j4vcv   0/1     CrashLoopBackOff   4 (17s ago)   4m24s
metrics-server-5c8978b444-mhx2c           0/1     CrashLoopBackOff   4 (13s ago)   4m24s

Check failing pods logs

coredns pod ...

``` $ kubectl describe -n kube-system pod/coredns-597584b69b-pwlmm (...redacted for brevity...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14m default-scheduler Successfully assigned kube-system/coredns-597584b69b-pwlmm to solid Normal Pulled 14m kubelet Container image "rancher/mirrored-coredns-coredns:1.9.4" already present on machine Normal Created 14m kubelet Created container coredns Normal Started 14m kubelet Started container coredns Warning Unhealthy 4m19s (x310 over 14m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503 $ kubectl logs -n kube-system pod/coredns-597584b69b-pwlmm [INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d CoreDNS-1.9.4 linux/amd64, go1.19.1, 1f0a41a [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.43.0.1:443/version": dial tcp 10.43.0.1:443: i/o timeout ```

local-path-provisioner pod ...

``` $ kubectl describe -n kube-system pod/local-path-provisioner-79f67d76f8-j4vcv (...redacted for brevity...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 26m default-scheduler Successfully assigned kube-system/local-path-provisioner-79f67d76f8-j4vcv to solid Normal Pulled 23m (x5 over 26m) kubelet Container image "rancher/local-path-provisioner:v0.0.23" already present on machine Normal Created 23m (x5 over 26m) kubelet Created container local-path-provisioner Normal Started 23m (x5 over 26m) kubelet Started container local-path-provisioner Warning BackOff 82s (x96 over 25m) kubelet Back-off restarting failed container $ kubectl logs -n kube-system pod/local-path-provisioner-79f67d76f8-j4vcv time="2024-04-23T10:15:26Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get \"https://10.43.0.1:443/version?timeout=32s\": dial tcp 10.43.0.1:443: i/o timeout" ```

local-path-provisioner pod ...

``` $ kubectl describe -n kube-system pod/metrics-server-5c8978b444-mhx2c (...redacted for brevity...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28m default-scheduler Successfully assigned kube-system/metrics-server-5c8978b444-mhx2c to solid Warning Unhealthy 28m kubelet Readiness probe failed: Get "https://10.42.0.7:10250/readyz": read tcp 10.42.0.1:52682->10.42.0.7:10250: read: connection reset by peer Normal Created 28m (x2 over 28m) kubelet Created container metrics-server Normal Started 28m (x2 over 28m) kubelet Started container metrics-server Warning Unhealthy 28m (x13 over 28m) kubelet Readiness probe failed: Get "https://10.42.0.7:10250/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 28m (x5 over 28m) kubelet Readiness probe failed: Get "https://10.42.0.7:10250/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 13m kubelet Readiness probe failed: Get "https://10.42.0.7:10250/readyz": read tcp 10.42.0.1:54188->10.42.0.7:10250: read: connection reset by peer Normal Pulled 8m39s (x9 over 28m) kubelet Container image "rancher/mirrored-metrics-server:v0.6.1" already present on machine Warning BackOff 3m41s (x99 over 27m) kubelet Back-off restarting failed container $ kubectl logs -n kube-system pod/metrics-server-5c8978b444-mhx2c Error: unable to load configmap based request-header-client-ca-file: Get "https://10.43.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.43.0.1:443: i/o timeout ```

[Stolz]

Adding server logs as well sice original message was too long

K3s server logs ...

``` cat /var/log/k3s/k3s.log time="2024-04-23T18:32:32+08:00" level=info msg="Starting k3s v1.25.4+k3s- ()" time="2024-04-23T18:32:32+08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s" time="2024-04-23T18:32:32+08:00" level=info msg="Configuring database table schema and indexes, this may take a moment..." time="2024-04-23T18:32:32+08:00" level=info msg="Database tables and indexes are up to date" time="2024-04-23T18:32:32+08:00" level=info msg="Kine available at unix://kine.sock" time="2024-04-23T18:32:32+08:00" level=info msg="Reconciling bootstrap data between datastore and disk" time="2024-04-23T18:32:32+08:00" level=info msg="Tunnel server egress proxy mode: agent" time="2024-04-23T18:32:32+08:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" time="2024-04-23T18:32:32+08:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" time="2024-04-23T18:32:32+08:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259" time="2024-04-23T18:32:32+08:00" level=info msg="Waiting for API server to become available" time="2024-04-23T18:32:32+08:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" time="2024-04-23T18:32:32+08:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false" I0423 18:32:32.189884 15620 server.go:581] external host was not specified, using 192.168.0.9 time="2024-04-23T18:32:32+08:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token" I0423 18:32:32.190082 15620 server.go:171] Version: v1.25.4+k3s- I0423 18:32:32.190096 15620 server.go:173] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" time="2024-04-23T18:32:32+08:00" level=info msg="To join server node to cluster: k3s server -s https://192.168.0.9:6443 -t ${SERVER_NODE_TOKEN}" time="2024-04-23T18:32:32+08:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token" time="2024-04-23T18:32:32+08:00" level=info msg="To join agent node to cluster: k3s agent -s https://192.168.0.9:6443 -t ${AGENT_NODE_TOKEN}" time="2024-04-23T18:32:32+08:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml" time="2024-04-23T18:32:32+08:00" level=info msg="Run: k3s kubectl" I0423 18:32:32.198981 15620 shared_informer.go:255] Waiting for caches to sync for node_authorizer I0423 18:32:32.199462 15620 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0423 18:32:32.199471 15620 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I0423 18:32:32.199974 15620 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0423 18:32:32.199981 15620 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. {"level":"warn","ts":"2024-04-23T18:32:32.201+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b7d6c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} time="2024-04-23T18:32:32+08:00" level=info msg="certificate CN=solid signed by CN=k3s-server-ca@1713864012: notBefore=2024-04-23 09:20:12 +0000 UTC notAfter=2025-04-23 10:32:32 +0000 UTC" time="2024-04-23T18:32:32+08:00" level=info msg="certificate CN=system:node:solid,O=system:nodes signed by CN=k3s-client-ca@1713864012: notBefore=2024-04-23 09:20:12 +0000 UTC notAfter=2025-04-23 10:32:32 +0000 UTC" W0423 18:32:32.211272 15620 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. I0423 18:32:32.211744 15620 instance.go:261] Using reconciler: lease time="2024-04-23T18:32:32+08:00" level=info msg="Module overlay was already loaded" time="2024-04-23T18:32:32+08:00" level=info msg="Module nf_conntrack was already loaded" time="2024-04-23T18:32:32+08:00" level=info msg="Module br_netfilter was already loaded" time="2024-04-23T18:32:32+08:00" level=info msg="Module iptable_nat was already loaded" W0423 18:32:32.226966 15620 sysinfo.go:203] Nodes topology is not available, providing CPU topology time="2024-04-23T18:32:32+08:00" level=warning msg="Flannel is using external addresses with an insecure backend: vxlan. Please consider using an encrypting flannel backend." time="2024-04-23T18:32:32+08:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log" time="2024-04-23T18:32:32+08:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd" W0423 18:32:32.231203 15620 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/run/k3s/containerd/containerd.sock /run/k3s/containerd/containerd.sock 0 }. Err: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused". Reconnecting... I0423 18:32:32.269777 15620 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping. W0423 18:32:32.344638 15620 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.345367 15620 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.347019 15620 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources. W0423 18:32:32.349787 15620 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources. W0423 18:32:32.350662 15620 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.351415 15620 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.351436 15620 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.353425 15620 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.353433 15620 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.354137 15620 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.354143 15620 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.354162 15620 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources. W0423 18:32:32.356563 15620 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.356572 15620 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.357308 15620 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.357314 15620 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.359539 15620 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.361592 15620 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. W0423 18:32:32.363665 15620 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources. W0423 18:32:32.363672 15620 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources. W0423 18:32:32.364576 15620 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. W0423 18:32:32.365347 15620 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources. I0423 18:32:32.365849 15620 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0423 18:32:32.365856 15620 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. W0423 18:32:32.377071 15620 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. I0423 18:32:33.041143 15620 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt" I0423 18:32:33.041261 15620 secure_serving.go:210] Serving securely on 127.0.0.1:6444 I0423 18:32:33.041325 15620 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" I0423 18:32:33.041432 15620 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0423 18:32:33.041439 15620 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller I0423 18:32:33.041463 15620 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key" I0423 18:32:33.041495 15620 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0423 18:32:33.041529 15620 available_controller.go:491] Starting AvailableConditionController I0423 18:32:33.041541 15620 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I0423 18:32:33.041546 15620 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt" I0423 18:32:33.041569 15620 controller.go:80] Starting OpenAPI V3 AggregationController I0423 18:32:33.041592 15620 controller.go:83] Starting OpenAPI AggregationController I0423 18:32:33.041616 15620 apf_controller.go:300] Starting API Priority and Fairness config controller I0423 18:32:33.041641 15620 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt" I0423 18:32:33.041694 15620 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt" I0423 18:32:33.041696 15620 autoregister_controller.go:141] Starting autoregister controller I0423 18:32:33.041713 15620 cache.go:32] Waiting for caches to sync for autoregister controller I0423 18:32:33.041740 15620 apiservice_controller.go:97] Starting APIServiceRegistrationController I0423 18:32:33.041741 15620 customresource_discovery_controller.go:209] Starting DiscoveryController I0423 18:32:33.041747 15620 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0423 18:32:33.041767 15620 crdregistration_controller.go:111] Starting crd-autoregister controller I0423 18:32:33.041773 15620 shared_informer.go:255] Waiting for caches to sync for crd-autoregister I0423 18:32:33.041775 15620 controller.go:85] Starting OpenAPI controller I0423 18:32:33.041798 15620 controller.go:85] Starting OpenAPI V3 controller I0423 18:32:33.041812 15620 naming_controller.go:291] Starting NamingConditionController I0423 18:32:33.041824 15620 establishing_controller.go:76] Starting EstablishingController I0423 18:32:33.041836 15620 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController I0423 18:32:33.041859 15620 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I0423 18:32:33.041879 15620 crd_finalizer.go:266] Starting CRDFinalizer I0423 18:32:33.099974 15620 shared_informer.go:262] Caches are synced for node_authorizer I0423 18:32:33.141596 15620 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller I0423 18:32:33.141713 15620 cache.go:39] Caches are synced for AvailableConditionController controller I0423 18:32:33.141725 15620 apf_controller.go:305] Running API Priority and Fairness config worker I0423 18:32:33.142126 15620 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0423 18:32:33.142145 15620 cache.go:39] Caches are synced for autoregister controller I0423 18:32:33.142134 15620 shared_informer.go:262] Caches are synced for crd-autoregister time="2024-04-23T18:32:33+08:00" level=info msg="Containerd is now running" time="2024-04-23T18:32:33+08:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" time="2024-04-23T18:32:33+08:00" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=solid --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" time="2024-04-23T18:32:33+08:00" level=info msg="Handling backend connection request [solid]" time="2024-04-23T18:32:33+08:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" I0423 18:32:33.914054 15620 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0423 18:32:34.045605 15620 storage_scheduling.go:111] all system priority classes are created successfully or already exist. W0423 18:32:34.155220 15620 handler_proxy.go:105] no RequestInfo found in the context E0423 18:32:34.155247 15620 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]] I0423 18:32:34.155259 15620 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. W0423 18:32:34.155287 15620 handler_proxy.go:105] no RequestInfo found in the context E0423 18:32:34.155303 15620 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService I0423 18:32:34.156283 15620 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. Flag --cloud-provider has been deprecated, will be removed in 1.25 or later, in favor of removing cloud provider code from Kubelet. Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed. Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI. I0423 18:32:34.247025 15620 server.go:199] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" time="2024-04-23T18:32:34+08:00" level=info msg="Annotations and labels have already set on node: solid" I0423 18:32:34.247757 15620 server.go:408] "Kubelet version" kubeletVersion="v1.25.4+k3s-" I0423 18:32:34.247765 15620 server.go:410] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" time="2024-04-23T18:32:34+08:00" level=info msg="Starting flannel with backend vxlan" I0423 18:32:34.248447 15620 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt" W0423 18:32:34.249538 15620 sysinfo.go:203] Nodes topology is not available, providing CPU topology time="2024-04-23T18:32:34+08:00" level=info msg="Stopped tunnel to 127.0.0.1:6443" time="2024-04-23T18:32:34+08:00" level=info msg="Connecting to proxy" url="wss://192.168.0.9:6443/v1-k3s/connect" time="2024-04-23T18:32:34+08:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect" time="2024-04-23T18:32:34+08:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" time="2024-04-23T18:32:34+08:00" level=info msg="Flannel found PodCIDR assigned for node solid" time="2024-04-23T18:32:34+08:00" level=info msg="The interface enp3s0 with ipv4 address 192.168.0.9 will be used by flannel" time="2024-04-23T18:32:34+08:00" level=info msg="Tunnel authorizer set Kubelet Port 10250" I0423 18:32:34.250584 15620 kube.go:126] Waiting 10m0s for node controller to sync I0423 18:32:34.250605 15620 kube.go:420] Starting kube subnet manager time="2024-04-23T18:32:34+08:00" level=info msg="Handling backend connection request [solid]" I0423 18:32:34.252015 15620 server.go:655] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /" I0423 18:32:34.252180 15620 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] I0423 18:32:34.252225 15620 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none} I0423 18:32:34.252255 15620 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container" I0423 18:32:34.252261 15620 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true I0423 18:32:34.252294 15620 state_mem.go:36] "Initialized new in-memory state store" I0423 18:32:34.254032 15620 kubelet.go:381] "Attempting to sync node with API server" I0423 18:32:34.254043 15620 kubelet.go:270] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests" I0423 18:32:34.254065 15620 kubelet.go:281] "Adding apiserver pod source" I0423 18:32:34.254074 15620 apiserver.go:42] "Waiting for node sync before watching apiserver pods" I0423 18:32:34.254649 15620 kuberuntime_manager.go:240] "Container runtime initialized" containerRuntime="containerd" version="v1.6.8-k3s1" apiVersion="v1" I0423 18:32:34.254886 15620 server.go:1170] "Started kubelet" E0423 18:32:34.255342 15620 cri_stats_provider.go:452] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs" E0423 18:32:34.255362 15620 kubelet.go:1317] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem" I0423 18:32:34.255396 15620 server.go:155] "Starting to listen" address="0.0.0.0" port=10250 I0423 18:32:34.255441 15620 scope.go:115] "RemoveContainer" containerID="b0ecda05d4750ccb2d24807e58d68c7a51f76560366bef4c563c003c511c3815" I0423 18:32:34.255765 15620 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" I0423 18:32:34.255787 15620 volume_manager.go:293] "Starting Kubelet Volume Manager" I0423 18:32:34.255853 15620 desired_state_of_world_populator.go:149] "Desired state populator starts to run" I0423 18:32:34.256210 15620 server.go:438] "Adding debug handlers to kubelet server" W0423 18:32:34.256238 15620 iptables.go:221] Error checking iptables version, assuming version at least 1.4.11: executable file not found in $PATH I0423 18:32:34.277112 15620 scope.go:115] "RemoveContainer" containerID="1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3" I0423 18:32:34.295100 15620 controller.go:616] quota admission added evaluator for: leases.coordination.k8s.io I0423 18:32:34.302634 15620 kubelet_network_linux.go:63] "Initialized iptables rules." protocol=IPv4 E0423 18:32:34.302687 15620 kubelet_network_linux.go:83] "Failed to ensure that iptables hint chain exists" err="error creating chain \"KUBE-IPTABLES-HINT\": executable file not found in $PATH: " I0423 18:32:34.302695 15620 kubelet_network_linux.go:71] "Failed to initialize iptables rules; some functionality may be missing." protocol=IPv6 I0423 18:32:34.302707 15620 status_manager.go:161] "Starting to sync pod status with apiserver" I0423 18:32:34.302722 15620 kubelet.go:2010] "Starting kubelet main sync loop" E0423 18:32:34.302750 15620 kubelet.go:2034] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" I0423 18:32:34.355957 15620 kuberuntime_manager.go:1050] "Updating runtime config through cri with podcidr" CIDR="10.42.0.0/24" I0423 18:32:34.363355 15620 kubelet_network.go:60] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.42.0.0/24" I0423 18:32:34.363365 15620 kubelet_node_status.go:70] "Attempting to register node" node="solid" I0423 18:32:34.367431 15620 kubelet_node_status.go:108] "Node was previously registered" node="solid" I0423 18:32:34.367470 15620 kubelet_node_status.go:73] "Successfully registered node" node="solid" I0423 18:32:34.368510 15620 setters.go:545] "Node became not ready" node="solid" condition={Type:Ready Status:False LastHeartbeatTime:2024-04-23 18:32:34.368481389 +0800 HKT m=+2.237562317 LastTransitionTime:2024-04-23 18:32:34.368481389 +0800 HKT m=+2.237562317 Reason:KubeletNotReady Message:container runtime status check may not have completed yet} I0423 18:32:34.386858 15620 cpu_manager.go:213] "Starting CPU manager" policy="none" I0423 18:32:34.386868 15620 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s" I0423 18:32:34.386880 15620 state_mem.go:36] "Initialized new in-memory state store" I0423 18:32:34.386964 15620 state_mem.go:88] "Updated default CPUSet" cpuSet="" I0423 18:32:34.386975 15620 state_mem.go:96] "Updated CPUSet assignments" assignments=map[] I0423 18:32:34.386981 15620 policy_none.go:49] "None policy: Start" I0423 18:32:34.387341 15620 memory_manager.go:168] "Starting memorymanager" policy="None" I0423 18:32:34.387351 15620 state_mem.go:35] "Initializing new in-memory state store" I0423 18:32:34.387413 15620 state_mem.go:75] "Updated machine memory state" I0423 18:32:34.387857 15620 manager.go:447] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found" I0423 18:32:34.387979 15620 plugin_manager.go:118] "Starting Kubelet Plugin Manager" I0423 18:32:34.403374 15620 scope.go:115] "RemoveContainer" containerID="1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3" E0423 18:32:34.404019 15620 remote_runtime.go:625] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3\": not found" containerID="1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3" I0423 18:32:34.404062 15620 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:containerd ID:1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3} err="failed to get container status \"1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3\": rpc error: code = NotFound desc = an error occurred when try to find container \"1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3\": not found" I0423 18:32:34.404080 15620 scope.go:115] "RemoveContainer" containerID="1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3" I0423 18:32:34.404540 15620 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:containerd ID:1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3} err="failed to get container status \"1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3\": rpc error: code = NotFound desc = an error occurred when try to find container \"1f90ef7b958f4cbe209bb93a6a22b04d9bce400b8973ac39a57c2a7407cc76d3\": not found" time="2024-04-23T18:32:34+08:00" level=info msg="Starting the netpol controller version v1.5.2-0.20221026101626-e01045262706, built on 2024-04-22T13:36:07Z, go1.22.2" I0423 18:32:34.475672 15620 network_policy_controller.go:163] Starting network policy controller I0423 18:32:34.492531 15620 network_policy_controller.go:175] Starting network policy controller full sync goroutine E0423 18:32:34.510925 15620 network_policy_controller.go:284] Aborting sync. Failed to run iptables-restore: failed to call iptables-restore: exit status 1 (Warning: Extension physdev revision 0 not supported, missing kernel module? Warning: Extension NFLOG revision 0 not supported, missing kernel module? Warning: Extension limit revision 0 not supported, missing kernel module? Warning: Extension REJECT revision 0 not supported, missing kernel module? iptables-restore: line 110 failed ) *filter :INPUT ACCEPT [2244:919380] - [0:0] :FORWARD ACCEPT [2:120] - [0:0] :OUTPUT ACCEPT [2255:960816] - [0:0] :KUBE-FIREWALL - [0:0] - [0:0] :KUBE-KUBELET-CANARY - [0:0] - [0:0] :KUBE-NWPLCY-DEFAULT - [0:0] - [0:0] :KUBE-ROUTER-FORWARD - [0:0] - [0:0] :KUBE-ROUTER-INPUT - [0:0] - [0:0] :KUBE-ROUTER-OUTPUT - [0:0] - [0:0] :KUBE-POD-FW-TYHU6IIERJPDEGRV - [0:0] :KUBE-POD-FW-FQUNU3C5ZHX4AEG5 - [0:0] :KUBE-POD-FW-WTFN6XE7KXIJSX7I - [0:0] :KUBE-POD-FW-M7LSKS7EAJSJVMFW - [0:0] :KUBE-POD-FW-KS7IBLLYYIFYJXB6 - [0:0] -A INPUT -m comment --comment "kube-router netpol - 4IA2OSFRMVNDXBVV" -j KUBE-ROUTER-INPUT -A INPUT -j KUBE-FIREWALL -A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD -A OUTPUT -m comment --comment "kube-router netpol - VEAAIY32XVBHCSCY" -j KUBE-ROUTER-OUTPUT -A OUTPUT -j KUBE-FIREWALL -A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP -A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP -A KUBE-NWPLCY-DEFAULT -m comment --comment "rule to mark traffic matching a network policy" -j MARK --set-xmark 0x10000/0x10000 -A KUBE-ROUTER-INPUT -d 10.43.0.0/16 -m comment --comment "allow traffic to primary cluster IP range - TZZOAXOCHPHEHX7M" -j RETURN -A KUBE-ROUTER-INPUT -p tcp -m comment --comment "allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN -A KUBE-ROUTER-INPUT -p udp -m comment --comment "allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN -I KUBE-POD-FW-TYHU6IIERJPDEGRV 1 -d 10.42.0.9 -m comment --comment "run through default ingress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-TYHU6IIERJPDEGRV 1 -s 10.42.0.9 -m comment --comment "run through default egress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-TYHU6IIERJPDEGRV 1 -m comment --comment "rule to permit the traffic to pods when source is the pod's local node" -m addrtype --src-type LOCAL -d 10.42.0.9 -j ACCEPT -I KUBE-POD-FW-TYHU6IIERJPDEGRV 1 -m comment --comment "rule to drop invalid state for pod" -m conntrack --ctstate INVALID -j DROP -I KUBE-POD-FW-TYHU6IIERJPDEGRV 1 -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic destined to POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -d 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic destined to POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -d 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic destined to POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -d 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-INPUT -m comment --comment "rule to jump traffic from POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -s 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic from POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -s 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic from POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -s 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic from POD name:coredns-597584b69b-pwlmm namespace: kube-system to chain KUBE-POD-FW-TYHU6IIERJPDEGRV" -s 10.42.0.9 -j KUBE-POD-FW-TYHU6IIERJPDEGRV -A KUBE-POD-FW-TYHU6IIERJPDEGRV -m comment --comment "rule to log dropped traffic POD name:coredns-597584b69b-pwlmm namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10 -A KUBE-POD-FW-TYHU6IIERJPDEGRV -m comment --comment "rule to REJECT traffic destined for POD name:coredns-597584b69b-pwlmm namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j REJECT -A KUBE-POD-FW-TYHU6IIERJPDEGRV -j MARK --set-mark 0/0x10000 -A KUBE-POD-FW-TYHU6IIERJPDEGRV -m comment --comment "set mark to ACCEPT traffic that comply to network policies" -j MARK --set-mark 0x20000/0x20000 -I KUBE-POD-FW-FQUNU3C5ZHX4AEG5 1 -d 10.42.0.10 -m comment --comment "run through default ingress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-FQUNU3C5ZHX4AEG5 1 -s 10.42.0.10 -m comment --comment "run through default egress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-FQUNU3C5ZHX4AEG5 1 -m comment --comment "rule to permit the traffic to pods when source is the pod's local node" -m addrtype --src-type LOCAL -d 10.42.0.10 -j ACCEPT -I KUBE-POD-FW-FQUNU3C5ZHX4AEG5 1 -m comment --comment "rule to drop invalid state for pod" -m conntrack --ctstate INVALID -j DROP -I KUBE-POD-FW-FQUNU3C5ZHX4AEG5 1 -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -d 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -d 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -d 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-INPUT -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -s 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -s 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -s 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-bskvm namespace: kube-system to chain KUBE-POD-FW-FQUNU3C5ZHX4AEG5" -s 10.42.0.10 -j KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -A KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -m comment --comment "rule to log dropped traffic POD name:helm-install-traefik-bskvm namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10 -A KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -m comment --comment "rule to REJECT traffic destined for POD name:helm-install-traefik-bskvm namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j REJECT -A KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -j MARK --set-mark 0/0x10000 -A KUBE-POD-FW-FQUNU3C5ZHX4AEG5 -m comment --comment "set mark to ACCEPT traffic that comply to network policies" -j MARK --set-mark 0x20000/0x20000 -I KUBE-POD-FW-WTFN6XE7KXIJSX7I 1 -d 10.42.0.11 -m comment --comment "run through default ingress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-WTFN6XE7KXIJSX7I 1 -s 10.42.0.11 -m comment --comment "run through default egress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-WTFN6XE7KXIJSX7I 1 -m comment --comment "rule to permit the traffic to pods when source is the pod's local node" -m addrtype --src-type LOCAL -d 10.42.0.11 -j ACCEPT -I KUBE-POD-FW-WTFN6XE7KXIJSX7I 1 -m comment --comment "rule to drop invalid state for pod" -m conntrack --ctstate INVALID -j DROP -I KUBE-POD-FW-WTFN6XE7KXIJSX7I 1 -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -d 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -d 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic destined to POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -d 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-INPUT -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -s 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -s 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -s 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic from POD name:helm-install-traefik-crd-t7q8d namespace: kube-system to chain KUBE-POD-FW-WTFN6XE7KXIJSX7I" -s 10.42.0.11 -j KUBE-POD-FW-WTFN6XE7KXIJSX7I -A KUBE-POD-FW-WTFN6XE7KXIJSX7I -m comment --comment "rule to log dropped traffic POD name:helm-install-traefik-crd-t7q8d namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10 -A KUBE-POD-FW-WTFN6XE7KXIJSX7I -m comment --comment "rule to REJECT traffic destined for POD name:helm-install-traefik-crd-t7q8d namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j REJECT -A KUBE-POD-FW-WTFN6XE7KXIJSX7I -j MARK --set-mark 0/0x10000 -A KUBE-POD-FW-WTFN6XE7KXIJSX7I -m comment --comment "set mark to ACCEPT traffic that comply to network policies" -j MARK --set-mark 0x20000/0x20000 -I KUBE-POD-FW-M7LSKS7EAJSJVMFW 1 -d 10.42.0.7 -m comment --comment "run through default ingress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-M7LSKS7EAJSJVMFW 1 -s 10.42.0.7 -m comment --comment "run through default egress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-M7LSKS7EAJSJVMFW 1 -m comment --comment "rule to permit the traffic to pods when source is the pod's local node" -m addrtype --src-type LOCAL -d 10.42.0.7 -j ACCEPT -I KUBE-POD-FW-M7LSKS7EAJSJVMFW 1 -m comment --comment "rule to drop invalid state for pod" -m conntrack --ctstate INVALID -j DROP -I KUBE-POD-FW-M7LSKS7EAJSJVMFW 1 -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic destined to POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -d 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic destined to POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -d 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic destined to POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -d 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-INPUT -m comment --comment "rule to jump traffic from POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -s 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic from POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -s 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic from POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -s 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic from POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system to chain KUBE-POD-FW-M7LSKS7EAJSJVMFW" -s 10.42.0.7 -j KUBE-POD-FW-M7LSKS7EAJSJVMFW -A KUBE-POD-FW-M7LSKS7EAJSJVMFW -m comment --comment "rule to log dropped traffic POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10 -A KUBE-POD-FW-M7LSKS7EAJSJVMFW -m comment --comment "rule to REJECT traffic destined for POD name:metrics-server-5c8978b444-mhx2c namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j REJECT -A KUBE-POD-FW-M7LSKS7EAJSJVMFW -j MARK --set-mark 0/0x10000 -A KUBE-POD-FW-M7LSKS7EAJSJVMFW -m comment --comment "set mark to ACCEPT traffic that comply to network policies" -j MARK --set-mark 0x20000/0x20000 -I KUBE-POD-FW-KS7IBLLYYIFYJXB6 1 -d 10.42.0.8 -m comment --comment "run through default ingress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-KS7IBLLYYIFYJXB6 1 -s 10.42.0.8 -m comment --comment "run through default egress network policy chain" -j KUBE-NWPLCY-DEFAULT -I KUBE-POD-FW-KS7IBLLYYIFYJXB6 1 -m comment --comment "rule to permit the traffic to pods when source is the pod's local node" -m addrtype --src-type LOCAL -d 10.42.0.8 -j ACCEPT -I KUBE-POD-FW-KS7IBLLYYIFYJXB6 1 -m comment --comment "rule to drop invalid state for pod" -m conntrack --ctstate INVALID -j DROP -I KUBE-POD-FW-KS7IBLLYYIFYJXB6 1 -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic destined to POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -d 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic destined to POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -d 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic destined to POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -d 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to jump traffic from POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -s 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-INPUT -m comment --comment "rule to jump traffic from POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -s 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-FORWARD -m comment --comment "rule to jump traffic from POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -s 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-ROUTER-FORWARD -m physdev --physdev-is-bridged -m comment --comment "rule to jump traffic from POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system to chain KUBE-POD-FW-KS7IBLLYYIFYJXB6" -s 10.42.0.8 -j KUBE-POD-FW-KS7IBLLYYIFYJXB6 -A KUBE-POD-FW-KS7IBLLYYIFYJXB6 -m comment --comment "rule to log dropped traffic POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10 -A KUBE-POD-FW-KS7IBLLYYIFYJXB6 -m comment --comment "rule to REJECT traffic destined for POD name:local-path-provisioner-79f67d76f8-j4vcv namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j REJECT -A KUBE-POD-FW-KS7IBLLYYIFYJXB6 -j MARK --set-mark 0/0x10000 -A KUBE-POD-FW-KS7IBLLYYIFYJXB6 -m comment --comment "set mark to ACCEPT traffic that comply to network policies" -j MARK --set-mark 0x20000/0x20000 -A KUBE-ROUTER-FORWARD -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT -A KUBE-ROUTER-OUTPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT -A KUBE-ROUTER-INPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT COMMIT time="2024-04-23T18:32:35+08:00" level=info msg="Kube API server is now running" time="2024-04-23T18:32:35+08:00" level=info msg="ETCD server is now running" time="2024-04-23T18:32:35+08:00" level=info msg="Waiting for cloud-controller-manager privileges to become available" time="2024-04-23T18:32:35+08:00" level=info msg="k3s is up and running" time="2024-04-23T18:32:35+08:00" level=info msg="Applying CRD addons.k3s.cattle.io" time="2024-04-23T18:32:35+08:00" level=info msg="Applying CRD helmcharts.helm.cattle.io" time="2024-04-23T18:32:35+08:00" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io" time="2024-04-23T18:32:35+08:00" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-19.0.400.tgz" time="2024-04-23T18:32:35+08:00" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-19.0.400.tgz" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml" time="2024-04-23T18:32:35+08:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" E0423 18:32:35.199055 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.200611 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:35.200928 15620 serving.go:355] Generated self-signed cert in-memory time="2024-04-23T18:32:35+08:00" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller" time="2024-04-23T18:32:35+08:00" level=info msg="Creating deploy event broadcaster" I0423 18:32:35.202139 15620 event.go:294] "Event occurred" object="kube-system/ccm" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\"" E0423 18:32:35.204065 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.206349 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.209396 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.212815 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:35+08:00" level=info msg="Starting /v1, Kind=Secret controller" time="2024-04-23T18:32:35+08:00" level=info msg="Creating helm-controller event broadcaster" time="2024-04-23T18:32:35+08:00" level=info msg="Updating TLS secret for kube-system/k3s-serving (count: 10): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.0.9:192.168.0.9 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-solid:solid listener.cattle.io/fingerprint:SHA1=EA1A9A31BCC70E0BD1F05321026E1114FE6C74CF]" time="2024-04-23T18:32:35+08:00" level=info msg="Cluster dns configmap already exists" I0423 18:32:35.228234 15620 serving.go:355] Generated self-signed cert in-memory I0423 18:32:35.229932 15620 controller.go:616] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io I0423 18:32:35.230925 15620 event.go:294] "Event occurred" object="kube-system/ccm" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\"" I0423 18:32:35.231954 15620 controller.go:616] quota admission added evaluator for: addons.k3s.cattle.io I0423 18:32:35.235349 15620 event.go:294] "Event occurred" object="kube-system/coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\"" I0423 18:32:35.251338 15620 kube.go:133] Node controller sync successful I0423 18:32:35.251367 15620 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false time="2024-04-23T18:32:35+08:00" level=info msg="Wrote flannel subnet file to /run/flannel/subnet.env" time="2024-04-23T18:32:35+08:00" level=info msg="Running flannel backend." I0423 18:32:35.253754 15620 vxlan_network.go:61] watching for new subnet leases I0423 18:32:35.254150 15620 apiserver.go:52] "Watching apiserver" I0423 18:32:35.255712 15620 topology_manager.go:205] "Topology Admit Handler" I0423 18:32:35.255774 15620 topology_manager.go:205] "Topology Admit Handler" I0423 18:32:35.255797 15620 topology_manager.go:205] "Topology Admit Handler" I0423 18:32:35.255828 15620 topology_manager.go:205] "Topology Admit Handler" I0423 18:32:35.255862 15620 topology_manager.go:205] "Topology Admit Handler" I0423 18:32:35.257912 15620 iptables.go:260] bootstrap done I0423 18:32:35.259101 15620 iptables.go:260] bootstrap done I0423 18:32:35.260748 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"content\" (UniqueName: \"kubernetes.io/configmap/3b3b200f-15bf-49ac-be5a-0b754801b204-content\") pod \"helm-install-traefik-bskvm\" (UID: \"3b3b200f-15bf-49ac-be5a-0b754801b204\") " pod="kube-system/helm-install-traefik-bskvm" I0423 18:32:35.260769 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtsk\" (UniqueName: \"kubernetes.io/projected/3b3b200f-15bf-49ac-be5a-0b754801b204-kube-api-access-kxtsk\") pod \"helm-install-traefik-bskvm\" (UID: \"3b3b200f-15bf-49ac-be5a-0b754801b204\") " pod="kube-system/helm-install-traefik-bskvm" I0423 18:32:35.260786 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"values\" (UniqueName: \"kubernetes.io/configmap/e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b-values\") pod \"helm-install-traefik-crd-t7q8d\" (UID: \"e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b\") " pod="kube-system/helm-install-traefik-crd-t7q8d" I0423 18:32:35.260802 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kh2\" (UniqueName: \"kubernetes.io/projected/e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b-kube-api-access-h2kh2\") pod \"helm-install-traefik-crd-t7q8d\" (UID: \"e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b\") " pod="kube-system/helm-install-traefik-crd-t7q8d" I0423 18:32:35.260825 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c385e-e43e-4086-9f9a-3237125c6e9b-config-volume\") pod \"local-path-provisioner-79f67d76f8-j4vcv\" (UID: \"7a8c385e-e43e-4086-9f9a-3237125c6e9b\") " pod="kube-system/local-path-provisioner-79f67d76f8-j4vcv" I0423 18:32:35.260844 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a4e41d-fd27-44d7-b881-28beb20f3bd7-config-volume\") pod \"coredns-597584b69b-pwlmm\" (UID: \"b8a4e41d-fd27-44d7-b881-28beb20f3bd7\") " pod="kube-system/coredns-597584b69b-pwlmm" I0423 18:32:35.260862 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw28h\" (UniqueName: \"kubernetes.io/projected/b8a4e41d-fd27-44d7-b881-28beb20f3bd7-kube-api-access-jw28h\") pod \"coredns-597584b69b-pwlmm\" (UID: \"b8a4e41d-fd27-44d7-b881-28beb20f3bd7\") " pod="kube-system/coredns-597584b69b-pwlmm" I0423 18:32:35.260875 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"values\" (UniqueName: \"kubernetes.io/configmap/3b3b200f-15bf-49ac-be5a-0b754801b204-values\") pod \"helm-install-traefik-bskvm\" (UID: \"3b3b200f-15bf-49ac-be5a-0b754801b204\") " pod="kube-system/helm-install-traefik-bskvm" I0423 18:32:35.260885 15620 controller.go:616] quota admission added evaluator for: deployments.apps I0423 18:32:35.260889 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"content\" (UniqueName: \"kubernetes.io/configmap/e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b-content\") pod \"helm-install-traefik-crd-t7q8d\" (UID: \"e5441f52-e7c4-4fe1-9d60-9acef7dc2c8b\") " pod="kube-system/helm-install-traefik-crd-t7q8d" I0423 18:32:35.260905 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b64c32b-c536-4fe7-8bb2-8f44fde12f1a-tmp-dir\") pod \"metrics-server-5c8978b444-mhx2c\" (UID: \"8b64c32b-c536-4fe7-8bb2-8f44fde12f1a\") " pod="kube-system/metrics-server-5c8978b444-mhx2c" I0423 18:32:35.260924 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgq9\" (UniqueName: \"kubernetes.io/projected/8b64c32b-c536-4fe7-8bb2-8f44fde12f1a-kube-api-access-xrgq9\") pod \"metrics-server-5c8978b444-mhx2c\" (UID: \"8b64c32b-c536-4fe7-8bb2-8f44fde12f1a\") " pod="kube-system/metrics-server-5c8978b444-mhx2c" I0423 18:32:35.260943 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdkp\" (UniqueName: \"kubernetes.io/projected/7a8c385e-e43e-4086-9f9a-3237125c6e9b-kube-api-access-9tdkp\") pod \"local-path-provisioner-79f67d76f8-j4vcv\" (UID: \"7a8c385e-e43e-4086-9f9a-3237125c6e9b\") " pod="kube-system/local-path-provisioner-79f67d76f8-j4vcv" I0423 18:32:35.260956 15620 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a4e41d-fd27-44d7-b881-28beb20f3bd7-custom-config-volume\") pod \"coredns-597584b69b-pwlmm\" (UID: \"b8a4e41d-fd27-44d7-b881-28beb20f3bd7\") " pod="kube-system/coredns-597584b69b-pwlmm" I0423 18:32:35.260965 15620 reconciler.go:169] "Reconciler: start to sync state" I0423 18:32:35.265614 15620 event.go:294] "Event occurred" object="kube-system/coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\"" I0423 18:32:35.268734 15620 event.go:294] "Event occurred" object="kube-system/local-storage" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\"" I0423 18:32:35.283611 15620 event.go:294] "Event occurred" object="kube-system/local-storage" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\"" I0423 18:32:35.284661 15620 serving.go:355] Generated self-signed cert in-memory I0423 18:32:35.286147 15620 event.go:294] "Event occurred" object="kube-system/aggregated-metrics-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\"" I0423 18:32:35.289140 15620 event.go:294] "Event occurred" object="kube-system/aggregated-metrics-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\"" time="2024-04-23T18:32:35+08:00" level=warning msg="Error ensuring node password secret for pre-validated node 'solid': unable to verify hash for node 'solid': hash does not match" I0423 18:32:35.291279 15620 event.go:294] "Event occurred" object="kube-system/auth-delegator" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\"" I0423 18:32:35.293870 15620 event.go:294] "Event occurred" object="kube-system/auth-delegator" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\"" time="2024-04-23T18:32:35+08:00" level=info msg="Starting /v1, Kind=Node controller" time="2024-04-23T18:32:35+08:00" level=info msg="Starting /v1, Kind=ConfigMap controller" time="2024-04-23T18:32:35+08:00" level=info msg="Starting /v1, Kind=ServiceAccount controller" I0423 18:32:35.296106 15620 event.go:294] "Event occurred" object="kube-system/auth-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\"" time="2024-04-23T18:32:35+08:00" level=info msg="Labels and annotations have been set successfully on node: solid" I0423 18:32:35.297954 15620 event.go:294] "Event occurred" object="kube-system/auth-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\"" I0423 18:32:35.300637 15620 event.go:294] "Event occurred" object="kube-system/metrics-apiservice" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\"" I0423 18:32:35.304423 15620 event.go:294] "Event occurred" object="kube-system/metrics-apiservice" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\"" I0423 18:32:35.306756 15620 event.go:294] "Event occurred" object="kube-system/metrics-server-deployment" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\"" time="2024-04-23T18:32:35+08:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller" time="2024-04-23T18:32:35+08:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller" I0423 18:32:35.315266 15620 event.go:294] "Event occurred" object="kube-system/traefik-crd" fieldPath="" kind="HelmChart" apiVersion="helm.cattle.io/v1" type="Normal" reason="ApplyJob" message="Applying HelmChart using Job kube-system/helm-install-traefik-crd" I0423 18:32:35.315285 15620 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="HelmChart" apiVersion="helm.cattle.io/v1" type="Normal" reason="ApplyJob" message="Applying HelmChart using Job kube-system/helm-install-traefik" I0423 18:32:35.316798 15620 event.go:294] "Event occurred" object="kube-system/metrics-server-deployment" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\"" E0423 18:32:35.318528 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.320605 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:35.320894 15620 event.go:294] "Event occurred" object="kube-system/metrics-server-service" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\"" time="2024-04-23T18:32:35+08:00" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller" I0423 18:32:35.323953 15620 event.go:294] "Event occurred" object="kube-system/metrics-server-service" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\"" E0423 18:32:35.324515 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:35.325526 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:35+08:00" level=info msg="Starting batch/v1, Kind=Job controller" I0423 18:32:35.326516 15620 event.go:294] "Event occurred" object="kube-system/resource-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\"" I0423 18:32:35.332774 15620 event.go:294] "Event occurred" object="kube-system/resource-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\"" I0423 18:32:35.423056 15620 controllermanager.go:145] Version: v1.25.4+k3s- I0423 18:32:35.424743 15620 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0423 18:32:35.424746 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0423 18:32:35.424753 15620 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController I0423 18:32:35.424755 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.424771 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0423 18:32:35.424782 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.424868 15620 secure_serving.go:210] Serving securely on 127.0.0.1:10258 I0423 18:32:35.424948 15620 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0423 18:32:35.496668 15620 event.go:294] "Event occurred" object="kube-system/rolebindings" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\"" I0423 18:32:35.497554 15620 controllermanager.go:178] Version: v1.25.4+k3s- I0423 18:32:35.497564 15620 controllermanager.go:180] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" I0423 18:32:35.499998 15620 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0423 18:32:35.500011 15620 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController I0423 18:32:35.500048 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0423 18:32:35.500048 15620 secure_serving.go:210] Serving securely on 127.0.0.1:10257 I0423 18:32:35.500055 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.500055 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0423 18:32:35.500071 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.500278 15620 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0423 18:32:35.508818 15620 event.go:294] "Event occurred" object="kube-system/rolebindings" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\"" I0423 18:32:35.518986 15620 server.go:148] "Starting Kubernetes Scheduler" version="v1.25.4+k3s-" I0423 18:32:35.518995 15620 server.go:150] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" I0423 18:32:35.520607 15620 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0423 18:32:35.520609 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0423 18:32:35.520616 15620 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController I0423 18:32:35.520617 15620 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0423 18:32:35.520627 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.520619 15620 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.520700 15620 secure_serving.go:210] Serving securely on 127.0.0.1:10259 I0423 18:32:35.520770 15620 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0423 18:32:35.525376 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.525387 15620 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController I0423 18:32:35.525392 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.600653 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.600688 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.600743 15620 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController I0423 18:32:35.621321 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 18:32:35.621351 15620 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController I0423 18:32:35.621390 15620 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 18:32:35.697951 15620 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\"" I0423 18:32:35.701337 15620 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\"" I0423 18:32:36.454771 15620 request.go:682] Waited for 1.092380736s due to client-side throttling, not priority and fairness, request: POST:https://127.0.0.1:6443/api/v1/namespaces/kube-system/serviceaccounts/local-path-provisioner-service-account/token I0423 18:32:36.456322 15620 scope.go:115] "RemoveContainer" containerID="887459ec8feb302633b910f1a75a4073b4983a020e302a450a734a204a206d03" I0423 18:32:36.756223 15620 scope.go:115] "RemoveContainer" containerID="553d28bdf40ed381fb01ba4413a6135e4ec9b96b43ddc6855f013a2317c1b660" I0423 18:32:36.756304 15620 scope.go:115] "RemoveContainer" containerID="e6810303180ea10a9eb8f56b0645578cfe8ea7fa1ebd1f24133c4202514a209c" E0423 18:32:36.829237 15620 controllermanager.go:476] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:36+08:00" level=info msg="Creating service-controller event broadcaster" I0423 18:32:36.831111 15620 controller.go:616] quota admission added evaluator for: namespaces I0423 18:32:36.833514 15620 controller.go:616] quota admission added evaluator for: serviceaccounts E0423 18:32:36.904902 15620 controllermanager.go:475] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:36.905358 15620 shared_informer.go:255] Waiting for caches to sync for tokens I0423 18:32:36.906942 15620 controllermanager.go:603] Started "serviceaccount" I0423 18:32:36.906997 15620 serviceaccounts_controller.go:117] Starting service account controller I0423 18:32:36.907005 15620 shared_informer.go:255] Waiting for caches to sync for service account I0423 18:32:36.908587 15620 controllermanager.go:603] Started "deployment" W0423 18:32:36.908595 15620 controllermanager.go:568] "bootstrapsigner" is disabled I0423 18:32:36.908686 15620 deployment_controller.go:160] "Starting controller" controller="deployment" I0423 18:32:36.908692 15620 shared_informer.go:255] Waiting for caches to sync for deployment I0423 18:32:36.909968 15620 controllermanager.go:603] Started "ttl-after-finished" I0423 18:32:36.910052 15620 ttlafterfinished_controller.go:109] Starting TTL after finished controller I0423 18:32:36.910058 15620 shared_informer.go:255] Waiting for caches to sync for TTL after finished I0423 18:32:36.913791 15620 garbagecollector.go:154] Starting garbage collector controller I0423 18:32:36.913802 15620 shared_informer.go:255] Waiting for caches to sync for garbage collector I0423 18:32:36.913823 15620 graph_builder.go:291] GraphBuilder running I0423 18:32:36.913927 15620 controllermanager.go:603] Started "garbagecollector" I0423 18:32:36.915444 15620 controllermanager.go:603] Started "daemonset" I0423 18:32:36.915556 15620 daemon_controller.go:291] Starting daemon sets controller I0423 18:32:36.915565 15620 shared_informer.go:255] Waiting for caches to sync for daemon sets I0423 18:32:36.917085 15620 controllermanager.go:603] Started "persistentvolume-binder" I0423 18:32:36.917203 15620 pv_controller_base.go:318] Starting persistent volume controller I0423 18:32:36.917215 15620 shared_informer.go:255] Waiting for caches to sync for persistent volume I0423 18:32:36.918685 15620 controllermanager.go:603] Started "endpointslicemirroring" I0423 18:32:36.918786 15620 endpointslicemirroring_controller.go:216] Starting EndpointSliceMirroring controller I0423 18:32:36.918793 15620 shared_informer.go:255] Waiting for caches to sync for endpoint_slice_mirroring I0423 18:32:36.919842 15620 controllermanager.go:603] Started "csrcleaner" I0423 18:32:36.919902 15620 cleaner.go:82] Starting CSR cleaner controller I0423 18:32:36.923298 15620 controllermanager.go:603] Started "clusterrole-aggregation" I0423 18:32:36.923356 15620 clusterroleaggregation_controller.go:194] Starting ClusterRoleAggregator I0423 18:32:36.923364 15620 shared_informer.go:255] Waiting for caches to sync for ClusterRoleAggregator I0423 18:32:36.924628 15620 controllermanager.go:603] Started "statefulset" I0423 18:32:36.924759 15620 stateful_set.go:152] Starting stateful set controller I0423 18:32:36.924767 15620 shared_informer.go:255] Waiting for caches to sync for stateful set I0423 18:32:36.926664 15620 certificate_controller.go:112] Starting certificate controller "csrsigning-kubelet-serving" I0423 18:32:36.926673 15620 shared_informer.go:255] Waiting for caches to sync for certificate-csrsigning-kubelet-serving I0423 18:32:36.926687 15620 dynamic_serving_content.go:132] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/server-ca.crt::/var/lib/rancher/k3s/server/tls/server-ca.key" I0423 18:32:36.926816 15620 certificate_controller.go:112] Starting certificate controller "csrsigning-kubelet-client" I0423 18:32:36.926822 15620 shared_informer.go:255] Waiting for caches to sync for certificate-csrsigning-kubelet-client I0423 18:32:36.926837 15620 dynamic_serving_content.go:132] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key" I0423 18:32:36.926898 15620 certificate_controller.go:112] Starting certificate controller "csrsigning-kube-apiserver-client" I0423 18:32:36.926907 15620 shared_informer.go:255] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client I0423 18:32:36.926930 15620 dynamic_serving_content.go:132] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key" I0423 18:32:36.926942 15620 controllermanager.go:603] Started "csrsigning" I0423 18:32:36.927018 15620 certificate_controller.go:112] Starting certificate controller "csrsigning-legacy-unknown" I0423 18:32:36.927025 15620 shared_informer.go:255] Waiting for caches to sync for certificate-csrsigning-legacy-unknown I0423 18:32:36.927042 15620 dynamic_serving_content.go:132] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/server-ca.crt::/var/lib/rancher/k3s/server/tls/server-ca.key" W0423 18:32:36.927070 15620 garbagecollector.go:752] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request] I0423 18:32:36.928378 15620 controllermanager.go:603] Started "ttl" I0423 18:32:36.928435 15620 ttl_controller.go:120] Starting TTL controller I0423 18:32:36.928442 15620 shared_informer.go:255] Waiting for caches to sync for TTL I0423 18:32:36.929826 15620 controllermanager.go:603] Started "endpoint" I0423 18:32:36.929861 15620 endpoints_controller.go:182] Starting endpoint controller I0423 18:32:36.929869 15620 shared_informer.go:255] Waiting for caches to sync for endpoint I0423 18:32:36.931112 15620 controllermanager.go:603] Started "replicationcontroller" I0423 18:32:36.931195 15620 replica_set.go:205] Starting replicationcontroller controller I0423 18:32:36.931201 15620 shared_informer.go:255] Waiting for caches to sync for ReplicationController I0423 18:32:36.932337 15620 controllermanager.go:603] Started "podgc" I0423 18:32:36.932408 15620 gc_controller.go:99] Starting GC controller I0423 18:32:36.932429 15620 shared_informer.go:255] Waiting for caches to sync for GC I0423 18:32:36.934909 15620 controllermanager.go:603] Started "job" I0423 18:32:36.935092 15620 job_controller.go:196] Starting job controller I0423 18:32:36.935102 15620 shared_informer.go:255] Waiting for caches to sync for job E0423 18:32:36.940591 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:36.940899 15620 controllermanager.go:603] Started "horizontalpodautoscaling" W0423 18:32:36.940910 15620 controllermanager.go:568] "route" is disabled I0423 18:32:36.940986 15620 horizontal.go:168] Starting HPA controller I0423 18:32:36.940993 15620 shared_informer.go:255] Waiting for caches to sync for HPA E0423 18:32:36.942787 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:36+08:00" level=info msg="Starting /v1, Kind=Node controller" E0423 18:32:36.946478 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:36.948604 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:36+08:00" level=info msg="Starting /v1, Kind=Pod controller" E0423 18:32:36.953208 15620 resource_quota_controller.go:165] initial discovery check failure, continuing and counting on future sync update: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:36.953253 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for controllerrevisions.apps I0423 18:32:36.953285 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for leases.coordination.k8s.io I0423 18:32:36.953311 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io I0423 18:32:36.953321 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for limitranges I0423 18:32:36.953420 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for serviceaccounts I0423 18:32:36.953435 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for statefulsets.apps I0423 18:32:36.953450 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for deployments.apps I0423 18:32:36.953461 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for replicasets.apps I0423 18:32:36.953469 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for daemonsets.apps I0423 18:32:36.953484 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for addons.k3s.cattle.io I0423 18:32:36.953495 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling I0423 18:32:36.953506 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io I0423 18:32:36.953519 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io W0423 18:32:36.953532 15620 shared_informer.go:533] resyncPeriod 14h42m58.862098435s is smaller than resyncCheckPeriod 16h50m44.422279355s and the informer has already started. Changing it to 16h50m44.422279355s I0423 18:32:36.953566 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io I0423 18:32:36.953576 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for endpoints I0423 18:32:36.953587 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io I0423 18:32:36.953601 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io I0423 18:32:36.953624 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for jobs.batch I0423 18:32:36.953640 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io I0423 18:32:36.953656 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy I0423 18:32:36.953671 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for podtemplates I0423 18:32:36.953682 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for cronjobs.batch I0423 18:32:36.953694 15620 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for csistoragecapacities.storage.k8s.io I0423 18:32:36.953702 15620 controllermanager.go:603] Started "resourcequota" I0423 18:32:36.953748 15620 resource_quota_controller.go:277] Starting resource quota controller I0423 18:32:36.953757 15620 shared_informer.go:255] Waiting for caches to sync for resource quota I0423 18:32:36.953767 15620 resource_quota_monitor.go:295] QuotaMonitor running E0423 18:32:36.953790 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:36.954860 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request time="2024-04-23T18:32:36+08:00" level=info msg="Starting apps/v1, Kind=DaemonSet controller" I0423 18:32:36.955653 15620 controllermanager.go:301] Started "cloud-node-lifecycle" I0423 18:32:36.955760 15620 node_lifecycle_controller.go:113] Sending events to api server I0423 18:32:36.955810 15620 controllermanager.go:301] Started "service" W0423 18:32:36.955817 15620 controllermanager.go:278] "route" is disabled I0423 18:32:36.955892 15620 controllermanager.go:301] Started "cloud-node" I0423 18:32:36.955903 15620 controller.go:237] Starting service controller I0423 18:32:36.955916 15620 shared_informer.go:255] Waiting for caches to sync for service I0423 18:32:36.955970 15620 node_controller.go:157] Sending events to api server. I0423 18:32:36.955992 15620 node_controller.go:166] Waiting for informer caches to sync I0423 18:32:36.956081 15620 controllermanager.go:603] Started "csrapproving" I0423 18:32:36.956146 15620 certificate_controller.go:112] Starting certificate controller "csrapproving" I0423 18:32:36.956153 15620 shared_informer.go:255] Waiting for caches to sync for certificate-csrapproving E0423 18:32:36.957955 15620 resource_quota_controller.go:417] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:36.958281 15620 controllermanager.go:603] Started "pv-protection" I0423 18:32:36.958319 15620 pv_protection_controller.go:79] Starting PV protection controller I0423 18:32:36.958326 15620 shared_informer.go:255] Waiting for caches to sync for PV protection I0423 18:32:36.959569 15620 controllermanager.go:603] Started "ephemeral-volume" I0423 18:32:36.959598 15620 controller.go:169] Starting ephemeral volume controller I0423 18:32:36.959603 15620 shared_informer.go:255] Waiting for caches to sync for ephemeral I0423 18:32:37.005438 15620 shared_informer.go:262] Caches are synced for tokens I0423 18:32:37.007101 15620 node_lifecycle_controller.go:497] Controller will reconcile labels. I0423 18:32:37.007120 15620 controllermanager.go:603] Started "nodelifecycle" I0423 18:32:37.007157 15620 node_lifecycle_controller.go:532] Sending events to api server. I0423 18:32:37.007177 15620 node_lifecycle_controller.go:543] Starting node controller I0423 18:32:37.007181 15620 shared_informer.go:255] Waiting for caches to sync for taint I0423 18:32:37.056208 15620 shared_informer.go:262] Caches are synced for service I0423 18:32:37.057540 15620 controllermanager.go:603] Started "pvc-protection" W0423 18:32:37.057559 15620 controllermanager.go:568] "service" is disabled I0423 18:32:37.057601 15620 pvc_protection_controller.go:103] "Starting PVC protection controller" I0423 18:32:37.057612 15620 shared_informer.go:255] Waiting for caches to sync for PVC protection I0423 18:32:37.109356 15620 controllermanager.go:603] Started "persistentvolume-expander" I0423 18:32:37.109402 15620 expand_controller.go:340] Starting expand controller I0423 18:32:37.109407 15620 shared_informer.go:255] Waiting for caches to sync for expand I0423 18:32:37.158959 15620 controllermanager.go:603] Started "endpointslice" I0423 18:32:37.159047 15620 endpointslice_controller.go:261] Starting endpoint slice controller I0423 18:32:37.159097 15620 shared_informer.go:255] Waiting for caches to sync for endpoint_slice I0423 18:32:37.207374 15620 controllermanager.go:603] Started "replicaset" I0423 18:32:37.207424 15620 replica_set.go:205] Starting replicaset controller I0423 18:32:37.207430 15620 shared_informer.go:255] Waiting for caches to sync for ReplicaSet I0423 18:32:37.308808 15620 controllermanager.go:603] Started "disruption" W0423 18:32:37.308825 15620 controllermanager.go:568] "tokencleaner" is disabled I0423 18:32:37.308875 15620 disruption.go:421] Sending events to api server. I0423 18:32:37.308897 15620 disruption.go:432] Starting disruption controller I0423 18:32:37.308904 15620 shared_informer.go:255] Waiting for caches to sync for disruption I0423 18:32:37.356978 15620 scope.go:115] "RemoveContainer" containerID="91f139a5093542d5e7ed934463b3ec49d24b04a544c00a729ff3b2be6b89910d" I0423 18:32:37.358606 15620 node_ipam_controller.go:91] Sending events to api server. time="2024-04-23T18:32:38+08:00" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=solid --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables" I0423 18:32:38.243429 15620 server.go:230] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP" I0423 18:32:38.244947 15620 proxier.go:666] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_wrr" I0423 18:32:38.245348 15620 proxier.go:666] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_sh" I0423 18:32:38.249663 15620 node.go:163] Successfully retrieved node IP: 192.168.0.9 I0423 18:32:38.249674 15620 server_others.go:138] "Detected node IP" address="192.168.0.9" W0423 18:32:38.250020 15620 iptables.go:221] Error checking iptables version, assuming version at least 1.4.11: executable file not found in $PATH I0423 18:32:38.250486 15620 server_others.go:199] "kube-proxy running in single-stack mode, this ipFamily is not supported" ipFamily=IPv6 I0423 18:32:38.250494 15620 server_others.go:206] "Using iptables Proxier" I0423 18:32:38.250501 15620 proxier.go:262] "Setting route_localnet=1, use nodePortAddresses to filter loopback addresses for NodePorts to skip it https://issues.k8s.io/90259" I0423 18:32:38.250646 15620 server.go:661] "Version info" version="v1.25.4+k3s-" I0423 18:32:38.250654 15620 server.go:663] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" I0423 18:32:38.251002 15620 config.go:226] "Starting endpoint slice config controller" I0423 18:32:38.251013 15620 shared_informer.go:255] Waiting for caches to sync for endpoint slice config I0423 18:32:38.251032 15620 config.go:317] "Starting service config controller" I0423 18:32:38.251032 15620 config.go:444] "Starting node config controller" I0423 18:32:38.251040 15620 shared_informer.go:255] Waiting for caches to sync for service config I0423 18:32:38.251041 15620 shared_informer.go:255] Waiting for caches to sync for node config I0423 18:32:38.352095 15620 shared_informer.go:262] Caches are synced for node config I0423 18:32:38.352326 15620 shared_informer.go:262] Caches are synced for service config I0423 18:32:38.352372 15620 shared_informer.go:262] Caches are synced for endpoint slice config E0423 18:32:38.387944 15620 proxier.go:1504] "Failed to execute iptables-restore" err=< exit status 1: Ignoring deprecated --wait-interval option. Warning: Extension REJECT revision 0 not supported, missing kernel module? iptables-restore: line 14 failed > I0423 18:32:38.387962 15620 proxier.go:855] "Sync failed" retryingTime="30s" I0423 18:32:47.364315 15620 range_allocator.go:76] Sending events to api server. I0423 18:32:47.364387 15620 range_allocator.go:110] No Secondary Service CIDR provided. Skipping filtering out secondary service addresses. I0423 18:32:47.364413 15620 controllermanager.go:603] Started "nodeipam" I0423 18:32:47.364481 15620 node_ipam_controller.go:154] Starting ipam controller I0423 18:32:47.364488 15620 shared_informer.go:255] Waiting for caches to sync for node E0423 18:32:47.373876 15620 namespaced_resources_deleter.go:162] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:47.373924 15620 controllermanager.go:603] Started "namespace" I0423 18:32:47.373989 15620 namespace_controller.go:200] Starting namespace controller I0423 18:32:47.373997 15620 shared_informer.go:255] Waiting for caches to sync for namespace I0423 18:32:47.375153 15620 controllermanager.go:603] Started "cronjob" W0423 18:32:47.375161 15620 controllermanager.go:568] "cloud-node-lifecycle" is disabled I0423 18:32:47.375171 15620 cronjob_controllerv2.go:135] "Starting cronjob controller v2" I0423 18:32:47.375181 15620 shared_informer.go:255] Waiting for caches to sync for cronjob I0423 18:32:47.376432 15620 controllermanager.go:603] Started "attachdetach" I0423 18:32:47.376488 15620 attach_detach_controller.go:328] Starting attach detach controller I0423 18:32:47.376504 15620 shared_informer.go:255] Waiting for caches to sync for attach detach I0423 18:32:47.378931 15620 controllermanager.go:603] Started "root-ca-cert-publisher" I0423 18:32:47.379025 15620 publisher.go:107] Starting root CA certificate configmap publisher I0423 18:32:47.379035 15620 shared_informer.go:255] Waiting for caches to sync for crt configmap I0423 18:32:47.381996 15620 shared_informer.go:255] Waiting for caches to sync for resource quota W0423 18:32:47.385799 15620 actual_state_of_world.go:541] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="solid" does not exist I0423 18:32:47.386154 15620 job_controller.go:510] enqueueing job kube-system/helm-install-traefik-crd I0423 18:32:47.386168 15620 job_controller.go:510] enqueueing job kube-system/helm-install-traefik E0423 18:32:47.388998 15620 memcache.go:206] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0423 18:32:47.390809 15620 memcache.go:104] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request I0423 18:32:47.392172 15620 shared_informer.go:255] Waiting for caches to sync for garbage collector I0423 18:32:47.408157 15620 shared_informer.go:262] Caches are synced for ReplicaSet I0423 18:32:47.408304 15620 shared_informer.go:262] Caches are synced for taint I0423 18:32:47.408331 15620 shared_informer.go:262] Caches are synced for service account I0423 18:32:47.408370 15620 node_lifecycle_controller.go:1443] Initializing eviction metric for zone: I0423 18:32:47.408373 15620 taint_manager.go:204] "Starting NoExecuteTaintManager" I0423 18:32:47.408396 15620 taint_manager.go:209] "Sending events to api server" W0423 18:32:47.408419 15620 node_lifecycle_controller.go:1058] Missing timestamp for Node solid. Assuming now as a timestamp. I0423 18:32:47.408461 15620 node_lifecycle_controller.go:1259] Controller detected that zone is now in state Normal. I0423 18:32:47.408497 15620 event.go:294] "Event occurred" object="solid" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node solid event: Registered Node solid in Controller" I0423 18:32:47.409547 15620 shared_informer.go:262] Caches are synced for expand I0423 18:32:47.410112 15620 shared_informer.go:262] Caches are synced for TTL after finished I0423 18:32:47.416165 15620 shared_informer.go:262] Caches are synced for daemon sets I0423 18:32:47.417280 15620 shared_informer.go:262] Caches are synced for persistent volume I0423 18:32:47.419476 15620 shared_informer.go:262] Caches are synced for endpoint_slice_mirroring I0423 18:32:47.423820 15620 shared_informer.go:262] Caches are synced for ClusterRoleAggregator I0423 18:32:47.425114 15620 shared_informer.go:262] Caches are synced for stateful set I0423 18:32:47.427278 15620 shared_informer.go:262] Caches are synced for certificate-csrsigning-kubelet-client I0423 18:32:47.427303 15620 shared_informer.go:262] Caches are synced for certificate-csrsigning-kube-apiserver-client I0423 18:32:47.427313 15620 shared_informer.go:262] Caches are synced for certificate-csrsigning-kubelet-serving I0423 18:32:47.427359 15620 shared_informer.go:262] Caches are synced for certificate-csrsigning-legacy-unknown I0423 18:32:47.428522 15620 shared_informer.go:262] Caches are synced for TTL I0423 18:32:47.430800 15620 shared_informer.go:262] Caches are synced for endpoint I0423 18:32:47.431994 15620 shared_informer.go:262] Caches are synced for ReplicationController I0423 18:32:47.432934 15620 shared_informer.go:262] Caches are synced for GC I0423 18:32:47.435223 15620 shared_informer.go:262] Caches are synced for job I0423 18:32:47.441515 15620 shared_informer.go:262] Caches are synced for HPA I0423 18:32:47.456511 15620 shared_informer.go:262] Caches are synced for certificate-csrapproving I0423 18:32:47.458164 15620 shared_informer.go:262] Caches are synced for PVC protection I0423 18:32:47.458371 15620 shared_informer.go:262] Caches are synced for PV protection I0423 18:32:47.459818 15620 shared_informer.go:262] Caches are synced for ephemeral I0423 18:32:47.459906 15620 shared_informer.go:262] Caches are synced for endpoint_slice I0423 18:32:47.464633 15620 shared_informer.go:262] Caches are synced for node I0423 18:32:47.464658 15620 range_allocator.go:166] Starting range CIDR allocator I0423 18:32:47.464671 15620 shared_informer.go:255] Waiting for caches to sync for cidrallocator I0423 18:32:47.464705 15620 shared_informer.go:262] Caches are synced for cidrallocator I0423 18:32:47.474877 15620 shared_informer.go:262] Caches are synced for namespace I0423 18:32:47.476013 15620 shared_informer.go:262] Caches are synced for cronjob I0423 18:32:47.477133 15620 shared_informer.go:262] Caches are synced for attach detach I0423 18:32:47.479364 15620 shared_informer.go:262] Caches are synced for crt configmap I0423 18:32:47.608831 15620 shared_informer.go:262] Caches are synced for deployment I0423 18:32:47.608972 15620 shared_informer.go:262] Caches are synced for disruption I0423 18:32:47.654933 15620 shared_informer.go:262] Caches are synced for resource quota I0423 18:32:47.682181 15620 shared_informer.go:262] Caches are synced for resource quota I0423 18:32:47.992816 15620 shared_informer.go:262] Caches are synced for garbage collector I0423 18:32:48.014006 15620 shared_informer.go:262] Caches are synced for garbage collector I0423 18:32:48.014028 15620 garbagecollector.go:163] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 18:33:07.377301 15620 scope.go:115] "RemoveContainer" containerID="887459ec8feb302633b910f1a75a4073b4983a020e302a450a734a204a206d03" I0423 18:33:07.377523 15620 scope.go:115] "RemoveContainer" containerID="7b2c8eea70069b5553be4748322b1b7aee3014c4e2043124874edc563679c122" E0423 18:33:07.377858 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-5c8978b444-mhx2c_kube-system(8b64c32b-c536-4fe7-8bb2-8f44fde12f1a)\"" pod="kube-system/metrics-server-5c8978b444-mhx2c" podUID=8b64c32b-c536-4fe7-8bb2-8f44fde12f1a I0423 18:33:07.379026 15620 scope.go:115] "RemoveContainer" containerID="6fb8ceca14c76516aecd90edd83fc8875507eeddb2641cb2f376ce882a5dcbc3" E0423 18:33:07.379398 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-79f67d76f8-j4vcv_kube-system(7a8c385e-e43e-4086-9f9a-3237125c6e9b)\"" pod="kube-system/local-path-provisioner-79f67d76f8-j4vcv" podUID=7a8c385e-e43e-4086-9f9a-3237125c6e9b I0423 18:33:07.394551 15620 scope.go:115] "RemoveContainer" containerID="e6810303180ea10a9eb8f56b0645578cfe8ea7fa1ebd1f24133c4202514a209c" {"level":"warn","ts":"2024-04-23T18:33:07.946+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b7d6c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} E0423 18:33:08.414331 15620 proxier.go:1504] "Failed to execute iptables-restore" err=< exit status 1: Ignoring deprecated --wait-interval option. Warning: Extension REJECT revision 0 not supported, missing kernel module? iptables-restore: line 14 failed > I0423 18:33:08.414361 15620 proxier.go:855] "Sync failed" retryingTime="30s" I0423 18:33:14.404501 15620 scope.go:115] "RemoveContainer" containerID="7b2c8eea70069b5553be4748322b1b7aee3014c4e2043124874edc563679c122" E0423 18:33:14.404902 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-5c8978b444-mhx2c_kube-system(8b64c32b-c536-4fe7-8bb2-8f44fde12f1a)\"" pod="kube-system/metrics-server-5c8978b444-mhx2c" podUID=8b64c32b-c536-4fe7-8bb2-8f44fde12f1a E0423 18:33:17.689613 15620 resource_quota_controller.go:417] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request W0423 18:33:18.003607 15620 garbagecollector.go:752] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request] I0423 18:33:21.303198 15620 scope.go:115] "RemoveContainer" containerID="6fb8ceca14c76516aecd90edd83fc8875507eeddb2641cb2f376ce882a5dcbc3" I0423 18:33:29.304180 15620 scope.go:115] "RemoveContainer" containerID="7b2c8eea70069b5553be4748322b1b7aee3014c4e2043124874edc563679c122" W0423 18:33:34.156090 15620 handler_proxy.go:105] no RequestInfo found in the context E0423 18:33:34.156203 15620 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]] I0423 18:33:34.156227 15620 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. W0423 18:33:34.157237 15620 handler_proxy.go:105] no RequestInfo found in the context E0423 18:33:34.157301 15620 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService I0423 18:33:34.157321 15620 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. E0423 18:33:38.437643 15620 proxier.go:1504] "Failed to execute iptables-restore" err=< exit status 1: Ignoring deprecated --wait-interval option. Warning: Extension REJECT revision 0 not supported, missing kernel module? iptables-restore: line 14 failed > I0423 18:33:38.437671 15620 proxier.go:855] "Sync failed" retryingTime="30s" E0423 18:33:47.696218 15620 resource_quota_controller.go:417] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request W0423 18:33:48.012651 15620 garbagecollector.go:752] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request] {"level":"warn","ts":"2024-04-23T18:33:49.104+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b7d6c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} I0423 18:33:51.476831 15620 scope.go:115] "RemoveContainer" containerID="6fb8ceca14c76516aecd90edd83fc8875507eeddb2641cb2f376ce882a5dcbc3" I0423 18:33:51.476956 15620 scope.go:115] "RemoveContainer" containerID="5f323254aa0171f3519912eacfbb9c93f76643ddf8086090df1e7225c09a265a" E0423 18:33:51.477071 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-79f67d76f8-j4vcv_kube-system(7a8c385e-e43e-4086-9f9a-3237125c6e9b)\"" pod="kube-system/local-path-provisioner-79f67d76f8-j4vcv" podUID=7a8c385e-e43e-4086-9f9a-3237125c6e9b I0423 18:34:00.496938 15620 scope.go:115] "RemoveContainer" containerID="7b2c8eea70069b5553be4748322b1b7aee3014c4e2043124874edc563679c122" I0423 18:34:00.497202 15620 scope.go:115] "RemoveContainer" containerID="4cec70423b4bdf53a98c67d0c1338f6922d7da3d1055e6dca9ef9dc8149371dd" E0423 18:34:00.497625 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 20s restarting failed container=metrics-server pod=metrics-server-5c8978b444-mhx2c_kube-system(8b64c32b-c536-4fe7-8bb2-8f44fde12f1a)\"" pod="kube-system/metrics-server-5c8978b444-mhx2c" podUID=8b64c32b-c536-4fe7-8bb2-8f44fde12f1a I0423 18:34:04.303538 15620 scope.go:115] "RemoveContainer" containerID="5f323254aa0171f3519912eacfbb9c93f76643ddf8086090df1e7225c09a265a" E0423 18:34:04.303860 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-79f67d76f8-j4vcv_kube-system(7a8c385e-e43e-4086-9f9a-3237125c6e9b)\"" pod="kube-system/local-path-provisioner-79f67d76f8-j4vcv" podUID=7a8c385e-e43e-4086-9f9a-3237125c6e9b I0423 18:34:04.404447 15620 scope.go:115] "RemoveContainer" containerID="4cec70423b4bdf53a98c67d0c1338f6922d7da3d1055e6dca9ef9dc8149371dd" E0423 18:34:04.404829 15620 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 20s restarting failed container=metrics-server pod=metrics-server-5c8978b444-mhx2c_kube-system(8b64c32b-c536-4fe7-8bb2-8f44fde12f1a)\"" pod="kube-system/metrics-server-5c8978b444-mhx2c" podUID=8b64c32b-c536-4fe7-8bb2-8f44fde12f1a E0423 18:34:08.463665 15620 proxier.go:1504] "Failed to execute iptables-restore" err=< exit status 1: Ignoring deprecated --wait-interval option. Warning: Extension REJECT revision 0 not supported, missing kernel module? iptables-restore: line 14 failed > I0423 18:34:08.463692 15620 proxier.go:855] "Sync failed" retryingTime="30s" ```