search

rancher / rke

Rancher Kubernetes Engine (RKE), an extremely simple, lightning fast Kubernetes distribution that runs entirely within containers.
Apache License 2.0
3.21k stars 582 forks source link

rke failed to support ipv6 dual-stack because deploy rke network failed. #2786

Closed gongxiao2 closed 2 years ago

gongxiao2 commented 2 years ago

RKE version: v1.3.2

Docker version: (docker version,docker info preferred)

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:58:10 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:56:55 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Operating system and kernel: (cat /etc/os-release, uname -r preferred) uname -a

Linux lenovo 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/centos-release

CentOS Linux release 7.8.2003 (Core)

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) VirtualBox

cluster.yml file:

addon_job_timeout: 300
authentication:
  sans:
  - 192.168.200.59
  strategy: x509
cluster_name: cluster_example
dns:
  provider: coredns
network:
  plugin: calico
node_passwd: root
nodes:
- address: master1
  role:
  - controlplane
  - etcd
  ssh_key_path: /root/.ssh/id_rsa
  user: rke
- address: master2
  role:
  - controlplane
  - worker
  ssh_key_path: /root/.ssh/id_rsa
  user: rke
- address: worker1
  role:
  - worker
  ssh_key_path: /root/.ssh/id_rsa
  user: rke
private_registries:
- is_default: true
  password: '123456'
  url: master1:5000
  user: docker
registry_password: '123456'
services:
  kube-api:
    extra_args:
      feature-gates: IPv6DualStack=true
    service_cluster_ip_range: 10.43.0.0/16,2001:43::/108
  kube-controller:
    cluster_cidr: 10.42.0.0/16,2001:42::/108
    extra_args:
      feature-gates: IPv6DualStack=true
    node_cidr_mask_size_ipv4: 24
    node_cidr_mask_size_ipv6: 108
    service_cluster_ip_range: 10.43.0.0/16,2001:43::/108
  kubelet:
    extra_args:
      feature-gates: IPv6DualStack=true
  kubeproxy:
    cluster_cidr: 10.42.0.0/16,2001:42::/108
    extra_args:
      proxy-mode: ipvs
      feature-gates: IPv6DualStack=true

Steps to Reproduce: rke node exec follow command: rke up --config tc-cluster.yml

Results: rke deploy k8s failed:

INFO[0076] [addons] Executing deploy job rke-network-plugin
FATA[0378] Failed to get job complete status for job rke-network-plugin-deploy-job in namespace kube-system

k8s deploy result:

# docker ps
CONTAINER ID   IMAGE                                                       COMMAND                  CREATED          STATUS          PORTS                    NAMES
f1da40005b1b   master1:5000/rancher/hyperkube:v1.21.6-rancher1              "/opt/rke-tools/entr…"   43 minutes ago   Up 43 minutes                            kube-proxy
d49f9caf6741   master1:5000/rancher/hyperkube:v1.21.6-rancher1              "/opt/rke-tools/entr…"   43 minutes ago   Up 43 minutes                            kubelet
602e9ce11aa1   master1:5000/rancher/hyperkube:v1.21.6-rancher1              "/opt/rke-tools/entr…"   43 minutes ago   Up 43 minutes                            kube-scheduler
4f703c1c5c56   master1:5000/rancher/hyperkube:v1.21.6-rancher1              "/opt/rke-tools/entr…"   43 minutes ago   Up 29 seconds                            kube-controller-manager
5358e2cc5bfa   master1:5000/rancher/hyperkube:v1.21.6-rancher1              "/opt/rke-tools/entr…"   44 minutes ago   Up 44 minutes                            kube-apiserver
575c98561bda   master1:5000/rancher/rke-tools:v0.1.78                       "/docker-entrypoint.…"   44 minutes ago   Up 44 minutes                            etcd-rolling-snapshots
3b3cb1baded6   master1:5000/rancher/mirrored-coreos-etcd:v3.4.16-rancher1   "/usr/local/bin/etcd…"   44 minutes ago   Up 44 minutes                            etcd
a98dd9a74425   registry:2                                                  "/entrypoint.sh /etc…"   21 hours ago     Up 4 hours      0.0.0.0:5000->5000/tcp   docker-registry

kubelet error log:

+ echo kubelet --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=master1:5000/rancher/mirrored-pause:3.4.1 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --root-dir=/var/lib/kubelet --cgroups-per-qos=True --cni-bin-dir=/opt/cni/bin --read-only-port=0 --v=2 --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --network-plugin=cni --resolv-conf=/etc/resolv.conf --cloud-provider= --address=0.0.0.0 --feature-gates=IPv6DualStack=true --cluster-dns=10.43.0.10 --hostname-override=master1 --register-with-taints=node-role.kubernetes.io/controlplane=true:NoSchedule --make-iptables-util-chains=true --fail-swap-on=false --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --authentication-token-webhook=true --cluster-domain=cluster.local --cni-conf-dir=/etc/cni/net.d --authorization-mode=Webhook --anonymous-auth=false --streaming-connection-idle-timeout=30m
+ grep -q -- --resolv-conf=/etc/resolv.conf
+ pgrep -f systemd-resolved
+ '[' '!' -z eyJhdXRocyI6eyJneDUtbTE6NTAwMCI6eyJhdXRoIjoiWkc5amEyVnlPakV5TXpRMU5nPT0ifX19 ']'
+ base64 -d
+ echo eyJhdXRocyI6eyJneDUtbTE6NTAwMCI6eyJhdXRoIjoiWkc5amEyVnlPakV5TXpRMU5nPT0ifX19
+ tee /var/lib/kubelet/config.json
++ /opt/rke-tools/bin/docker info
++ grep -i 'cgroup driver'
++ awk '{print $3}'
+ CGROUPDRIVER=cgroupfs
+ '[' '' == true ']'
+ exec kubelet --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=master1:5000/rancher/mirrored-pause:3.4.1 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --root-dir=/var/lib/kubelet --cgroups-per-qos=True --cni-bin-dir=/opt/cni/bin --read-only-port=0 --v=2 --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --network-plugin=cni --resolv-conf=/etc/resolv.conf --cloud-provider= --address=0.0.0.0 --feature-gates=IPv6DualStack=true --cluster-dns=10.43.0.10 --hostname-override=master1 --register-with-taints=node-role.kubernetes.io/controlplane=true:NoSchedule --make-iptables-util-chains=true --fail-swap-on=false --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --authentication-token-webhook=true --cluster-domain=cluster.local --cni-conf-dir=/etc/cni/net.d --authorization-mode=Webhook --anonymous-auth=false --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroups-per-qos has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --event-qps has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --network-plugin has been deprecated, will be removed along with dockershim.
Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --make-iptables-util-chains has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --streaming-connection-idle-timeout has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I1221 06:42:21.565115    1291 flags.go:59] FLAG: --add-dir-header="false"
I1221 06:42:21.565306    1291 flags.go:59] FLAG: --address="0.0.0.0"
I1221 06:42:21.565314    1291 flags.go:59] FLAG: --allowed-unsafe-sysctls="[]"
I1221 06:42:21.565331    1291 flags.go:59] FLAG: --alsologtostderr="false"
I1221 06:42:21.565336    1291 flags.go:59] FLAG: --anonymous-auth="false"
I1221 06:42:21.565342    1291 flags.go:59] FLAG: --application-metrics-count-limit="100"
I1221 06:42:21.565347    1291 flags.go:59] FLAG: --authentication-token-webhook="true"
I1221 06:42:21.565352    1291 flags.go:59] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
I1221 06:42:21.565358    1291 flags.go:59] FLAG: --authorization-mode="Webhook"
I1221 06:42:21.565365    1291 flags.go:59] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
I1221 06:42:21.565369    1291 flags.go:59] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
I1221 06:42:21.565374    1291 flags.go:59] FLAG: --azure-container-registry-config=""
I1221 06:42:21.565378    1291 flags.go:59] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I1221 06:42:21.565386    1291 flags.go:59] FLAG: --bootstrap-kubeconfig=""
I1221 06:42:21.565390    1291 flags.go:59] FLAG: --cert-dir="/var/lib/kubelet/pki"
I1221 06:42:21.565395    1291 flags.go:59] FLAG: --cgroup-driver="cgroupfs"
I1221 06:42:21.565399    1291 flags.go:59] FLAG: --cgroup-root=""
I1221 06:42:21.565403    1291 flags.go:59] FLAG: --cgroups-per-qos="true"
I1221 06:42:21.565407    1291 flags.go:59] FLAG: --chaos-chance="0"
I1221 06:42:21.565414    1291 flags.go:59] FLAG: --client-ca-file="/etc/kubernetes/ssl/kube-ca.pem"
I1221 06:42:21.565420    1291 flags.go:59] FLAG: --cloud-config=""
I1221 06:42:21.565424    1291 flags.go:59] FLAG: --cloud-provider=""
I1221 06:42:21.565428    1291 flags.go:59] FLAG: --cluster-dns="[10.43.0.10]"
I1221 06:42:21.565434    1291 flags.go:59] FLAG: --cluster-domain="cluster.local"
I1221 06:42:21.565438    1291 flags.go:59] FLAG: --cni-bin-dir="/opt/cni/bin"
I1221 06:42:21.565444    1291 flags.go:59] FLAG: --cni-cache-dir="/var/lib/cni/cache"
I1221 06:42:21.565448    1291 flags.go:59] FLAG: --cni-conf-dir="/etc/cni/net.d"
I1221 06:42:21.565452    1291 flags.go:59] FLAG: --config=""
I1221 06:42:21.565462    1291 flags.go:59] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I1221 06:42:21.565468    1291 flags.go:59] FLAG: --container-log-max-files="5"
I1221 06:42:21.565473    1291 flags.go:59] FLAG: --container-log-max-size="10Mi"
I1221 06:42:21.565479    1291 flags.go:59] FLAG: --container-runtime="docker"
I1221 06:42:21.565484    1291 flags.go:59] FLAG: --container-runtime-endpoint="unix:///var/run/dockershim.sock"
I1221 06:42:21.565490    1291 flags.go:59] FLAG: --containerd="/run/containerd/containerd.sock"
I1221 06:42:21.565495    1291 flags.go:59] FLAG: --containerd-namespace="k8s.io"
I1221 06:42:21.565499    1291 flags.go:59] FLAG: --contention-profiling="false"
I1221 06:42:21.565503    1291 flags.go:59] FLAG: --cpu-cfs-quota="true"
I1221 06:42:21.565508    1291 flags.go:59] FLAG: --cpu-cfs-quota-period="100ms"
I1221 06:42:21.565513    1291 flags.go:59] FLAG: --cpu-manager-policy="none"
I1221 06:42:21.565517    1291 flags.go:59] FLAG: --cpu-manager-reconcile-period="10s"
I1221 06:42:21.565521    1291 flags.go:59] FLAG: --docker="unix:///var/run/docker.sock"
I1221 06:42:21.565527    1291 flags.go:59] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
I1221 06:42:21.565532    1291 flags.go:59] FLAG: --docker-env-metadata-whitelist=""
I1221 06:42:21.565536    1291 flags.go:59] FLAG: --docker-only="false"
I1221 06:42:21.565541    1291 flags.go:59] FLAG: --docker-root="/var/lib/docker"
I1221 06:42:21.565545    1291 flags.go:59] FLAG: --docker-tls="false"
I1221 06:42:21.565550    1291 flags.go:59] FLAG: --docker-tls-ca="ca.pem"
I1221 06:42:21.565554    1291 flags.go:59] FLAG: --docker-tls-cert="cert.pem"
I1221 06:42:21.565558    1291 flags.go:59] FLAG: --docker-tls-key="key.pem"
I1221 06:42:21.565564    1291 flags.go:59] FLAG: --dynamic-config-dir=""
I1221 06:42:21.565589    1291 flags.go:59] FLAG: --enable-controller-attach-detach="true"
I1221 06:42:21.565598    1291 flags.go:59] FLAG: --enable-debugging-handlers="true"
I1221 06:42:21.565605    1291 flags.go:59] FLAG: --enable-load-reader="false"
I1221 06:42:21.565612    1291 flags.go:59] FLAG: --enable-server="true"
I1221 06:42:21.565617    1291 flags.go:59] FLAG: --enforce-node-allocatable="[pods]"
I1221 06:42:21.565637    1291 flags.go:59] FLAG: --event-burst="10"
I1221 06:42:21.565642    1291 flags.go:59] FLAG: --event-qps="0"
I1221 06:42:21.565647    1291 flags.go:59] FLAG: --event-storage-age-limit="default=0"
I1221 06:42:21.565651    1291 flags.go:59] FLAG: --event-storage-event-limit="default=0"
I1221 06:42:21.565655    1291 flags.go:59] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
I1221 06:42:21.565674    1291 flags.go:59] FLAG: --eviction-max-pod-grace-period="0"
I1221 06:42:21.565679    1291 flags.go:59] FLAG: --eviction-minimum-reclaim=""
I1221 06:42:21.565692    1291 flags.go:59] FLAG: --eviction-pressure-transition-period="5m0s"
I1221 06:42:21.565697    1291 flags.go:59] FLAG: --eviction-soft=""
I1221 06:42:21.565701    1291 flags.go:59] FLAG: --eviction-soft-grace-period=""
I1221 06:42:21.565705    1291 flags.go:59] FLAG: --exit-on-lock-contention="false"
I1221 06:42:21.565709    1291 flags.go:59] FLAG: --experimental-allocatable-ignore-eviction="false"
I1221 06:42:21.565713    1291 flags.go:59] FLAG: --experimental-bootstrap-kubeconfig=""
I1221 06:42:21.565717    1291 flags.go:59] FLAG: --experimental-check-node-capabilities-before-mount="false"
I1221 06:42:21.565721    1291 flags.go:59] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
I1221 06:42:21.565726    1291 flags.go:59] FLAG: --experimental-kernel-memcg-notification="false"
I1221 06:42:21.565729    1291 flags.go:59] FLAG: --experimental-logging-sanitization="false"
I1221 06:42:21.565733    1291 flags.go:59] FLAG: --experimental-mounter-path=""
I1221 06:42:21.565737    1291 flags.go:59] FLAG: --fail-swap-on="false"
I1221 06:42:21.565741    1291 flags.go:59] FLAG: --feature-gates="IPv6DualStack=true"
I1221 06:42:21.565749    1291 flags.go:59] FLAG: --file-check-frequency="20s"
I1221 06:42:21.565753    1291 flags.go:59] FLAG: --global-housekeeping-interval="1m0s"
I1221 06:42:21.565757    1291 flags.go:59] FLAG: --hairpin-mode="promiscuous-bridge"
I1221 06:42:21.565761    1291 flags.go:59] FLAG: --healthz-bind-address="127.0.0.1"
I1221 06:42:21.565766    1291 flags.go:59] FLAG: --healthz-port="10248"
I1221 06:42:21.565771    1291 flags.go:59] FLAG: --help="false"
I1221 06:42:21.565775    1291 flags.go:59] FLAG: --hostname-override="master1"
I1221 06:42:21.565779    1291 flags.go:59] FLAG: --housekeeping-interval="10s"
I1221 06:42:21.565783    1291 flags.go:59] FLAG: --http-check-frequency="20s"
I1221 06:42:21.565787    1291 flags.go:59] FLAG: --image-credential-provider-bin-dir=""
I1221 06:42:21.565791    1291 flags.go:59] FLAG: --image-credential-provider-config=""
I1221 06:42:21.565795    1291 flags.go:59] FLAG: --image-gc-high-threshold="85"
I1221 06:42:21.565799    1291 flags.go:59] FLAG: --image-gc-low-threshold="80"
I1221 06:42:21.565803    1291 flags.go:59] FLAG: --image-pull-progress-deadline="1m0s"
I1221 06:42:21.565807    1291 flags.go:59] FLAG: --image-service-endpoint=""
I1221 06:42:21.565811    1291 flags.go:59] FLAG: --iptables-drop-bit="15"
I1221 06:42:21.565814    1291 flags.go:59] FLAG: --iptables-masquerade-bit="14"
I1221 06:42:21.565818    1291 flags.go:59] FLAG: --keep-terminated-pod-volumes="false"
I1221 06:42:21.565822    1291 flags.go:59] FLAG: --kernel-memcg-notification="false"
I1221 06:42:21.565826    1291 flags.go:59] FLAG: --kube-api-burst="10"
I1221 06:42:21.565830    1291 flags.go:59] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I1221 06:42:21.565834    1291 flags.go:59] FLAG: --kube-api-qps="5"
I1221 06:42:21.565838    1291 flags.go:59] FLAG: --kube-reserved=""
I1221 06:42:21.565842    1291 flags.go:59] FLAG: --kube-reserved-cgroup=""
I1221 06:42:21.565846    1291 flags.go:59] FLAG: --kubeconfig="/etc/kubernetes/ssl/kubecfg-kube-node.yaml"
I1221 06:42:21.565850    1291 flags.go:59] FLAG: --kubelet-cgroups=""
I1221 06:42:21.565860    1291 flags.go:59] FLAG: --lock-file=""
I1221 06:42:21.565865    1291 flags.go:59] FLAG: --log-backtrace-at=":0"
I1221 06:42:21.565870    1291 flags.go:59] FLAG: --log-cadvisor-usage="false"
I1221 06:42:21.565874    1291 flags.go:59] FLAG: --log-dir=""
I1221 06:42:21.565878    1291 flags.go:59] FLAG: --log-file=""
I1221 06:42:21.565882    1291 flags.go:59] FLAG: --log-file-max-size="1800"
I1221 06:42:21.565886    1291 flags.go:59] FLAG: --log-flush-frequency="5s"
I1221 06:42:21.565890    1291 flags.go:59] FLAG: --logging-format="text"
I1221 06:42:21.565902    1291 flags.go:59] FLAG: --logtostderr="true"
I1221 06:42:21.565925    1291 flags.go:59] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I1221 06:42:21.565930    1291 flags.go:59] FLAG: --make-iptables-util-chains="true"
I1221 06:42:21.565933    1291 flags.go:59] FLAG: --manifest-url=""
I1221 06:42:21.565937    1291 flags.go:59] FLAG: --manifest-url-header=""
I1221 06:42:21.565943    1291 flags.go:59] FLAG: --master-service-namespace="default"
I1221 06:42:21.565947    1291 flags.go:59] FLAG: --max-open-files="1000000"
I1221 06:42:21.565953    1291 flags.go:59] FLAG: --max-pods="110"
I1221 06:42:21.565957    1291 flags.go:59] FLAG: --maximum-dead-containers="-1"
I1221 06:42:21.565961    1291 flags.go:59] FLAG: --maximum-dead-containers-per-container="1"
I1221 06:42:21.565965    1291 flags.go:59] FLAG: --memory-manager-policy="None"
I1221 06:42:21.565969    1291 flags.go:59] FLAG: --minimum-container-ttl-duration="0s"
I1221 06:42:21.565973    1291 flags.go:59] FLAG: --minimum-image-ttl-duration="2m0s"
I1221 06:42:21.565977    1291 flags.go:59] FLAG: --network-plugin="cni"
I1221 06:42:21.565981    1291 flags.go:59] FLAG: --network-plugin-mtu="0"
I1221 06:42:21.565986    1291 flags.go:59] FLAG: --node-ip=""
I1221 06:42:21.565990    1291 flags.go:59] FLAG: --node-labels=""
I1221 06:42:21.565995    1291 flags.go:59] FLAG: --node-status-max-images="50"
I1221 06:42:21.565999    1291 flags.go:59] FLAG: --node-status-update-frequency="10s"
I1221 06:42:21.566004    1291 flags.go:59] FLAG: --non-masquerade-cidr="10.0.0.0/8"
I1221 06:42:21.566007    1291 flags.go:59] FLAG: --one-output="false"
I1221 06:42:21.566011    1291 flags.go:59] FLAG: --oom-score-adj="-999"
I1221 06:42:21.566016    1291 flags.go:59] FLAG: --pod-cidr=""
I1221 06:42:21.566020    1291 flags.go:59] FLAG: --pod-infra-container-image="master1:5000/rancher/mirrored-pause:3.4.1"
I1221 06:42:21.566025    1291 flags.go:59] FLAG: --pod-manifest-path=""
I1221 06:42:21.566028    1291 flags.go:59] FLAG: --pod-max-pids="-1"
I1221 06:42:21.566033    1291 flags.go:59] FLAG: --pods-per-core="0"
I1221 06:42:21.566037    1291 flags.go:59] FLAG: --port="10250"
I1221 06:42:21.566040    1291 flags.go:59] FLAG: --protect-kernel-defaults="false"
I1221 06:42:21.566044    1291 flags.go:59] FLAG: --provider-id=""
I1221 06:42:21.566048    1291 flags.go:59] FLAG: --qos-reserved=""
I1221 06:42:21.566058    1291 flags.go:59] FLAG: --read-only-port="0"
I1221 06:42:21.566062    1291 flags.go:59] FLAG: --really-crash-for-testing="false"
I1221 06:42:21.566066    1291 flags.go:59] FLAG: --redirect-container-streaming="false"
I1221 06:42:21.566070    1291 flags.go:59] FLAG: --register-node="true"
I1221 06:42:21.566074    1291 flags.go:59] FLAG: --register-schedulable="true"
I1221 06:42:21.566079    1291 flags.go:59] FLAG: --register-with-taints="node-role.kubernetes.io/controlplane=true:NoSchedule"
I1221 06:42:21.566106    1291 flags.go:59] FLAG: --registry-burst="10"
I1221 06:42:21.566110    1291 flags.go:59] FLAG: --registry-qps="5"
I1221 06:42:21.566114    1291 flags.go:59] FLAG: --reserved-cpus=""
I1221 06:42:21.566119    1291 flags.go:59] FLAG: --reserved-memory=""
I1221 06:42:21.566128    1291 flags.go:59] FLAG: --resolv-conf="/etc/resolv.conf"
I1221 06:42:21.566136    1291 flags.go:59] FLAG: --root-dir="/var/lib/kubelet"
I1221 06:42:21.566143    1291 flags.go:59] FLAG: --rotate-certificates="false"
I1221 06:42:21.566149    1291 flags.go:59] FLAG: --rotate-server-certificates="false"
I1221 06:42:21.566155    1291 flags.go:59] FLAG: --runonce="false"
I1221 06:42:21.566162    1291 flags.go:59] FLAG: --runtime-cgroups=""
I1221 06:42:21.566168    1291 flags.go:59] FLAG: --runtime-request-timeout="2m0s"
I1221 06:42:21.566174    1291 flags.go:59] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
I1221 06:42:21.566178    1291 flags.go:59] FLAG: --serialize-image-pulls="true"
I1221 06:42:21.566182    1291 flags.go:59] FLAG: --skip-headers="false"
I1221 06:42:21.566187    1291 flags.go:59] FLAG: --skip-log-headers="false"
I1221 06:42:21.566190    1291 flags.go:59] FLAG: --stderrthreshold="2"
I1221 06:42:21.566194    1291 flags.go:59] FLAG: --storage-driver-buffer-duration="1m0s"
I1221 06:42:21.566199    1291 flags.go:59] FLAG: --storage-driver-db="cadvisor"
I1221 06:42:21.566203    1291 flags.go:59] FLAG: --storage-driver-host="localhost:8086"
I1221 06:42:21.566207    1291 flags.go:59] FLAG: --storage-driver-password="root"
I1221 06:42:21.566211    1291 flags.go:59] FLAG: --storage-driver-secure="false"
I1221 06:42:21.566215    1291 flags.go:59] FLAG: --storage-driver-table="stats"
I1221 06:42:21.566219    1291 flags.go:59] FLAG: --storage-driver-user="root"
I1221 06:42:21.566222    1291 flags.go:59] FLAG: --streaming-connection-idle-timeout="30m0s"
I1221 06:42:21.566248    1291 flags.go:59] FLAG: --sync-frequency="1m0s"
I1221 06:42:21.566254    1291 flags.go:59] FLAG: --system-cgroups=""
I1221 06:42:21.566258    1291 flags.go:59] FLAG: --system-reserved=""
I1221 06:42:21.566263    1291 flags.go:59] FLAG: --system-reserved-cgroup=""
I1221 06:42:21.566266    1291 flags.go:59] FLAG: --tls-cert-file=""
I1221 06:42:21.566270    1291 flags.go:59] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305]"
I1221 06:42:21.566282    1291 flags.go:59] FLAG: --tls-min-version=""
I1221 06:42:21.566287    1291 flags.go:59] FLAG: --tls-private-key-file=""
I1221 06:42:21.566291    1291 flags.go:59] FLAG: --topology-manager-policy="none"
I1221 06:42:21.566303    1291 flags.go:59] FLAG: --topology-manager-scope="container"
I1221 06:42:21.566308    1291 flags.go:59] FLAG: --v="2"
I1221 06:42:21.566312    1291 flags.go:59] FLAG: --version="false"
I1221 06:42:21.566323    1291 flags.go:59] FLAG: --vmodule=""
I1221 06:42:21.566327    1291 flags.go:59] FLAG: --volume-plugin-dir="/var/lib/kubelet/volumeplugins"
I1221 06:42:21.566331    1291 flags.go:59] FLAG: --volume-stats-agg-period="1m0s"
I1221 06:42:21.566412    1291 feature_gate.go:243] feature gates: &{map[IPv6DualStack:true]}
I1221 06:42:21.566523    1291 feature_gate.go:243] feature gates: &{map[IPv6DualStack:true]}
I1221 06:42:21.567252    1291 mount_linux.go:197] Detected OS without systemd
I1221 06:42:21.567767    1291 server.go:440] "Kubelet version" kubeletVersion="v1.21.6"
I1221 06:42:21.567986    1291 feature_gate.go:243] feature gates: &{map[IPv6DualStack:true]}
I1221 06:42:21.568299    1291 feature_gate.go:243] feature gates: &{map[IPv6DualStack:true]}
I1221 06:42:21.636241    1291 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem"
I1221 06:42:21.636578    1291 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem
I1221 06:42:21.637219    1291 manager.go:165] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct"
I1221 06:42:21.765557    1291 fs.go:131] Filesystem UUIDs: map[1C38-F977:/dev/vda2 54cc07ba-db45-4bc0-ab9a-eec5c00c02cb:/dev/dm-2 9404f155-2f0a-4acd-ac84-8b0558e70579:/dev/dm-0 c9baa142-297a-4f7b-9ae1-894d21c2f5f2:/dev/dm-1 f8c7ce50-ef94-4647-8669-8ead1a8a4b3d:/dev/vda1]
I1221 06:42:21.765613    1291 fs.go:132] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:103 fsType:tmpfs blockSize:0} /dev/mapper/centos-docker:{mountpoint:/var/lib/docker major:253 minor:2 fsType:xfs blockSize:0} /dev/mapper/centos-root:{mountpoint:/etc/resolv.conf major:253 minor:0 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:18 fsType:tmpfs blockSize:0} /host/dev/shm:{mountpoint:/host/dev/shm major:0 minor:18 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:19 fsType:tmpfs blockSize:0} /run/user/0:{mountpoint:/run/user/0 major:0 minor:88 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:36 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:20 fsType:tmpfs blockSize:0} overlay_0-101:{mountpoint:/ major:0 minor:101 fsType:overlay blockSize:0}]
I1221 06:42:21.766765    1291 nvidia.go:61] NVIDIA setup failed: no NVIDIA devices found
I1221 06:42:21.772541    1291 manager.go:213] Machine: {Timestamp:2021-12-21 06:42:21.772192583 +0000 UTC m=+0.397564609 NumCores:4 NumPhysicalCores:1 NumSockets:4 CpuFrequency:2399988 MemoryCapacity:3973103616 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:80581A4B-FAB0-4EB9-AC86-D40B6A3F1153 BootID:6a074e0a-6f1f-43ca-b8ea-c92629a09a70 Filesystems:[{Device:/run/user/1000 DeviceMajor:0 DeviceMinor:36 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:/dev/mapper/centos-root DeviceMajor:253 DeviceMinor:0 Capacity:36485206016 Type:vfs Inodes:17823744 HasInodes:true} {Device:/host/dev/shm DeviceMajor:0 DeviceMinor:18 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:12874416128 Type:vfs Inodes:6291456 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:18 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:19 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:/run/user/0 DeviceMajor:0 DeviceMinor:88 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:/dev DeviceMajor:0 DeviceMinor:103 Capacity:1974284288 Type:vfs Inodes:482003 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:20 Capacity:1986551808 Type:vfs Inodes:484998 HasInodes:true} {Device:/dev/mapper/centos-docker DeviceMajor:253 DeviceMinor:2 Capacity:12874416128 Type:vfs Inodes:6291456 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:64424509440 Scheduler:mq-deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:36503027712 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:12884901888 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:12884901888 Scheduler:none}] NetworkDevices:[{Name:dummy0 MacAddress:22:63:1a:ce:c7:cb Speed:0 Mtu:1500} {Name:eth0 MacAddress:52:54:00:8e:b5:35 Speed:0 Mtu:1500} {Name:eth1 MacAddress:52:54:00:34:c7:91 Speed:0 Mtu:1500} {Name:kube-ipvs0 MacAddress:16:ba:52:f9:db:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:3973103616 HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}] SocketID:0} {Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}] SocketID:1} {Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}] SocketID:2} {Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}] SocketID:3}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1221 06:42:21.772841    1291 manager_no_libpfm.go:28] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
I1221 06:42:21.819810    1291 manager.go:229] Version: {KernelVersion:3.10.0-1127.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:20.10.0 DockerAPIVersion:1.41 CadvisorVersion: CadvisorRevision:}
I1221 06:42:21.820250    1291 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I1221 06:42:21.820847    1291 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I1221 06:42:21.821037    1291 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
I1221 06:42:21.821112    1291 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I1221 06:42:21.821129    1291 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
I1221 06:42:21.821139    1291 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
I1221 06:42:21.821196    1291 manager.go:136] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
I1221 06:42:21.821417    1291 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
I1221 06:42:21.821515    1291 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
I1221 06:42:21.821557    1291 client.go:97] "Start docker client with request timeout" timeout="2m0s"
I1221 06:42:21.845268    1291 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
I1221 06:42:21.845314    1291 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
I1221 06:42:21.845615    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
I1221 06:42:21.882259    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
I1221 06:42:21.882356    1291 plugins.go:168] "Loaded network plugin" networkPluginName="cni"
I1221 06:42:21.882437    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
I1221 06:42:21.882543    1291 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
I1221 06:42:21.908729    1291 docker_service.go:264] "Docker Info" dockerInfo=&{ID:IAIJ:IBRN:SWTX:EXCL:LMNQ:IW2F:QUWF:MN5K:37YL:R5ZI:OQTI:KXXR Containers:10 ContainersRunning:7 ContainersPaused:0 ContainersStopped:3 Images:5 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:61 OomKillDisable:true NGoroutines:64 SystemTime:2021-12-21T14:42:21.883687484+08:00 LoggingDriver:json-file CgroupDriver:cgroupfs CgroupVersion:1 NEventsListener:0 KernelVersion:3.10.0-1127.el7.x86_64 OperatingSystem:CentOS Linux 7 (Core) OSVersion:7 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000babab0 NCPU:4 MemTotal:3973103616 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:master1 Labels:[] ExperimentalBuild:false ServerVersion:20.10.0 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d71fcd7d8303cbf684402823e425e9dd2e99285d Expected:d71fcd7d8303cbf684402823e425e9dd2e99285d} RuncCommit:{ID:v1.0.1-0-g4144b63 Expected:v1.0.1-0-g4144b63} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[]}
I1221 06:42:21.908760    1291 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="cgroupfs"
I1221 06:42:21.908876    1291 kubelet_dockershim.go:62] "Starting the GRPC server for the docker CRI shim."
I1221 06:42:21.908952    1291 docker_server.go:62] "Start dockershim grpc server"
I1221 06:42:21.926869    1291 remote_runtime.go:62] parsed scheme: ""
I1221 06:42:21.926904    1291 remote_runtime.go:62] scheme "" not registered, fallback to default scheme
I1221 06:42:21.927013    1291 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
I1221 06:42:21.927059    1291 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1221 06:42:21.927180    1291 remote_image.go:50] parsed scheme: ""
I1221 06:42:21.927191    1291 remote_image.go:50] scheme "" not registered, fallback to default scheme
I1221 06:42:21.927204    1291 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
I1221 06:42:21.927213    1291 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1221 06:42:21.927292    1291 server.go:1131] "Using root directory" path="/var/lib/kubelet"
I1221 06:42:21.927448    1291 kubelet.go:404] "Attempting to sync node with API server"
I1221 06:42:21.927490    1291 kubelet.go:283] "Adding apiserver pod source"
I1221 06:42:21.927593    1291 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
I1221 06:42:21.928252    1291 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I1221 06:42:21.928354    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0000be2c0, {CONNECTING <nil>}
I1221 06:42:21.928689    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0000be580, {CONNECTING <nil>}
I1221 06:42:21.931122    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0000be580, {READY <nil>}
I1221 06:42:21.931207    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0000be2c0, {READY <nil>}
I1221 06:42:21.953191    1291 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="docker" version="20.10.0" apiVersion="1.41.0"
I1221 06:42:22.930471    1291 apiserver.go:52] "Watching apiserver"
I1221 06:42:26.882771    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:27.969861    1291 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I1221 06:42:27.971030    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/azure-file"
I1221 06:42:27.971069    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/vsphere-volume"
I1221 06:42:27.971085    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/aws-ebs"
I1221 06:42:27.971099    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/gce-pd"
I1221 06:42:27.971114    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/cinder"
I1221 06:42:27.971143    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/azure-disk"
I1221 06:42:27.971178    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/empty-dir"
I1221 06:42:27.971206    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/git-repo"
I1221 06:42:27.971261    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/host-path"
I1221 06:42:27.971291    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/nfs"
I1221 06:42:27.971318    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/secret"
I1221 06:42:27.971344    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/iscsi"
I1221 06:42:27.971369    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/glusterfs"
I1221 06:42:27.971399    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/rbd"
I1221 06:42:27.971425    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/quobyte"
I1221 06:42:27.971451    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/cephfs"
I1221 06:42:27.971476    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/downward-api"
I1221 06:42:27.971501    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/fc"
I1221 06:42:27.971527    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/flocker"
I1221 06:42:27.971560    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/configmap"
I1221 06:42:27.971620    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/projected"
I1221 06:42:27.971722    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/portworx-volume"
I1221 06:42:27.971774    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/scaleio"
I1221 06:42:27.971819    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/local-volume"
I1221 06:42:27.971849    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/storageos"
I1221 06:42:27.971976    1291 plugins.go:639] Loaded volume plugin "kubernetes.io/csi"
I1221 06:42:27.972482    1291 server.go:1190] "Started kubelet"
E1221 06:42:27.973384    1291 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
I1221 06:42:27.973640    1291 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
I1221 06:42:27.977569    1291 server.go:409] "Adding debug handlers to kubelet server"
I1221 06:42:27.979406    1291 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I1221 06:42:27.981093    1291 volume_manager.go:269] "The desired_state_of_world populator starts"
I1221 06:42:27.981117    1291 volume_manager.go:271] "Starting Kubelet Volume Manager"
I1221 06:42:27.981633    1291 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
E1221 06:42:28.027763    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
I1221 06:42:28.027933    1291 factory.go:55] Registering systemd factory
I1221 06:42:28.034357    1291 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master1" not found
E1221 06:42:28.045763    1291 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"master1\" not found" node="master1"
I1221 06:42:28.052723    1291 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master1" not found
I1221 06:42:28.056460    1291 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
I1221 06:42:28.081963    1291 kubelet_node_status.go:362] "Setting node annotation to enable volume controller attach/detach"
E1221 06:42:28.084037    1291 kubelet.go:2291] "Error getting node" err="node \"master1\" not found"
I1221 06:42:28.088585    1291 factory.go:372] Registering Docker factory
I1221 06:42:28.088747    1291 client.go:86] parsed scheme: "unix"
I1221 06:42:28.088759    1291 client.go:86] scheme "unix" not registered, fallback to default scheme
I1221 06:42:28.088784    1291 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
I1221 06:42:28.088794    1291 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1221 06:42:28.089006    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0014ec6e0, {CONNECTING <nil>}
I1221 06:42:28.091431    1291 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0014ec6e0, {READY <nil>}
I1221 06:42:28.094509    1291 factory.go:137] Registering containerd factory
I1221 06:42:28.095795    1291 factory.go:101] Registering Raw factory
I1221 06:42:28.096568    1291 manager.go:1203] Started watching for new ooms in manager
I1221 06:42:28.099858    1291 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
I1221 06:42:28.099988    1291 status_manager.go:157] "Starting to sync pod status with apiserver"
I1221 06:42:28.100110    1291 kubelet.go:1846] "Starting kubelet main sync loop"
E1221 06:42:28.100326    1291 kubelet.go:1870] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
I1221 06:42:28.103346    1291 manager.go:301] Starting recovery of all containers
I1221 06:42:28.137745    1291 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master1" not found
I1221 06:42:28.145694    1291 kubelet_node_status.go:554] "Recording event message for node" node="master1" event="NodeHasSufficientMemory"
I1221 06:42:28.145775    1291 kubelet_node_status.go:554] "Recording event message for node" node="master1" event="NodeHasNoDiskPressure"
I1221 06:42:28.145802    1291 kubelet_node_status.go:554] "Recording event message for node" node="master1" event="NodeHasSufficientPID"
I1221 06:42:28.145925    1291 kubelet_node_status.go:71] "Attempting to register node" node="master1"
I1221 06:42:28.158475    1291 kubelet_node_status.go:74] "Successfully registered node" node="master1"
I1221 06:42:28.186279    1291 manager.go:306] Recovery completed
E1221 06:42:28.200849    1291 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
I1221 06:42:28.262112    1291 cpu_manager.go:199] "Starting CPU manager" policy="none"
I1221 06:42:28.262147    1291 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
I1221 06:42:28.262182    1291 state_mem.go:36] "Initialized new in-memory state store"
I1221 06:42:28.262493    1291 state_mem.go:88] "Updated default CPUSet" cpuSet=""
I1221 06:42:28.262522    1291 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
I1221 06:42:28.262539    1291 state_checkpoint.go:136] "State checkpoint: restored state from checkpoint"
I1221 06:42:28.262550    1291 state_checkpoint.go:137] "State checkpoint: defaultCPUSet" defaultCpuSet=""
I1221 06:42:28.262569    1291 policy_none.go:44] "None policy: Start"
I1221 06:42:28.263693    1291 manager.go:242] "Starting Device Plugin manager"
I1221 06:42:28.263755    1291 manager.go:600] "Failed to retrieve checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
I1221 06:42:28.263988    1291 manager.go:284] "Serving device plugin registration server on socket" path="/var/lib/kubelet/device-plugins/kubelet.sock"
I1221 06:42:28.264174    1291 plugin_watcher.go:52] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
I1221 06:42:28.264219    1291 container_manager_linux.go:510] "Discovered runtime cgroup name" cgroupName="/system.slice/docker.service"
I1221 06:42:28.264273    1291 plugin_manager.go:112] "The desired_state_of_world populator (plugin watcher) starts"
I1221 06:42:28.264283    1291 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
I1221 06:42:28.401444    1291 kubelet.go:1932] "SyncLoop ADD" source="api" pods=[]
I1221 06:42:28.484544    1291 reconciler.go:157] "Reconciler: start to sync state"
I1221 06:42:31.883609    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:33.285833    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
I1221 06:42:36.884617    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:38.319259    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
I1221 06:42:41.885532    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:43.347998    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
I1221 06:42:46.886821    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:48.379948    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
I1221 06:42:51.887806    1291 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
E1221 06:42:53.410761    1291 kubelet.go:2211] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
superseb commented 2 years ago

Please post the output of the job as seen on https://rancher.com/docs/rke/latest/en/troubleshooting/provisioning-errors/

stale[bot] commented 2 years ago

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.