harvester startup failure

w13915984028 / harvester-develop-summary

Summary of Harvester develop.

2 stars 4 forks source link

harvester startup failure #2

Open w13915984028 opened 2 years ago

w13915984028 commented 2 years ago

the cluster runs 15+ days, when rebooted, startup failed

rancher@harv2:~> ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master harvester-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:05:c2:1d brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: harvester-mgmt: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:05:c2:1d brd ff:ff:ff:ff:ff:ff
rancher@harv2:~>
rancher@harv2:~>

w13915984028 commented 2 years ago

re-install with master-head ISO of today, its log:

rancher@harv2:~> sudo -i journalctl --since 01:20 | grep kubelet | more
Jan 07 13:19:40 harv2 systemd[1]: sysroot-var-lib-kubelet.mount: Succeeded.
Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --b
ind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --encryption-provider-config=/var/lib/rancher/rke2/server/cred/en
cryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=
0 --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --profiling=false --p
roxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/re
quest-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-accoun
t-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.53.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher
/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key"
Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.52.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/r
ancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/
lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var
/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/rke2/server/cred/contro
ller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --use-service-account-credentials=true"
Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kubelet] is unhealthy\""
Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var
/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.53.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --
eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=harv2 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log
-file-max-size=50 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --logtostderr=false --node-labels=harvesterhci.io/managed=true --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pul
ls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key"
Jan 07 13:20:11 harv2 rke2[2279]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 07 13:20:11 harv2 rke2[2279]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more informatio
n.

w13915984028 commented 2 years ago

Jan 07 13:20:00 harv2 systemd[1]: cos-setup-network.service: Succeeded. Jan 07 13:20:00 harv2 systemd[1]: Finished cOS setup after network. Jan 07 13:20:00 harv2 rancherd[1943]: time="2022-01-07T13:20:00Z" level=info msg="Bootstrapping Rancher (master-head/v1.21.7+rke2r1)" Jan 07 13:20:00 harv2 rancherd[1943]: time="2022-01-07T13:20:00Z" level=info msg="Writing plan file to /var/lib/rancher/rancherd/plan/plan.json" Jan 07 13:20:00 harv2 rancherd[1943]: time="2022-01-07T13:20:00Z" level=info msg="Applying plan with checksum " Jan 07 13:20:00 harv2 rancherd[1943]: time="2022-01-07T13:20:00Z" level=info msg="Extracting image rancher/system-agent-installer-rke2:v1.21.7-rke2r1 to directory /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_0" Jan 07 13:20:00 harv2 rancherd[1943]: time="2022-01-07T13:20:00Z" level=info msg="Checking local image archives in /var/lib/rancher/agent/images for index.docker.io/rancher/system-agent-installer-rke2:v1.21.7-rke2r1" Jan 07 13:20:01 harv2 sshd-gen-keys-start[1952]: ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 Jan 07 13:20:01 harv2 sshd[2011]: Server listening on 0.0.0.0 port 22. Jan 07 13:20:01 harv2 sshd[2011]: Server listening on :: port 22.

Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + INSTALL_RKE2_TYPE=server" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + '[' -n '2022-01-07 13:20:00.730364665 +0000 UTC m=+0.179448132' ']'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + echo '2022-01-07 13:20:00.730364665 +0000 UTC m=+0.179448132'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + systemctl daemon-reload" Jan 07 13:20:06 harv2 systemd[1]: Reloading. Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + '[' '' = true ']'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + '[' server = server ']'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + systemctl is-active --quiet rke2-agent" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + systemctl enable rke2-server" Jan 07 13:20:06 harv2 systemd[1]: Reloading. Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service." Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + '[' '' = true ']'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + '[' true = true ']'" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="[stderr]: + systemctl --no-block restart rke2-server" Jan 07 13:20:06 harv2 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)... Jan 07 13:20:06 harv2 sh[2168]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="Extracting image rancher/system-agent-installer-rke2:v1.21.7-rke2r1 to directory /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1" Jan 07 13:20:06 harv2 rancherd[1943]: time="2022-01-07T13:20:06Z" level=info msg="Checking local image archives in /var/lib/rancher/agent/images for index.docker.io/rancher/system-agent-installer-rke2:v1.21.7-rke2r1" Jan 07 13:20:06 harv2 sh[2170]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory Jan 07 13:20:06 harv2 kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. Jan 07 13:20:06 harv2 kernel: Bridge firewalling registered Jan 07 13:20:06 harv2 harv-update-rke2-server-url[2173]: + HARVESTER_CONFIG_FILE=/oem/harvester.config Jan 07 13:20:06 harv2 harv-update-rke2-server-url[2173]: + RKE2_VIP_CONFIG_FILE=/etc/rancher/rke2/config.yaml.d/90-harvester-vip.yaml Jan 07 13:20:06 harv2 harv-update-rke2-server-url[2173]: + case $1 in Jan 07 13:20:06 harv2 harv-update-rke2-server-url[2173]: + rm -f /etc/rancher/rke2/config.yaml.d/90-harvester-vip.yaml Jan 07 13:20:07 harv2 systemd[1]: Received SIGRTMIN+21 from PID 2190 (kill).

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=warning msg="not running in CIS mode"

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Starting rke2 v1.21.7+rke2r1 (bb669e09bcf5220c3e34f481c996928a3c41ebba)" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Managed etcd cluster initializing"

w13915984028 commented 2 years ago

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Active TLS secret (ver=) (count 10): map[listener.cattle.io/cn-10.53.0.1:10.53.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.122.170:192.168.122.170 listener.cat tle.io/cn-192.168.122.71:192.168.122.71 listener.cattle.io/cn-harv2:harv2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-ku bernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=712095417CC18D19D7F668373BFBE15BD51E604E]"

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --b ind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --encryption-provider-config=/var/lib/rancher/rke2/server/cred/en cryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port= 0 --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --profiling=false --p roxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/re quest-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-accoun t-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.53.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher /rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --port=10251 --profiling=false --secure-port=0"

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.52.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/r ancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/ lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var /lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/rke2/server/cred/contro ller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --use-service-account-credentials=true"

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=rke2 --cluster-cidr=10.52.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/ranc her/rke2/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m0s --port=0 --profiling=false" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Node token is available at /var/lib/rancher/rke2/server/token" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="To join node to cluster: rke2 agent -s https://192.168.122.71:9345 -t ${NODE_TOKEN}" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Run: rke2 kubectl" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="certificate CN=harv2 signed by CN=rke2-server-ca@1641561607: notBefore=2022-01-07 13:20:07 +0000 UTC notAfter=2023-01-07 13:20:07 +0000 UTC" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="certificate CN=system:node:harv2,O=system:nodes signed by CN=rke2-client-ca@1641561607: notBefore=2022-01-07 13:20:07 +0000 UTC notAfter=2023-01-07 13:20:07 +0000 UTC" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/installer.sh" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/rke2.linux-amd64.tar.gz" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/sha256sum-amd64.txt" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file run.sh to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/run.sh" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Running command: /usr/bin/rancherd [probe]" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:08Z\" level=info msg=\"Running probes defined in /var/lib/rancher/rancherd/plan/plan.json\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-controller-manager] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-apiserver] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-scheduler] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kubelet] is unhealthy\"" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Module overlay was already loaded" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Module br_netfilter was already loaded" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 327680" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Runtime image index.docker.io/rancher/rke2-runtime:v1.21.7-rke2r1 bin and charts directories already exist; skipping extract" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"

w13915984028 commented 2 years ago

Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="Run: rke2 kubectl" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="certificate CN=harv2 signed by CN=rke2-server-ca@1641561607: notBefore=2022-01-07 13:20:07 +0000 UTC notAfter=2023-01-07 13:20:07 +0000 UTC" Jan 07 13:20:07 harv2 rke2[2175]: time="2022-01-07T13:20:07Z" level=info msg="certificate CN=system:node:harv2,O=system:nodes signed by CN=rke2-client-ca@1641561607: notBefore=2022-01-07 13:20:07 +0000 UTC notAfter=2023-01-07 13:20:07 +0000 UTC"

Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/installer.sh" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/rke2.linux-amd64.tar.gz" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/sha256sum-amd64.txt" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Extracting file run.sh to /var/lib/rancher/rancherd/plan/work/20220107-132000-applied.plan/_1/run.sh" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="Running command: /usr/bin/rancherd [probe]" Jan 07 13:20:08 harv2 rancherd[1943]: time="2022-01-07T13:20:08Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:08Z\" level=info msg=\"Running probes defined in /var/lib/rancher/rancherd/plan/plan.json\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-controller-manager] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-apiserver] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kube-scheduler] is unhealthy\"" Jan 07 13:20:09 harv2 rancherd[1943]: time="2022-01-07T13:20:09Z" level=info msg="[stderr]: time=\"2022-01-07T13:20:09Z\" level=info msg=\"Probe [kubelet] is unhealthy\"" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Module overlay was already loaded" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Module br_netfilter was already loaded"

w13915984028 commented 2 years ago

Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Containerd is now running" Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:9345/v1-rke2/connect" Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Handling backend connection request [harv2]" Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var /lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.53.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock -- eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=harv2 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log -file-max-size=50 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --logtostderr=false --node-labels=harvesterhci.io/managed=true --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pul ls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key" Jan 07 13:20:11 harv2 rke2[2175]: time="2022-01-07T13:20:11Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error" Jan 07 13:20:11 harv2 rke2[2279]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jan 07 13:20:11 harv2 rke2[2279]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more informatio n.

w13915984028 commented 2 years ago

Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Module br_netfilter was already loaded" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 327680" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Runtime image index.docker.io/rancher/rke2-runtime:v1.21.7-rke2r1 bin and charts directories already exist; skipping extract" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-kube-proxy.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"

Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"

w13915984028 commented 2 years ago

Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' -z '' ']'" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + INSTALL_RKE2_TYPE=server" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib /systemd/system/rke2-server.service" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' -n 1f3755775f70c6146fafeac4d8948ac8c799c765acc087cbbb25ff46924e5617 ']'" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + echo 1f3755775f70c6146fafeac4d8948ac8c799c765acc087cbbb25ff46924e5617" Jan 07 13:23:34 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:34Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + systemctl daemon-reload" Jan 07 13:23:34 harv2 systemd[1]: Reloading. Jan 07 13:23:35 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:35Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' '' = true ']'" Jan 07 13:23:35 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:35Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' server = server ']'" Jan 07 13:23:35 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:35Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + systemctl is-active --quiet rke2-agent" Jan 07 13:23:35 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:35Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + systemctl enable rke2-server" Jan 07 13:23:35 harv2 systemd[1]: Reloading. Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' '' = true ']'" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + '[' true = true ']'" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=info msg="[94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218_0:stderr]: + systemctl --no-block restart rke2-server"

Jan 07 13:23:36 harv2 systemd[1]: Stopping Rancher Kubernetes Engine v2 (server)...

Jan 07 13:23:36 harv2 rke2[2175]: time="2022-01-07T13:23:36Z" level=info msg="Shutting down /v1, Kind=Node workers"

Jan 07 13:23:36 harv2 rke2[2175]: time="2022-01-07T13:23:36Z" level=fatal msg="context canceled"

Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=debug msg="[K8s] writing an applied checksum value of 94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218 to the remote plan" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=debug msg="[Prober] (kube-apiserver) running probe" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=debug msg="[Prober] (kube-apiserver) retrieving existing probe status from map if existing" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=debug msg="[Prober] (kube-apiserver) probe status was not present in map, initializing" Jan 07 13:23:36 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:36Z" level=debug msg="sleeping for 1 seconds before running probe"

Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Failed with result 'exit-code'. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2373 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2473 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2632 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2643 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2805 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2908 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 3617 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 3699 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4921 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4946 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 5355 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 6002 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 8133 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 8154 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 9700 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 9917 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 10480 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 10853 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 11366 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 13292 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 13320 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 13354 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 13377 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 13536 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14070 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14180 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14218 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14261 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14262 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14331 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14384 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14473 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14488 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14499 (runc) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14503 (runc) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 15184 (runc) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 15230 (exe) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 15263 (containerd-shim) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 15272 (runc) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2262 (containerd) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2394 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2436 (etcd) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2494 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2527 (kube-apiserver) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2677 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2687 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2747 (kube-scheduler) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2746 (kube-controller) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2824 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2876 (cloud-controlle) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2928 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 2957 (kube-proxy) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 3637 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 3733 (pause) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4442 (entrypoint.sh) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4618 (sleep) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4574 (runsvdir) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 4648 (runsv) remains running after unit stopped.

w13915984028 commented 2 years ago

Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14534 (runc:[2:INIT]) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 14584 (runc:[2:INIT]) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Unit process 15279 (exe) remains running after unit stopped. Jan 07 13:23:36 harv2 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).

Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Found left-over process 2373 (containerd-shim) in control group while starting unit. Ignoring. Jan 07 13:23:36 harv2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Found left-over process 2473 (containerd-shim) in control group while starting unit. Ignoring. Jan 07 13:23:36 harv2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jan 07 13:23:36 harv2 systemd[1]: rke2-server.service: Found left-over process 2632 (containerd-shim) in control group while starting unit. Ignoring. ... Jan 07 13:23:36 harv2 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)... Jan 07 13:23:36 harv2 harv-update-rke2-server-url[15340]: + HARVESTER_CONFIG_FILE=/oem/harvester.config Jan 07 13:23:36 harv2 harv-update-rke2-server-url[15340]: + RKE2_VIP_CONFIG_FILE=/etc/rancher/rke2/config.yaml.d/90-harvester-vip.yaml Jan 07 13:23:36 harv2 harv-update-rke2-server-url[15340]: + case $1 in Jan 07 13:23:36 harv2 harv-update-rke2-server-url[15340]: + rm -f /etc/rancher/rke2/config.yaml.d/90-harvester-vip.yaml ...

Jan 07 13:23:36 harv2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jan 07 13:23:36 harv2 rke2[15342]: time="2022-01-07T13:23:36Z" level=warning msg="not running in CIS mode" Jan 07 13:23:36 harv2 rke2[15342]: time="2022-01-07T13:23:36Z" level=info msg="Starting rke2 v1.21.7+rke2r1 (bb669e09bcf5220c3e34f481c996928a3c41ebba)" Jan 07 13:23:36 harv2 rke2[15342]: time="2022-01-07T13:23:36Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=debug msg="Probe timeout duration: 5 seconds"

w13915984028 commented 2 years ago

Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=error msg="error loading CA cert /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such fil e or directory" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=error msg="error while appending ca cert to pool" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=error msg="error loading CA cert /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube -controller-manager.crt: no such file or directory" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=error msg="error while appending ca cert to pool" Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=debug msg="Probe timeout duration: 5 seconds"

w13915984028 commented 2 years ago

Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.226Z","caller":"embed/etcd.go:117","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.228Z","caller":"embed/etcd.go:127","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.229Z","caller":"embed/etcd.go:302","msg":"starting an etcd server","etcd-version":"3.4.13","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.16.10b7","go-os":"linux", "go-arch":"amd64","max-cpu-set":10,"max-cpu-available":10,"member-initialized":true,"name":"default","data-dir":"/var/lib/rancher/rke2/server/db/tmp-etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/tmp-etcd/member","force-new-clust er":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-cli ent-urls":["http://localhost:2379"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false," initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.238Z","caller":"etcdserver/backend.go:80","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/tmp-etcd/member/snap/db","took":"9.33273ms"} Jan 07 13:23:37 harv2 rancher-system-agent[12856]: time="2022-01-07T13:23:37Z" level=debug msg="Probe output was ok" ... Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.385Z","caller":"etcdserver/raft.go:632","msg":"forcing restart member","cluster-id":"69b872a8734d5c93","local-member-id":"b6eb8b8e53ef3910","commit-index":3419} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.385Z","caller":"raft/raft.go:1530","msg":"b6eb8b8e53ef3910 switched to configuration voters=()"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.386Z","caller":"raft/raft.go:700","msg":"b6eb8b8e53ef3910 became follower at term 2"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.386Z","caller":"raft/raft.go:383","msg":"newRaft b6eb8b8e53ef3910 [peers: [], term: 2, commit: 3419, applied: 0, lastindex: 3419, lastterm: 2]"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"warn","ts":"2022-01-07T13:23:37.624Z","caller":"auth/store.go:1366","msg":"simple token is not cryptographically signed"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.693Z","caller":"etcdserver/quota.go:98","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.724Z","caller":"etcdserver/server.go:803","msg":"starting etcd server","local-member-id":"b6eb8b8e53ef3910","local-server-version":"3.4.13","cluster-version":"to_be_decided"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.725Z","caller":"etcdserver/server.go:691","msg":"starting initial election tick advance","election-ticks":10} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.726Z","caller":"raft/raft.go:1530","msg":"b6eb8b8e53ef3910 switched to configuration voters=(13180782177839692048)"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.726Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"69b872a8734d5c93","local-member-id":"b6eb8b8e53ef3910","added-peer-id":"b6eb8b8e53ef3910","added-peer-peer-urls":[" https://192.168.122.71:2380"]} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.726Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"69b872a8734d5c93","local-member-id":"b6eb8b8e53ef3910","cluster-version":"3.4"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.726Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.737Z","caller":"embed/etcd.go:244","msg":"now serving peer/client/metrics","local-member-id":"b6eb8b8e53ef3910","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http ://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[]} Jan 07 13:23:37 harv2 rke2[15342]: {"level":"info","ts":"2022-01-07T13:23:37.737Z","caller":"embed/etcd.go:579","msg":"serving peer traffic","address":"127.0.0.1:2380"}

w13915984028 commented 2 years ago

Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC -- bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --encryption-provider-config=/var/lib/rancher/rke2/server/cred/e ncryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port =0 --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --profiling=false -- proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/r equest-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-accou nt-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.53.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/ranche r/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key" Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="etcd data store connection OK" Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubecon fig --port=10251 --profiling=false --secure-port=10259" Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Waiting for API server to become available" Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.5 2.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/ser ver/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/ser ver/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud -node-lifecycle --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/serv ice.key --use-service-account-credentials=true" Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=rke2 --cluster-cidr=10.52.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/ran cher/rke2/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m0s --port=0 --profiling=false"

w13915984028 commented 2 years ago

rancher@harv2:~> ps aux | grep kube-api root 2527 29.7 8.7 2623416 1429380 ? Ssl 13:20 19:14 kube-apiserver --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.53.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key

w13915984028 commented 2 years ago

Jan 07 13:23:44 harv2 rke2[15342]: time="2022-01-07T13:23:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"a4559d4b-215f-43d2-9839-c888830b56da\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"353 \", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml\"" Jan 07 13:23:44 harv2 rke2[15342]: time="2022-01-07T13:23:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"a4559d4b-215f-43d2-9839-c888830b56da\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"353 \", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml\"" Jan 07 13:23:44 harv2 rke2[15342]: time="2022-01-07T13:23:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"c42af404-67c5-427f-adb9-b1cc020ca691\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion :\"364\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml\"" Jan 07 13:23:44 harv2 rke2[15342]: time="2022-01-07T13:23:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"c42af404-67c5-427f-adb9-b1cc020ca691\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion :\"364\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml\"" Jan 07 13:23:45 harv2 rke2[15342]: I0107 13:23:45.047647 15342 leaderelection.go:253] successfully acquired lease kube-system/rke2 Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-kube-proxy\", UID:\"\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'DeletingManifest' Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-kube-proxy.yaml\""

w13915984028 commented 2 years ago

Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"b39cc0ec-a6ea-48e1-a487-45d258ac0767\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersio
n:\"386\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"b39cc0ec-a6ea-48e1-a487-45d258ac0767\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersio
n:\"386\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\
", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\
", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 13:23:45 harv2 rke2[15342]: {"level":"warn","ts":"2022-01-07T13:23:45.771Z","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {http://127.0.0.1:2399  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while
dialing dial tcp 127.0.0.1:2399: connect: connection refused\". Reconnecting..."}

w13915984028 commented 2 years ago

Jan 07 13:23:58 harv2 systemd-udevd[15851]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Jan 07 13:23:58 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali315006a2416: link becomes ready

Jan 07 13:23:58 harv2 systemd-udevd[15851]: Using default interface naming scheme 'v238'. Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): caliace2cb55f81: link becomes ready Jan 07 13:23:59 harv2 systemd-udevd[15851]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:23:59 harv2 systemd-udevd[15851]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:23:59 harv2 rke2[15342]: {"level":"warn","ts":"2022-01-07T13:23:59.897Z","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {http://127.0.0.1:2399 0 }. Err :connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2399: connect: connection refused\". Reconnecting..."} Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali561f7ca7121: link becomes ready Jan 07 13:23:59 harv2 systemd-udevd[15851]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali561f7ca7121: link becomes ready Jan 07 13:24:01 harv2 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount988782451.mount: Succeeded.

w13915984028 commented 2 years ago

Jan 07 13:24:33 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:33 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calif3572908c19: link becomes ready Jan 07 13:24:33 harv2 systemd-udevd[17955]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:34 harv2 systemd-udevd[17955]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali51a14b40837: link becomes ready Jan 07 13:24:34 harv2 systemd-udevd[17955]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali1df7168f7c4: link becomes ready Jan 07 13:24:34 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:34 harv2 systemd-udevd[19197]: Using default interface naming scheme 'v238'. Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali101014b67cc: link becomes ready Jan 07 13:24:34 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib90ee855843: link becomes ready Jan 07 13:24:35 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:35 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:35 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali50b7ab19939: link becomes ready Jan 07 13:24:35 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:35 harv2 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount958756371.mount: Succeeded. Jan 07 13:24:35 harv2 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount058476270.mount: Succeeded. Jan 07 13:24:35 harv2 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount680523631.mount: Succeeded. Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calie2d1a383b6f: link becomes ready Jan 07 13:24:36 harv2 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount458926145.mount: Succeeded. Jan 07 13:24:36 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calia9ee36d750f: link becomes ready Jan 07 13:24:36 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:36 harv2 systemd[1]: run-containerd-runc-k8s.io-f3c7c03ec3cea9648e49cf129ca895eb94eb312bc799da7d04e200b442652978-runc.KhaeOJ.mount: Succeeded. Jan 07 13:24:36 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali994de55819d: link becomes ready Jan 07 13:24:36 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0ab93ac548c: link becomes ready Jan 07 13:24:37 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali7f5e84770c3: link becomes ready Jan 07 13:24:37 harv2 systemd-udevd[19197]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

w13915984028 commented 2 years ago

Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] Processing secret custom-331a8e3227cd-machine-plan in namespace fleet-local at generation 0 with resource version 3518" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] Calculated checksum to be 94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] Remote plan had an applied checksum value of 94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] Applied checksum was the same as the plan from remote. Not applying." Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] last applied resource version (3518) did not change. running probes, skipping apply." Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] needsApplied was false, not applying" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[K8s] writing an applied checksum value of 94555a271a8038935ce00608f57d8fd39f68850c527ea1891ebf1d44bf278218 to the remote plan" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kubelet) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kubelet) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (calico) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (calico) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (etcd) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (etcd) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-apiserver) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-apiserver) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-controller-manager) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-controller-manager) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-scheduler) running probe" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="[Prober] (kube-scheduler) retrieving existing probe status from map if existing" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe timeout duration: 5 seconds" Jan 07 13:25:48 harv2 rancher-system-agent[12856]: time="2022-01-07T13:25:48Z" level=debug msg="Probe output was ok"

w13915984028 commented 2 years ago

link:

rancher@harv2:~> sudo -i journalctl --since 13:20:00 | grep "link becomes ready" | more Jan 07 13:21:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:21:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali695b881b527: link becomes ready Jan 07 13:21:29 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:21:29 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali4e3f24d13d3: link becomes ready Jan 07 13:21:30 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calic135dc09e57: link becomes ready Jan 07 13:21:30 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calid49b4d5e3be: link becomes ready Jan 07 13:21:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:21:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali5efd58a3d96: link becomes ready Jan 07 13:21:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali39c12ac9ca3: link becomes ready Jan 07 13:21:49 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0090a6bba35: link becomes ready Jan 07 13:22:13 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali48799b74d12: link becomes ready Jan 07 13:22:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali4550fd920fe: link becomes ready Jan 07 13:22:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali955b469c5af: link becomes ready Jan 07 13:22:27 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:27 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0334b7004ea: link becomes ready Jan 07 13:22:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali28484a44f2b: link becomes ready Jan 07 13:22:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calied9a756505e: link becomes ready Jan 07 13:22:40 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:40 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib48682827be: link becomes ready Jan 07 13:22:42 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:42 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali22641be30f8: link becomes ready Jan 07 13:22:46 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali8d91e76d406: link becomes ready Jan 07 13:22:46 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:22:46 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calid7f79f59651: link becomes ready Jan 07 13:22:49 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali595ee51cc4a: link becomes ready Jan 07 13:22:53 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali5dd9e824784: link becomes ready Jan 07 13:22:56 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calia579026dee0: link becomes ready Jan 07 13:23:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali731e1aeed73: link becomes ready Jan 07 13:23:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali93a6b476e89: link becomes ready Jan 07 13:23:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali169d0e15328: link becomes ready Jan 07 13:23:21 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:23:21 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib1b75183206: link becomes ready Jan 07 13:23:21 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali727c8d69f50: link becomes ready Jan 07 13:23:21 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali25638c47250: link becomes ready Jan 07 13:23:22 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali561f7ca7121: link becomes ready Jan 07 13:23:22 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali11137a869b4: link becomes ready Jan 07 13:23:22 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:23:22 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali315006a2416: link becomes ready Jan 07 13:23:32 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:23:32 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): caliace2cb55f81: link becomes ready Jan 07 13:23:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali076b9f526ae: link becomes ready Jan 07 13:23:58 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali315006a2416: link becomes ready Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): caliace2cb55f81: link becomes ready Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali561f7ca7121: link becomes ready Jan 07 13:23:59 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali561f7ca7121: link becomes ready Jan 07 13:24:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:23 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calif2ff3e4d4a7: link becomes ready Jan 07 13:24:28 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali41f62b79525: link becomes ready Jan 07 13:24:33 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:33 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calif3572908c19: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali51a14b40837: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali1df7168f7c4: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali101014b67cc: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:34 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib90ee855843: link becomes ready Jan 07 13:24:35 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali50b7ab19939: link becomes ready Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calie2d1a383b6f: link becomes ready Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calia9ee36d750f: link becomes ready Jan 07 13:24:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali994de55819d: link becomes ready Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0ab93ac548c: link becomes ready Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali7f5e84770c3: link becomes ready Jan 07 13:24:38 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:38 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calic060ca688a6: link becomes ready Jan 07 13:24:41 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:41 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calia9c976fedcf: link becomes ready Jan 07 13:24:56 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:56 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calid2ac83c57ae: link becomes ready Jan 07 13:24:56 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0f4bfd161dd: link becomes ready Jan 07 13:24:57 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali9a0ca5e0470: link becomes ready Jan 07 13:24:57 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:24:57 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali0f4bfd161dd: link becomes ready Jan 07 13:25:10 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:25:10 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali672ff30a431: link becomes ready Jan 07 13:25:10 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali8e2078c8378: link becomes ready Jan 07 13:25:11 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali2a4d1eb6a5c: link becomes ready Jan 07 13:25:14 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calie2ed40536cb: link becomes ready Jan 07 13:25:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:25:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali07b8621c058: link becomes ready Jan 07 13:25:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali42748360148: link becomes ready Jan 07 13:25:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib2077578a18: link becomes ready Jan 07 13:25:36 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali59740900e87: link becomes ready Jan 07 13:25:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calif83974f8c40: link becomes ready Jan 07 13:25:37 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:25:47 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calib951c0d8987: link becomes ready Jan 07 13:26:01 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calie2c8bb996c3: link becomes ready Jan 07 13:26:03 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:26:03 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): calid05e9936edd: link becomes ready Jan 07 13:26:07 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:26:07 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali47054f35f15: link becomes ready Jan 07 13:26:20 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali64c17718f98: link becomes ready Jan 07 13:26:38 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Jan 07 13:26:38 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali81cba82cd97: link becomes ready Jan 07 13:27:01 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali99ee9a714a6: link becomes ready Jan 07 13:27:03 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali70a5d2c8e40: link becomes ready Jan 07 13:27:06 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali9f5de02cf74: link becomes ready Jan 07 13:27:06 harv2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali9f5de02cf74: link becomes ready

w13915984028 commented 2 years ago

rancher@harv2:~> sudo -i journalctl --since 13:20:00 | grep "cali" | more Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values" Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values" Jan 07 13:20:42 harv2 rke2[2175]: time="2022-01-07T13:20:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-calico-crd\", UID:\"\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'DeletingManifest' Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml\"" Jan 07 13:20:42 harv2 rke2[2175]: time="2022-01-07T13:20:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-calico\", UID:\"\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reas on: 'DeletingManifest' Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-calico.yaml\"" Jan 07 13:21:20 harv2 systemd-udevd[3389]: calico_tmp_B: Could not generate persistent MAC: No data available

w13915984028 commented 2 years ago

rancher@harv2:~> sudo -i cat /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helm.cattle.io/chart-url: https://rke2-charts.rancher.io/assets/rke2-multus/rke2-multus-v3.7.1-build2021111906.tgz
  creationTimestamp: null
  name: rke2-multus
  namespace: kube-system
spec:
  bootstrap: true
  chartContent: H4sIFAAAAAAA/ykAK2FIUjBjSE02THk5NWIzVjBkUzVpWlM5Nk9WVjZNV2xqYW5keVRRbz1IZWxtAOxaX2/bOBLvsz7FwMWhwOEsyXH+7BpY4Lppsch1myuSQ+9hcShoaiTzQpE6cuQmW+S7H0jKsiQ7cdOmKQp4XmxR5HCG84c/jmiu8GBc1pJqm5wumKH4hpXy2aNSmqbp8eGh/03TdPibnkymzyaHk8PjdJpODqbP0sl0enT8DNLHFWM71ZaYeZZ+9VxD5X4QYpV4j8YKrWawnESsqtaP0/gknkQZWm5ERb7trfcUOD0/A1RsLtECI2J8IVQBzo1EJREU0kdtrkAoQpMzjhZIQ6UzC0LBm3qORiGhjaOFLnEGC6LKzpKkELSo5zHXZXL1k224VLIu50IVH4sk+OmYKxEJ7sRZDTTsYxwG1xYN14pQ0efwSUpmCU2SaZ6IkhVok6BiXKkiKplQxIRCY2fRGLBkQs6AuzCxfzdM8QUaN0sEoJhT5CK0we9sbqPQ1AmwyOracPS8vkDlZd8w43ktZHaQHkwmk8nP6XH0Rfbvxv+SyRrt4yeAXfF/NDkYxv/xZLKP/6egYPpZBODd3/0BMFhpK0ibmxk0bp4smMlQYTbueKTrS6zY7pFRxJWoZF0I9SD+XIlxM6wzQRr/vDkBwHOwV6LiSlgQFhhwXZYMLFbMMMIMpLAEOgeuBMyFYkaEXORGQW586D4HoSwxKYUq4ghajjPIJVMKZRQVUs+ZdMLbG0tYvsKc1ZIusBCWnBaj0ZeF33enbvwTlpVkhDb5sEBZobExVY+QCnbF/zQd7v9HJ4eH+/h/Cvr0aQwZ5kIhjIJvf8iCc38wjXePYHx7G7mOIof4fdgkQkTEW8Oh7V8ZoSiH0V9sMnrYSJQW24fRWgJU2cb/772EPzRtj38uaweLjJb4CHBgV/y7d/34Pz4+3u//T0JXQmUzOA32vtASo+6JwMwZj1lNC23En8wdAeKrn2wsdLKcRCUSyxixWQt/G5hraol+yx8Dq8RvRteVncEfIzeUKxFzxfNY6NF/GjDQwmLwNIYXf33h/y/RzDeau0zbV6PRXbzcqaP3kFhiTsotExRI7f+6yhjhjvncX1yiItusy11ShE7b5uQGwzyNhIz4YiDEt7T/zvifC5U5YPQVaWBX/B+mA/x/kB6k+/3/SWgj/n8N9n6MNKAlXmDu2lcRdA+rCGAzFw042nr+X+TkD9Ch8yWapeD4knNdKxr2D4+2Yhxn8OkTxBcokVmMz1fNcHv7vS3wfWl7/GcMS60sPk41cEf8Tw9Ojob7/zRN9/H/FNQNc1ZV1kV0CK1X3gcukbZE+FU9x1UhIPvMOItshdwfoVEiJ23CHli6De93NkfZboqDGF65ZdO/I4wj2Ru6MRhgNa2jhbZ0HmpsMyBTr/Zd0hKNz0adnVlXrk2bGby+FpZsOwXmOXKawbm+5AvMaokPGfP6Gnnd7vi2l8DOh8IDCCXoVK/LkKuZgp7DYokf4Qstzg6rlbvnaBfD7a2zWHM4W5dsYs8mXpdq4PZ2dl9PYoWz8koKi7w2gm6c7HhNawOBOxYuhcQCs54RAJZa1iW+dSvRsWhH17lQHT6l6/iO0WIGibNsoitKuBJJtxuq5Saryzdn7z6cnp99+PXs/OXF2evLDldfBQ1uvKlqW21qNeV32qYTJI9im8Bql116vQY28dUxF9t/jBJUZG4qLRTFdtFAcUfMFL21H43bip9W+TgXEn9hNelRv49zxKZA/UsaT+PJ+v0GHA6N/6vRUq8NgFf1DEaTNC1HvfYSS18sHB2lb0X3lRSl+Eom38RR7/NSJO69VCHF2aN4OaEphfL56zfDOL5DI3R2iVyrzM5gkkZdsXdK7GbxE/aWrwoiPFD6e3ltaqJ0hpe9/cHR1fqLkdCJtjOQQtXX+8rP19N2/Nd8hwqf9kpUvkIo/P4Yc5M9EBXuwn8nR5PB+e/gJN3Xf56E+vhP4DWhst7O6xNecy6rLenyoknmr1qP2AIPG/cZt/5D464HDepALTAswhFx+BrAcl3hDFo0ma0QZ0gRlawNk7umDWBQqKKWzOzo7PsGtRu4+LLt9qrfyy608bitEWbsODdcHcfIV3zsGlquUuVy0oGAgx3GkjYeKHTb+AJLtk6KukL18t3Z++nl4AVA75P9i3s0CB/NTi9eNdw9WBa5wAzmN0ALXKkP75pP0r00/m9troQqwJ/tgTTgdWXQWj9S+I/wkGvTuSLg7wCQBq0QtIFSGwSpC8GZdM/V4sa6/71ZGkvZGN667kLl2pR+swO2ZEKyuURg9LBrBGLszOTUfdGZjG6cn2lfZejhAIfrSeAAaXSCp9c+tIAzU+jXfGsJK9T4BWar1dc50EJYMOiWEdWAKQAFtXUOTDVixr4GgsY6T6xl5gDpEg2BQa4LJf5s2fuFpwVucPU5n8JtDcVkQMB/A6YyKNkNGHTzQK06HJurAmuLDBcAutZY5RKHP2sl6CZxqNmIeU3a2CTDJcrEimLMDF8IQk61wYRVYuyVUSFtlNnzFkq+GEwXDGfJDF3UB/G9xnkjVBbiIAwPurU2INfkrHXx+vJfLZb1dtpQOVhkPdSujeMWUqgcTbBwbnTpuaLKPAj3D1wKVLTB1tZzh3JbzAykYzhlSmmCOTZF4iyGMwWnrER56o7eHdN8E1O4Fbdjt7yfb4zh2b0/YCPs+mf37da7L7m50b2Ay9AK4+KB3MHLRVF7VWm9DWzXZotwd+cFCOfCXBSb7Q9RIPAIzvmPy3+ej0PmI8z8Dazwug6Fi6HYQ0Pcsf9vx39NVYKFqsTXFgF31f8nh8P7PyfTg339/0locP9va1X9zgr/vr6+pz3taU8/LP0/AAD//3eiN0AAMAAA
  set:
    global.clusterCIDR: 10.52.0.0/16
    global.clusterCIDRv4: 10.52.0.0/16
    global.clusterCIDRv6: ""
    global.clusterDNS: 10.53.0.10
    global.clusterDomain: cluster.local
    global.rke2DataDir: /var/lib/rancher/rke2
    global.serviceCIDR: 10.53.0.0/16
    global.systemDefaultRegistry: ""
status: {}
rancher@harv2:~>
rancher@harv2:~>
rancher@harv2:~>
rancher@harv2:~>
rancher@harv2:~> sudo -i journalctl --since 13:20:00 | grep multus | more
Jan 07 13:20:10 harv2 rke2[2175]: time="2022-01-07T13:20:10Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Jan 07 13:20:45 harv2 rke2[2175]: time="2022-01-07T13:20:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\"
, FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 13:20:45 harv2 rke2[2175]: time="2022-01-07T13:20:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\"
, FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 13:23:43 harv2 rke2[15342]: time="2022-01-07T13:23:43Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\
", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 13:23:45 harv2 rke2[15342]: time="2022-01-07T13:23:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"c94f867d-1280-4a91-be29-6c01622ebd44\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"395\
", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
Jan 07 14:50:13 harv2 sudo[29871]:  rancher : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/bash --login -c cat \/var\/lib\/rancher\/rke2\/server\/manifests\/rke2-multus\.yaml
rancher@harv2:~>

w13915984028 commented 2 years ago

first potential issue: in short: In harvester starting process, rancher-system-agent will do an "systemctl --no-block restart rke2-server", (not sure the reason, but it is easy to get such log from journalctl), which causes rke2 process is killed and started again, the rke2 will try to spawn kube related components again, here kube-api is example, it does not check if kube-api is already there. due to things like TCP port binding, the latter try will fail, from the process, its starting time is equal to the first rke2 run. but, the re-spawn same kube components still could harm.

raise discussion in slack.