Version (k3OS / kernel)
k3os version v0.21.1-k3s1r0
5.4.0-73-generic rancher/k3os#82 SMP Thu Jun 3 02:29:43 UTC 2021
Architecture
x86_64
Describe the bug
Hello,
i have some problems wiht my Kubernetes cluster. I use k3os version v0.21.1-k3s1r0. Last week my vHost goes offline for a short time. My provider had a failure.
Since then, my cluster does not boot up properly. Nothing has changed in the configurations.
kubectl -n cattle-system logs -f rancher-6d8598477d-h8cf2
2021/11/01 21:48:55 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:03 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:11 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:19 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:23 [ERROR] error syncing 'p-8mlxr/akomljen-charts': handler projectCatalog: Timeout in HTTP GET to [https://raw.githubusercontent.com/komljen/helm-charts/master/charts/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:23 [ERROR] error syncing 'p-g7r2v/wg-access-server': handler projectCatalog: Timeout in HTTP GET to [https://place1.github.io/wg-access-server/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:27 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:36 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
2021/11/01 21:49:42 [ERROR] error syncing 'p-8mlxr/elastic': handler projectCatalog: Timeout in HTTP GET to [https://helm.elastic.co/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:42 [ERROR] error syncing 'p-g7r2v/bitnami': handler projectCatalog: Timeout in HTTP GET to [https://charts.bitnami.com/bitnami/index.yaml], did not respond in 30s, requeuing
2021/11/01 21:49:44 [ERROR] Failed to connect to peer wss://10.42.0.108/v3/connect [local ID=10.42.0.138]: dial tcp 10.42.0.108:443: connect: no route to host
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-7448499f4d-plr2q 1/1 Running 0 85d
kubectl describe pod coredns-7448499f4d-plr2q --namespace=kube-system
Name: coredns-7448499f4d-plr2q
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: upspot-cluster/***.***.***.***
Start Time: Sun, 08 Aug 2021 12:36:18 +0000
Labels: k8s-app=kube-dns
pod-template-hash=7448499f4d
Annotations: <none>
Status: Running
IP: 10.42.0.76
IPs:
IP: 10.42.0.76
Controlled By: ReplicaSet/coredns-7448499f4d
Containers:
coredns:
Container ID: containerd://0620e642d29c8014e130153dfb461d4f2ed6c66d18d3591b69d94289288aa53e
Image: rancher/coredns-coredns:1.8.3
Image ID: docker.io/rancher/coredns-coredns@sha256:cd0f6ef34a9ee6145c0bcbc0a4511a04f81dc2c91582140f0f0250be834b5f27
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Running
Started: Mon, 01 Nov 2021 18:38:38 +0000
Ready: True
Restart Count: 0
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rwbk8 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
kube-api-access-rwbk8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51c9ae6a65d704a3a4040261f56158523d29dfd1292e3b0cbade6d9e9bd579e4": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b72d7dbff055840ff68f3d364795d23ff6235e5db889131e20b8241f6138e8ab": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ca3f773654508caddcc648cda24d9fe10e34547b9bb515860ffe266915b4b4c4": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d01b13574e05d65dcb8656b1c7041356f93510711068d21572071525c94d6000": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f570ffd13e461cdc0853a990281617c5091cc4e004a28c9ae67c8b3d26f098ae": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe6a0c549e77626942ad9e35ac4562ed95cb1508cb56c2d625ff5ca2386e982e": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ab0b2c71df867d1e95351b8841ad89ff75266fc4362e370b7aa0aef28bb10d33": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "41da419b6c354b515f27b35548c4a95c4fc9ea9e064317126ebce7690d243332": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ed13b48e3715625a62f60f25d487790075deeec66d85998a227f54fef954be3e": open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 31h (x3 over 31h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0e7febedb6943de8b7c295cc10afdafb0c1f848aeb85f4cc8b0ece0159eb14ef": open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 27h (x1149 over 31h) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedMount 27h kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
Normal Created 27h kubelet Created container coredns
Normal Pulled 27h kubelet Container image "rancher/coredns-coredns:1.8.3" already present on machine
Normal Started 27h kubelet Started container coredns
Warning NodeNotReady 27h node-controller Node is not ready
Normal Pulled 27h kubelet Container image "rancher/coredns-coredns:1.8.3" already present on machine
Normal Created 27h kubelet Created container coredns
Normal Started 27h kubelet Started container coredns
Warning Unhealthy 27h kubelet Readiness probe failed: Get "http://10.42.0.29:8181/ready": dial tcp 10.42.0.29:8181: connect: connection refused
Warning NodeNotReady 131m node-controller Node is not ready
Normal Pulled 125m kubelet Container image "rancher/coredns-coredns:1.8.3" already present on machine
Normal Created 125m kubelet Created container coredns
Normal Started 125m kubelet Started container coredns
kubectl describe nodes
Name: upspot-cluster
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=k3s
beta.kubernetes.io/os=linux
k3os.io/mode=local
k3os.io/version=v0.21.1-k3s1r0
kubernetes.io/arch=amd64
kubernetes.io/hostname=upspot-cluster
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=true
node-role.kubernetes.io/master=true
node.kubernetes.io/instance-type=k3s
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"***"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: *.*.*.*
k3s.io/hostname: upspot-cluster
k3s.io/internal-ip: *.*.*.*
k3s.io/node-args: ["server","--no-deploy","traefik","--node-label","k3os.io/mode=local","--node-label","k3os.io/version=v0.21.1-k3s1r0"]
k3s.io/node-config-hash: L3CO4RG3V7U4SLPXOQEKR6RYZMQJODZUCK2BVNSCXA7HTV36QU7Q====
k3s.io/node-env:
{"K3S_CLUSTER_SECRET":"********","K3S_DATA_DIR":"/var/lib/rancher/k3s/data/77457d0b09d8b94d6f5029bcbc70f94b7ae9c50a08b539b76612e713ea81825...
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 31 Oct 2021 12:51:56 +0000
Taints: node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: upspot-cluster
AcquireTime: <unset>
RenewTime: Mon, 01 Nov 2021 20:56:50 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Mon, 01 Nov 2021 20:50:45 +0000 Mon, 01 Nov 2021 20:50:45 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Mon, 01 Nov 2021 20:56:51 +0000 Mon, 01 Nov 2021 20:49:42 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 01 Nov 2021 20:56:51 +0000 Mon, 01 Nov 2021 20:49:42 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 01 Nov 2021 20:56:51 +0000 Mon, 01 Nov 2021 20:49:42 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Mon, 01 Nov 2021 20:56:51 +0000 Mon, 01 Nov 2021 20:50:45 +0000 KubeletNotReady PLEG is not healthy: pleg has yet to be successful
Addresses:
InternalIP: *.*.*.*
Hostname: upspot-cluster
Capacity:
cpu: 8
ephemeral-storage: 827392920Ki
hugepages-2Mi: 0
memory: 32886628Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 804887831945
hugepages-2Mi: 0
memory: 32886628Ki
pods: 110
System Info:
Machine ID: 4a9d40c633f866838c2c04d460b91f75
System UUID: 0a0f4d22-8e23-45b4-aa30-79f8b85f2989
Boot ID: 9773ef9f-9974-4931-b20e-b3e90fe51216
Kernel Version: 5.4.0-73-generic
OS Image: k3OS v0.21.1-k3s1r0
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.4.4-k3s2
Kubelet Version: v1.21.1+k3s1
Kube-Proxy Version: v1.21.1+k3s1
PodCIDR: 10.42.0.0/24
PodCIDRs: 10.42.0.0/24
ProviderID: k3s://upspot-cluster
Non-terminated Pods: (55 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default svclb-ingress-nginx-controller-n9vvp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 86d
default ingress-nginx-controller-68649d49b8-g25vh 100m (1%) 0 (0%) 90Mi (0%) 0 (0%) 85d
test order-management-api-postgresql-postgresql-0 250m (3%) 0 (0%) 256Mi (0%) 0 (0%) 3d18h
test document-management-api-mongodb-5f46494477-nqzbl 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
test notification-management-api-mongodb-8458655556-m86xx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
test document-management-api-7b6664966f-2grtb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
test order-management-api-axon-5c4f84d668-njwzh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13d
keycloak keycloak-postgresql-0 250m (3%) 0 (0%) 256Mi (0%) 0 (0%) 3d18h
test order-management-api-mongodb-774f8549cd-cnvd6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13d
test user-management-api-mongodb-7cfcc95977-4jr6q 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3d18h
cattle-resources-system rancher-backup-d9d984f74-h6cn7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
longhorn-system csi-snapshotter-69f8bc8dcf-dh85b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system csi-resizer-5c88bfd4cf-tfqf2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
kube-system coredns-7448499f4d-plr2q 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 85d
rancher-operator-system rancher-operator-595ddc6db9-w4jsj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
longhorn-system csi-attacher-75588bff58-pdmt7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system longhorn-driver-deployer-5fc857649c-mgp8m 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
cattle-monitoring-system rancher-monitoring-operator-9779869d-6r4mh 100m (1%) 200m (2%) 100Mi (0%) 500Mi (1%) 33d
longhorn-system longhorn-manager-ll268 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system engine-image-ei-0f7c4304-d5l2p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
cert-manager cert-manager-webhook-7dc86765b4-kdsvv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
test planung-ui-959b9d47c-x8djr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3d8h
cattle-monitoring-system alertmanager-rancher-monitoring-alertmanager-0 200m (2%) 1100m (13%) 150Mi (0%) 550Mi (1%) 33d
cattle-monitoring-system pushprox-k3s-server-client-qwhvq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 33d
longhorn-system csi-resizer-5c88bfd4cf-ghxfz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
kube-system metrics-server-86cbb8457f-tkq6f 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
fleet-system fleet-controller-79554fcbf5-9zw2d 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
cattle-monitoring-system rancher-monitoring-grafana-6c9976ccc6-njtt6 100m (1%) 200m (2%) 100Mi (0%) 200Mi (0%) 33d
longhorn-system engine-image-ei-a5a44787-7q78r 0 (0%) 0 (0%) 0 (0%) 0 (0%) 86d
cert-manager cert-manager-cainjector-595fc4d55-ljzn2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
cattle-system rancher-webhook-6cccfd96b5-v5v24 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
longhorn-system csi-snapshotter-69f8bc8dcf-cjvnk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system csi-attacher-75588bff58-rr57b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system csi-provisioner-6968cf94f9-wlrql 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
kube-system local-path-provisioner-5ff76fc89d-d4pdw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
longhorn-system csi-provisioner-6968cf94f9-fb894 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system csi-provisioner-6968cf94f9-vgxks 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
cert-manager cert-manager-58bbf5798-cs5x2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
cattle-monitoring-system pushprox-k3s-server-proxy-6946957f87-r75k8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 33d
longhorn-system csi-attacher-75588bff58-rlk42 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
longhorn-system csi-snapshotter-69f8bc8dcf-wcfbz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
cattle-monitoring-system rancher-monitoring-kube-state-metrics-5f56fc44ff-vfp4c 100m (1%) 100m (1%) 130Mi (0%) 200Mi (0%) 33d
longhorn-system csi-resizer-5c88bfd4cf-mqwj4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
fleet-system gitjob-568c57cfb9-c4mww 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
test notification-management-api-bd7d67c5d-wfs4q 0 (0%) 0 (0%) 0 (0%) 0 (0%) 35d
fleet-system fleet-agent-d59db746-h7nqt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
cattle-monitoring-system rancher-monitoring-prometheus-node-exporter-rmz2w 100m (1%) 200m (2%) 30Mi (0%) 50Mi (0%) 33d
cattle-monitoring-system rancher-monitoring-prometheus-adapter-58d5b96d85-wwh45 0 (0%) 0 (0%) 0 (0%) 0 (0%) 33d
cattle-system rancher-6d8598477d-h8cf2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 85d
cattle-monitoring-system prometheus-rancher-monitoring-prometheus-0 850m (10%) 1100m (13%) 1800Mi (5%) 2550Mi (7%) 33d
longhorn-system longhorn-ui-64b96684bc-8jbmk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
keycloak keycloak-0 100m (1%) 0 (0%) 0 (0%) 0 (0%) 15d
longhorn-system longhorn-csi-plugin-gxqb2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36d
test user-management-api-76c74dc97b-bxjtk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 25d
test order-managament-api-7d654d8d84-jvnf5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 2250m (28%) 2900m (36%)
memory 2982Mi (9%) 4220Mi (13%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning InvalidDiskCapacity 32h kubelet invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced 32h kubelet Updated Node Allocatable limit across pods
Normal Starting 32h kubelet Starting kubelet.
Normal NodeHasNoDiskPressure 32h (x7 over 32h) kubelet Node upspot-cluster status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 32h (x7 over 32h) kubelet Node upspot-cluster status is now: NodeHasSufficientPID
Normal Starting 32h kube-proxy Starting kube-proxy.
Normal NodeHasSufficientMemory 27h (x105183 over 32h) kubelet Node upspot-cluster status is now: NodeHasSufficientMemory
Normal Starting 27h kube-proxy Starting kube-proxy.
Normal Starting 27h kubelet Starting kubelet.
Warning InvalidDiskCapacity 27h kubelet invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced 27h kubelet Updated Node Allocatable limit across pods
Normal NodeReady 27h (x3 over 27h) kubelet Node upspot-cluster status is now: NodeReady
Normal NodeHasSufficientMemory 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasSufficientPID
Warning Rebooted 27h (x92 over 27h) kubelet Node upspot-cluster has been rebooted, boot id: 7cb8d273-5e32-47f1-9914-cc731dc83bda
Normal Starting 27h kube-proxy Starting kube-proxy.
Normal Starting 27h kubelet Starting kubelet.
Warning InvalidDiskCapacity 27h kubelet invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced 27h kubelet Updated Node Allocatable limit across pods
Normal NodeNotReady 27h (x2 over 27h) kubelet Node upspot-cluster status is now: NodeNotReady
Normal NodeReady 27h kubelet Node upspot-cluster status is now: NodeReady
Normal NodeHasSufficientPID 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientMemory 27h (x5 over 27h) kubelet Node upspot-cluster status is now: NodeHasSufficientMemory
Warning Rebooted 26h (x211 over 27h) kubelet Node upspot-cluster has been rebooted, boot id: f9d367b9-bf7a-4998-886f-3fb4fc1a83be
Normal Starting 138m kube-proxy Starting kube-proxy.
Normal Starting 138m kubelet Starting kubelet.
Warning InvalidDiskCapacity 138m kubelet invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced 138m kubelet Updated Node Allocatable limit across pods
Normal NodeNotReady 138m (x2 over 138m) kubelet Node upspot-cluster status is now: NodeNotReady
Normal NodeReady 138m kubelet Node upspot-cluster status is now: NodeReady
Normal NodeHasSufficientMemory 138m (x5 over 138m) kubelet Node upspot-cluster status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 138m (x5 over 138m) kubelet Node upspot-cluster status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 138m (x5 over 138m) kubelet Node upspot-cluster status is now: NodeHasSufficientPID
Warning Rebooted 8m22s (x777 over 138m) kubelet Node upspot-cluster has been rebooted, boot id: 1f0cbe8e-20a6-4c87-ab20-5dd1a4a8fc68
Normal Starting 6m13s kube-proxy Starting kube-proxy.
Normal Starting 6m9s kubelet Starting kubelet.
Warning InvalidDiskCapacity 6m9s kubelet invalid capacity 0 on image filesystem
Normal NodeHasNoDiskPressure 6m9s kubelet Node upspot-cluster status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 6m9s kubelet Node upspot-cluster status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 6m9s kubelet Updated Node Allocatable limit across pods
Normal NodeNotReady 6m9s (x2 over 6m9s) kubelet Node upspot-cluster status is now: NodeNotReady
Normal NodeHasSufficientMemory 6m9s kubelet Node upspot-cluster status is now: NodeHasSufficientMemory
Warning Rebooted 64s (x32 over 6m9s) kubelet Node upspot-cluster has been rebooted, boot id: 9773ef9f-9974-4931-b20e-b3e90fe51216
If you need more/other infomation let me know!
Thanks for your help.
Version (k3OS / kernel) k3os version v0.21.1-k3s1r0 5.4.0-73-generic rancher/k3os#82 SMP Thu Jun 3 02:29:43 UTC 2021
Architecture x86_64
Describe the bug Hello, i have some problems wiht my Kubernetes cluster. I use k3os version v0.21.1-k3s1r0. Last week my vHost goes offline for a short time. My provider had a failure. Since then, my cluster does not boot up properly. Nothing has changed in the configurations.
If you need more/other infomation let me know! Thanks for your help.