Open AllardKrings opened 11 months ago
Hi @AllardKrings
Can you share an inspection report from the cluster? Looks like the Calico CNI fails to initialize for some reason
Thanks, I see the following line in Kubelet logs, which causes the services to continuously restart
Nov 12 13:46:23 odroid microk8s.daemon-kubelite[359263]: E1112 13:46:23.145422 359263 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : open /sys/fs/cgroup/kubepods/besteffort/cpu.weight: no such file or directory"
This typically means that cgroups are not configured properly on the system.
As a workaround, can you try appending the following kubelet arguments:
echo '
--cgroups-per-qos=false
--enforce-node-allocatable=""
' | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite
Hello,
I tried this but to no avail.
The sbc refuses to boot now.
I wil flash a fresh image tonight and install microk8s again.
I find it strange that a fresh ubuntu and a fresh microk8s do not work together.
Any suggestions on another linux-flavour?
Kind regards.
Op di 14 nov 2023 om 12:52 schreef Angelos Kolaitis < @.***>
Thanks, I see the following line in Kubelet logs, which causes the services to continuously restart
Nov 12 13:46:23 odroid microk8s.daemon-kubelite[359263]: E1112 13:46:23.145422 359263 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : open /sys/fs/cgroup/kubepods/besteffort/cpu.weight: no such file or directory"
This typically means that cgroups are not configured properly on the system.
As a workaround, can you try appending the following kubelet arguments:
echo ' --cgroups-per-qos=false --enforce-node-allocatable="" ' | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite
— Reply to this email directly, view it on GitHub https://github.com/canonical/microk8s/issues/4301#issuecomment-1810061954, or unsubscribe https://github.com/notifications/unsubscribe-auth/AET5SGPDPTDKXQI63NRCRT3YENLPTAVCNFSM6AAAAAA7KSQOJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGA3DCOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Summary
when installing microk8s on a clean ubuntu-Mate on an odroid N2+ SBC microk8s does not start
ubuntu:
https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622 microk8s 1.28
What Should Happen Instead?
microk8s should start
Reproduction Steps
Install Ubuntu/Mate on Odroid N2+ https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622
Install Microk8s using snap
Introspection Report
kubectl get pod -n kube-system gives:
coredns-864597b5fd-nqn2p 0/1 Pending 0 8m2s calico-kube-controllers-77bd7c5b-fnzxk 0/1 Pending 0 8m2s calico-node-bf62c 0/1 Init:CrashLoopBackOff 5 (70s ago) 8m2s
kubectl logs calico-node-bf62c -n kube-system gives:
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init) Error from server (BadRequest): container "calico-node" in pod "calico-node-bf62c" is waiting to start: PodInitializing
kubectl describe pod calico-node-bf62c -n kube-system gives:
Name: calico-node-bf62c Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Service Account: calico-node Node: odroid/192.168.2.28 Start Time: Tue, 14 Nov 2023 10:19:08 +0000 Labels: controller-revision-hash=6fbb45588b k8s-app=calico-node pod-template-generation=1 Annotations:
Status: Pending
IP: 192.168.2.28
IPs:
IP: 192.168.2.28
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: containerd://6fdad669f5792ea1618ca5ae284d3725f77b59a7d15907cc17ff34c2a14b54ec
Image: docker.io/calico/cni:v3.25.1
Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a
Port:
Host Port:
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 14 Nov 2023 10:19:50 +0000
Finished: Tue, 14 Nov 2023 10:19:50 +0000
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
install-cni:
Container ID: containerd://be5e122180d5d88f1765c671d592e62ba0fd33deafc6bdf307d5d8b15d1f22e1
Image: docker.io/calico/cni:v3.25.1
Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a
Port:
Host Port:
Command:
/opt/cni/bin/install
State: Running
Started: Tue, 14 Nov 2023 10:34:18 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 14 Nov 2023 10:28:45 +0000
Finished: Tue, 14 Nov 2023 10:29:17 +0000
Ready: False
Restart Count: 7
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
CNI_NET_DIR: /var/snap/microk8s/current/args/cni-network
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
Containers:
calico-node:
Container ID:
Host Port:
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
IP_AUTODETECTION_METHOD: first-found
CALICO_IPV4POOL_VXLAN: Always
CALICO_IPV6POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 10.1.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
Image: docker.io/calico/node:v3.25.1 Image ID:
Port:
var-run-calico: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/run/calico HostPathType:
var-lib-calico: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/lib/calico HostPathType:
xtables-lock: Type: HostPath (bare host directory volume) Path: /run/xtables.lock HostPathType: FileOrCreate sys-fs: Type: HostPath (bare host directory volume) Path: /sys/fs/ HostPathType: DirectoryOrCreate cni-bin-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/opt/cni/bin HostPathType:
cni-net-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/args/cni-network HostPathType:
cni-log-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/common/var/log/calico/cni HostPathType:
host-local-net-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/lib/cni/networks HostPathType:
policysync: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/run/nodeagent HostPathType: DirectoryOrCreate kube-api-access-wlsfs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional:
Normal Pulled 15s (x8 over 14m) kubelet Container image "docker.io/calico/cni:v3.25.1" already present on machine