search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.48k stars 772 forks source link

Microk8s not starting on ubuntu/odroid #4301

Open AllardKrings opened 11 months ago

AllardKrings commented 11 months ago

Summary

when installing microk8s on a clean ubuntu-Mate on an odroid N2+ SBC microk8s does not start

ubuntu:

https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622 microk8s 1.28

What Should Happen Instead?

microk8s should start

Reproduction Steps

  1. Install Ubuntu/Mate on Odroid N2+ https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622

  2. Install Microk8s using snap

Introspection Report

kubectl get pod -n kube-system gives:

coredns-864597b5fd-nqn2p 0/1 Pending 0 8m2s calico-kube-controllers-77bd7c5b-fnzxk 0/1 Pending 0 8m2s calico-node-bf62c 0/1 Init:CrashLoopBackOff 5 (70s ago) 8m2s

kubectl logs calico-node-bf62c -n kube-system gives:

Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init) Error from server (BadRequest): container "calico-node" in pod "calico-node-bf62c" is waiting to start: PodInitializing

kubectl describe pod calico-node-bf62c -n kube-system gives:

Name: calico-node-bf62c Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Service Account: calico-node Node: odroid/192.168.2.28 Start Time: Tue, 14 Nov 2023 10:19:08 +0000 Labels: controller-revision-hash=6fbb45588b k8s-app=calico-node pod-template-generation=1 Annotations: Status: Pending IP: 192.168.2.28 IPs: IP: 192.168.2.28 Controlled By: DaemonSet/calico-node Init Containers: upgrade-ipam: Container ID: containerd://6fdad669f5792ea1618ca5ae284d3725f77b59a7d15907cc17ff34c2a14b54ec Image: docker.io/calico/cni:v3.25.1 Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a Port: Host Port: Command: /opt/cni/bin/calico-ipam -upgrade State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 14 Nov 2023 10:19:50 +0000 Finished: Tue, 14 Nov 2023 10:19:50 +0000 Ready: True Restart Count: 0 Environment Variables from: kubernetes-services-endpoint ConfigMap Optional: true Environment: KUBERNETES_NODE_NAME: (v1:spec.nodeName) CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false Mounts: /host/opt/cni/bin from cni-bin-dir (rw) /var/lib/cni/networks from host-local-net-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro) install-cni: Container ID: containerd://be5e122180d5d88f1765c671d592e62ba0fd33deafc6bdf307d5d8b15d1f22e1 Image: docker.io/calico/cni:v3.25.1 Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a Port: Host Port: Command: /opt/cni/bin/install State: Running Started: Tue, 14 Nov 2023 10:34:18 +0000 Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 14 Nov 2023 10:28:45 +0000 Finished: Tue, 14 Nov 2023 10:29:17 +0000 Ready: False Restart Count: 7 Environment Variables from: kubernetes-services-endpoint ConfigMap Optional: true Environment: CNI_CONF_NAME: 10-calico.conflist CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false KUBERNETES_NODE_NAME: (v1:spec.nodeName) CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false SLEEP: false CNI_NET_DIR: /var/snap/microk8s/current/args/cni-network Mounts: /host/etc/cni/net.d from cni-net-dir (rw) /host/opt/cni/bin from cni-bin-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro) Containers: calico-node: Container ID:
Image: docker.io/calico/node:v3.25.1 Image ID:
Port: Host Port: State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Requests: cpu: 250m Liveness: exec [/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6 Readiness: exec [/bin/calico-node -felix-ready] delay=0s timeout=10s period=10s #success=1 #failure=3 Environment Variables from: kubernetes-services-endpoint ConfigMap Optional: true Environment: DATASTORE_TYPE: kubernetes WAIT_FOR_DATASTORE: true NODENAME: (v1:spec.nodeName) CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false CLUSTER_TYPE: k8s,bgp IP: autodetect IP_AUTODETECTION_METHOD: first-found CALICO_IPV4POOL_VXLAN: Always CALICO_IPV6POOL_VXLAN: Never FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false CALICO_IPV4POOL_CIDR: 10.1.0.0/16 CALICO_DISABLE_FILE_LOGGING: true FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT FELIX_IPV6SUPPORT: false FELIX_HEALTHENABLED: true FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true Mounts: /host/etc/cni/net.d from cni-net-dir (rw) /lib/modules from lib-modules (ro) /run/xtables.lock from xtables-lock (rw) /var/lib/calico from var-lib-calico (rw) /var/log/calico/cni from cni-log-dir (ro) /var/run/calico from var-run-calico (rw) /var/run/nodeagent from policysync (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: lib-modules: Type: HostPath (bare host directory volume) Path: /lib/modules HostPathType:
var-run-calico: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/run/calico HostPathType:
var-lib-calico: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/lib/calico HostPathType:
xtables-lock: Type: HostPath (bare host directory volume) Path: /run/xtables.lock HostPathType: FileOrCreate sys-fs: Type: HostPath (bare host directory volume) Path: /sys/fs/ HostPathType: DirectoryOrCreate cni-bin-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/opt/cni/bin HostPathType:
cni-net-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/args/cni-network HostPathType:
cni-log-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/common/var/log/calico/cni HostPathType:
host-local-net-dir: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/lib/cni/networks HostPathType:
policysync: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/current/var/run/nodeagent HostPathType: DirectoryOrCreate kube-api-access-wlsfs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: :NoSchedule op=Exists :NoExecute op=Exists CriticalAddonsOnly op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message

Normal Pulled 15s (x8 over 14m) kubelet Container image "docker.io/calico/cni:v3.25.1" already present on machine

neoaggelos commented 11 months ago

Hi @AllardKrings

Can you share an inspection report from the cluster? Looks like the Calico CNI fails to initialize for some reason

AllardKrings commented 11 months ago

Thanks for your quick reply,

here is the report:

inspection-report-20231114_113118.tar.gz

neoaggelos commented 11 months ago

Thanks, I see the following line in Kubelet logs, which causes the services to continuously restart

Nov 12 13:46:23 odroid microk8s.daemon-kubelite[359263]: E1112 13:46:23.145422  359263 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : open /sys/fs/cgroup/kubepods/besteffort/cpu.weight: no such file or directory"

This typically means that cgroups are not configured properly on the system.

As a workaround, can you try appending the following kubelet arguments:

echo '
--cgroups-per-qos=false
--enforce-node-allocatable=""
' | sudo tee -a /var/snap/microk8s/current/args/kubelet

sudo snap restart microk8s.daemon-kubelite
AllardKrings commented 11 months ago

Hello,

I tried this but to no avail.

The sbc refuses to boot now.

I wil flash a fresh image tonight and install microk8s again.

I find it strange that a fresh ubuntu and a fresh microk8s do not work together.

Any suggestions on another linux-flavour?

Kind regards.

Op di 14 nov 2023 om 12:52 schreef Angelos Kolaitis < @.***>

Thanks, I see the following line in Kubelet logs, which causes the services to continuously restart

Nov 12 13:46:23 odroid microk8s.daemon-kubelite[359263]: E1112 13:46:23.145422 359263 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : open /sys/fs/cgroup/kubepods/besteffort/cpu.weight: no such file or directory"

This typically means that cgroups are not configured properly on the system.

As a workaround, can you try appending the following kubelet arguments:

echo ' --cgroups-per-qos=false --enforce-node-allocatable="" ' | sudo tee -a /var/snap/microk8s/current/args/kubelet

sudo snap restart microk8s.daemon-kubelite

— Reply to this email directly, view it on GitHub https://github.com/canonical/microk8s/issues/4301#issuecomment-1810061954, or unsubscribe https://github.com/notifications/unsubscribe-auth/AET5SGPDPTDKXQI63NRCRT3YENLPTAVCNFSM6AAAAAA7KSQOJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGA3DCOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>

stale[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.