search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.4k stars 767 forks source link

microk8s failing to start, all pods previously were stuck in pending #1830

Closed aliwatters closed 3 years ago

aliwatters commented 3 years ago

Hi, having weird issues with trying to start micrk8s. I'm currently learning kubernetes so following along with the tutorial at https://kubernetes.io/docs/tutorials/kubernetes-basics/ -- initially things worked in microk8s as well as minikube.

But after a while pods refused to start, and were listed as pending with a kubectl get pods, now the microk8s start command fails (with a "started" message)

ali@stinky:~$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
ali@stinky:~$ microk8s start
Started.
ali@stinky:~$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
ali@stinky:~$ microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-control-plane-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/1856/inspection-report-20201217_195024.tar.gz

inspection-report-20201217_195024.tar.gz

$  journalctl -b -u snap.microk8s.daemon-kubelet
Dec 17 19:35:32 stinky systemd[1]: Started Service for snap application microk8s.daemon-kubelet.
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + export PATH=/snap/microk8s/1856/usr/sbin:/snap/microk8s/1856/usr/bin:/snap/microk8s/1856/sbin:/snap/microk8s/1856/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + PATH=/snap/microk8s/1856/usr/sbin:/snap/microk8s/1856/usr/bin:/snap/microk8s/1856/sbin:/snap/microk8s/1856/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2200]: ++ /snap/microk8s/1856/bin/uname -m
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + ARCH=x86_64
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + export LD_LIBRARY_PATH=:/snap/microk8s/1856/lib:/snap/microk8s/1856/usr/lib:/snap/microk8s/1856/lib/x86_64-linux-gnu:/snap/microk8s/1856/usr/lib/x86_64-linux-gnu
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + LD_LIBRARY_PATH=:/snap/microk8s/1856/lib:/snap/microk8s/1856/usr/lib:/snap/microk8s/1856/lib/x86_64-linux-gnu:/snap/microk8s/1856/usr/lib/x86_64-linux-gnu
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1856/lib:/snap/microk8s/1856/usr/lib:/snap/microk8s/1856/lib/x86_64-linux-gnu:/snap/microk8s/1856/usr/lib/x86_64-linux-gnu
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1856/lib:/snap/microk8s/1856/usr/lib:/snap/microk8s/1856/lib/x86_64-linux-gnu:/snap/microk8s/1856/usr/lib/x86_64-linux-gnu
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + source /snap/microk8s/1856/actions/common/utils.sh
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + app=kubelet
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + '[' -e /var/snap/microk8s/1856/var/lock/clustered.lock ']'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + '[' kubelet = kubelet ']'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2208]: ++ grep pod-cidr
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2210]: ++ gawk '{print $2}'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2207]: ++ cat /var/snap/microk8s/1856/args/kubelet
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2209]: ++ tr = ' '
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + pod_cidr=
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + '[' -z '' ']'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2213]: ++ tr -d '\"'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2212]: ++ jq .Network /var/snap/microk8s/1856/args/flannel-network-mgr-config
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + pod_cidr=10.1.0.0/16
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + '[' -z 10.1.0.0/16 ']'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + iptables -C FORWARD -s 10.1.0.0/16 -m comment --comment 'generated for MicroK8s pods' -j ACCEPT
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + grep -e '--address ' /var/snap/microk8s/1856/args/containerd
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2221]: ++ grep -e '--address ' /var/snap/microk8s/1856/args/containerd
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2222]: ++ gawk '{print $2}'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + socket='${SNAP_COMMON}/run/containerd.sock'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2223]: ++ eval echo '${SNAP_COMMON}/run/containerd.sock'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2223]: +++ echo /var/snap/microk8s/common/run/containerd.sock
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + socket_file_expand=/var/snap/microk8s/common/run/containerd.sock
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + n=0
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + '[' 0 -ge 10 ']'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + test -S /var/snap/microk8s/common/run/containerd.sock
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + echo 'Waiting for containerd socket /var/snap/microk8s/common/run/containerd.sock to appear. (attempt 0)'
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: Waiting for containerd socket /var/snap/microk8s/common/run/containerd.sock to appear. (attempt 0)
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + n=1
Dec 17 19:35:32 stinky microk8s.daemon-kubelet[2177]: + sleep 2
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + '[' 1 -ge 10 ']'
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + test -S /var/snap/microk8s/common/run/containerd.sock
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + break
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + '[' -e /proc/2177/cgroup ']'
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2266]: ++ gawk -F '[:]' '(/cpu/ && !/cpuset/) || /memory/ {print $3}' /proc/2177/cgroup
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2267]: ++ uniq
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2268]: ++ wc -l
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + [[ 1 -eq 2 ]]
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2269]: ++ cat /var/snap/microk8s/1856/args/kubelet
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + declare -a 'args=(--kubeconfig=${SNAP_DATA}/credentials/kubelet.config
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --cert-dir=${SNAP_DATA}/certs
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --client-ca-file=${SNAP_DATA}/certs/ca.crt
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --anonymous-auth=false
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --network-plugin=cni
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --root-dir=${SNAP_COMMON}/var/lib/kubelet
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --fail-swap-on=false
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --cni-conf-dir=${SNAP_DATA}/args/cni-network/
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --cni-bin-dir=${SNAP_DATA}/opt/cni/bin/
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --feature-gates=DevicePlugins=true
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --container-runtime=remote
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --containerd=${SNAP_COMMON}/run/containerd.sock
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --node-labels="microk8s.io/cluster=true"
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --authentication-token-webhook=true
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --cluster-domain=cluster.local
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: --cluster-dns=10.152.183.10)'
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + '[' -f /snap/microk8s/1856/kubelet ']'
Dec 17 19:35:34 stinky microk8s.daemon-kubelet[2177]: + exec /snap/microk8s/1856/kubelet --kubeconfig=/var/snap/microk8s/1856/credentials/kubelet.config --cert-dir=/var/snap/microk8s/1856/certs --client-ca-file=/var/snap/microk8s/1856/certs/ca.crt --anonymous-auth=false --network-plugin=cni --root-dir=/var/snap/microk8s/common/var/lib/kubelet --fail-swap-on=false --cni-conf-dir=/var/snap/microk8s/1856/args/cni-network/ --cni-bin-dir=/var/snap/microk8s/1856/opt/cni/bin/ --feature-gates=DevicePlugins=true '--eviction-hard=memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi' --container-runtime=remote --container-runtime-endpoint=/var/snap/microk8s/common/run/containerd.sock --containerd=/var/snap/microk8s/common/run/containerd.sock --node-labels=microk8s.io/cluster=true --authentication-token-webhook=true --cluster-domain=cluster.local --cluster-dns=10.152.183.10
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.745653    2177 server.go:411] Version: v1.19.5-34+8af48932a5ef06
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.785178    2177 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/snap/microk8s/1856/certs/ca.crt
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.845615    2177 server.go:640] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.846731    2177 container_manager_linux.go:276] container manager verified user specified cgroup-root exists: []
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.846757    2177 container_manager_linux.go:281] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/snap/microk8s/common/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:1Gi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:1Gi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.849306    2177 topology_manager.go:126] [topologymanager] Creating topology manager with none policy
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.849327    2177 container_manager_linux.go:311] [topologymanager] Initializing Topology Manager with none policy
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.849332    2177 container_manager_linux.go:316] Creating device plugin manager: true
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: W1217 19:35:35.849530    2177 util_unix.go:103] Using "/var/snap/microk8s/common/run/containerd.sock" as endpoint is deprecated, please consider using full url format "unix:///var/snap/microk8s/common/run/containerd.sock".
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.849568    2177 remote_runtime.go:59] parsed scheme: ""
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.849578    2177 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854251    2177 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/snap/microk8s/common/run/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854277    2177 clientconn.go:948] ClientConn switching balancer to "pick_first"
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: W1217 19:35:35.854320    2177 util_unix.go:103] Using "/var/snap/microk8s/common/run/containerd.sock" as endpoint is deprecated, please consider using full url format "unix:///var/snap/microk8s/common/run/containerd.sock".
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854337    2177 remote_image.go:50] parsed scheme: ""
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854342    2177 remote_image.go:50] scheme "" not registered, fallback to default scheme
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854359    2177 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/snap/microk8s/common/run/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854365    2177 clientconn.go:948] ClientConn switching balancer to "pick_first"
Dec 17 19:35:35 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:35.854396    2177 kubelet.go:273] Watching apiserver
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: E1217 19:35:42.014726    2177 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.033855    2177 kuberuntime_manager.go:214] Container runtime containerd initialized, version: v1.3.7, apiVersion: v1alpha2
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.053886    2177 server.go:1147] Started kubelet
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.053939    2177 server.go:152] Starting to listen on 0.0.0.0:10250
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: E1217 19:35:42.064128    2177 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: E1217 19:35:42.064410    2177 kubelet.go:1218] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.064282    2177 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.065286    2177 server.go:424] Adding debug handlers to kubelet server.
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.068024    2177 volume_manager.go:265] Starting Kubelet Volume Manager
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.070536    2177 desired_state_of_world_populator.go:139] Desired state populator starts to run
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.092197    2177 status_manager.go:158] Starting to sync pod status with apiserver
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.092218    2177 kubelet.go:1741] Starting kubelet main sync loop.
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: E1217 19:35:42.092251    2177 kubelet.go:1765] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.128678    2177 client.go:87] parsed scheme: "unix"
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.128714    2177 client.go:87] scheme "unix" not registered, fallback to default scheme
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.128748    2177 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///var/snap/microk8s/common/run/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.128762    2177 clientconn.go:948] ClientConn switching balancer to "pick_first"
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.184964    2177 kubelet_node_status.go:71] Attempting to register node stinky
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: E1217 19:35:42.192496    2177 kubelet.go:1765] skipping pod synchronization - container runtime status check may not have completed yet
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193165    2177 cpu_manager.go:184] [cpumanager] starting with none policy
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193177    2177 cpu_manager.go:185] [cpumanager] reconciling every 10s
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193195    2177 state_mem.go:36] [cpumanager] initializing new in-memory state store
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193459    2177 state_mem.go:88] [cpumanager] updated default cpuset: ""
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193468    2177 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]"
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.193476    2177 policy_none.go:43] [cpumanager] none policy: Start
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.198923    2177 kubelet_node_status.go:109] Node stinky was previously registered
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.199549    2177 kubelet_node_status.go:74] Successfully registered node stinky
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.215353    2177 setters.go:555] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2020-12-17 19:35:42.21532954 -0800 PST m=+7.612461544 LastTransitionTime:2020-12-17 19:35:42.21532954 -0800 PST m=+7.612461544 Reason:KubeletNotReady Message:container runtime status check may not have completed yet}
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.233345    2177 plugin_manager.go:114] Starting Kubelet Plugin Manager
Dec 17 19:35:42 stinky microk8s.daemon-kubelet[2177]: I1217 19:35:42.476030    2177 reconciler.go:157] Reconciler: start to sync state

I've run microk8s reset, rebooted my system, checked I'm on ext4 (not zfs) -- what should else should I try?

balchua commented 3 years ago

Hi,

The inspect shows that there is nothing installed. It looks clean for now. So clean that even the calico CNI are not there. That can explain why status says its not running.

Can you do a sudo snap remove microk8s --purge and then reinstall it?

Sorry about the trouble.

aliwatters commented 3 years ago

Thank you @balchua -- I was holding off on that in case there was something obvious in the logs.

Went through the snap reinstall and all is working now.

$ microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    metrics-server       # K8s Metrics Server for API access to service metrics
...