Dunge
commented
1 year ago Small follow up,
I did a microk8s stop on all 3 nodes because they were just crash looping wit errors all the time. A few hours later I came back and tried to pop just the first node back up, and it doesn't come up.
$ microk8s status
microk8s is not running, try microk8s start
$ microk8s start
********************************** about 2-3 minutes before it respond **********************************
$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
$ microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
********************************** about 5 minutes before it respond **********************************
Inspecting dqlite
Inspect dqlite
Building the report tarball
Report tarball is at /var/snap/microk8s/4595/inspection-report-20230321_033014.tar.gz
$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
$ microk8s kubectl get nodes
E0321 03:32:43.569188 861346 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout
E0321 03:33:03.576104 861346 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout - error from a previous attempt: read tcp 127.0.0.1:47630->127.0.0.1:16443: read: connection reset by peer
E0321 03:33:24.569670 861346 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout - error from a previous attempt: read tcp 127.0.0.1:50388->127.0.0.1:16443: read: connection reset by peer
E0321 03:33:34.571132 861346 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout
E0321 03:33:45.981150 861346 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout - error from a previous attempt: read tcp 127.0.0.1:54130->127.0.0.1:16443: read: connection reset by peer
error: Get "https://127.0.0.1:16443/api?timeout=32s": net/http: TLS handshake timeout - error from a previous attempt: read tcp 127.0.0.1:54130->127.0.0.1:16443: read: connection reset by peerI uploaded the new (much smaller) tarball here: https://drive.google.com/file/d/1AEA23MpRMeCeHkfFlh7t_jMOKLXkj4Mf/view?usp=sharing
Might be easier to diagnose (EDIT: Do not analyze this one, this was when a single node was active, and do not show the same issue)
I see a lot of this error in the kubelite log, not sure what it could mean:
microk8s.daemon-kubelite[854750]: W0321 03:24:11.013071 854750 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
microk8s.daemon-kubelite[854750]: "Addr": "unix:///var/snap/microk8s/4595/var/kubernetes/backend/kine.sock:12379",
microk8s.daemon-kubelite[854750]: "ServerName": "kine.sock",
microk8s.daemon-kubelite[854750]: "Attributes": null,
microk8s.daemon-kubelite[854750]: "BalancerAttributes": null,
microk8s.daemon-kubelite[854750]: "Type": 0,
microk8s.daemon-kubelite[854750]: "Metadata": null
microk8s.daemon-kubelite[854750]: }. Err: connection error: desc = "transport: Error while dialing dial unix /var/snap/microk8s/4595/var/kubernetes/backend/kine.sock:12379: connect: connection refused"
balchua
alexgorbatchev
Summary
Ubuntu 20.04.5 LTS microk8s 1.26.1 installed via snap 3 nodes HA clusters
I came back one morning and my cluster that was working fine the previous day was completely down. Unlike my previous similar experience, this time the disks still have space remaining.
At this point
microk8s inspectwould just freeze at theGathering system informationstep after printing that the services were running.I then launched a
microk8s start, nothing for a few minutes, then exit with no message. status still say not running.I tried to check the logs using
sudo journalctl -u snap.microk8s.daemon-kubelitebut there’s too much stuff, couldn’t find anything relevant.I then rebooted the nodes
sudo rebootand they actually came back online status Ready.The condition look good:
But there is a Taint preventing any workload to get scheduled on them:
Events on pods:
Events seems to crash the kubelet service on loop:
The "Invalid Disk Capacity" part seems to be normal, because the pods handling the disk CSI (Longhorn) are not running because of the node taint.
Any command I send to microk8s kubectl print some error message about memcache.go:
Any idea what is causing this? From my point of view, nodes seems healthy, ram, cpu and disk are good, networking between them is also functional.
Introspection Report
Here is my microk8s inspect tarball file hosted on my Google Drive: https://drive.google.com/file/d/1LEaCI3O2Y82OI4Q2quo-uIBD-zV8rZBA/view?usp=sharing