minikube start --ha --driver=docker --container-runtime=containerd --profile ha-demo failed to start

FrankYu-USA commented 1 week ago

What Happened?

$ minikube start --ha --driver=docker --container-runtime=containerd --profile ha-demo 😄 [ha-demo] minikube v1.33.1 on Ubuntu 22.04 ✨ Using the docker driver based on user configuration 📌 Using Docker driver with root privileges 👍 Starting "ha-demo" primary control-plane node in "ha-demo" cluster 🚜 Pulling base image v0.0.44 ... 🔥 Creating docker container (CPUs=2, Memory=901MB) ... 📦 Preparing Kubernetes v1.30.0 on containerd 1.6.31 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring CNI (Container Networking Interface) ... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass

👍 Starting "ha-demo-m02" control-plane node in "ha-demo" cluster 🚜 Pulling base image v0.0.44 ... 🔥 Creating docker container (CPUs=2, Memory=901MB) ... 🌐 Found network options: ▪ NO_PROXY=192.168.58.2 📦 Preparing Kubernetes v1.30.0 on containerd 1.6.31 ... ▪ env NO_PROXY=192.168.58.2 🔎 Verifying Kubernetes components...

👍 Starting "ha-demo-m03" control-plane node in "ha-demo" cluster 🚜 Pulling base image v0.0.44 ... 🔥 Creating docker container (CPUs=2, Memory=901MB) ... ✋ Stopping node "ha-demo-m03" ... 🔥 Deleting "ha-demo-m03" in docker ... 🤦 StartHost failed, but will try again: creating host: create: provisioning: get ssh host-port: unable to inspect a not running container to get SSH port 🔥 Creating docker container (CPUs=2, Memory=901MB) ... 😿 Failed to start docker container. Running "minikube delete -p ha-demo" may fix it: creating host: create: provisioning: get ssh host-port: unable to inspect a not running container to get SSH port

❌ Exiting due to GUEST_START: failed to start node: adding node: Failed to start host: creating host: create: provisioning: get ssh host-port: unable to inspect a not running container to get SSH port

╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯

Attach the log file

logs.txt

Operating System

Ubuntu

Driver

Docker

medyagh commented 1 week ago

I am curious what does minikube profile list show?

@prezha I am guessing this is a flake since it passes on our integraiton tests?

FrankYu-USA commented 1 week ago

frankyulinux@frankyukubetest:~$ minikube profile list	----------	-----------	------------	----------------	------	---------	----------	-------	----------------
ha-demo	docker	containerd	192.168.58.254	8443	v1.30.0	Degraded	3		*
minikube	docker	docker	192.168.49.2	8443	v1.30.0	Running	1	*
----------	-----------	------------	----------------	------	---------	----------	-------	----------------	--------------------

prezha commented 1 week ago

@medyagh yes, i think this might be something going on with this specific host/setup/resources - based on the outputs and the log.txt:

cluster state is "degraded", which means it has two out of three control plane nodes up and running
3rd node was created but timed out while trying to ssh into it, this failed in the "retry" attempts as well
cpu/dmesg: there are plenty of "workqueue: <several components/drivers> hogged CPU for >10000us times, consider switching to WQ_UNBOUND", indicating some kind of performance issues or hitting a bug elsewhere?
memory: minikube config setting mandate only 901MB of memory to use for a node, this is way less than the recommended minimum of 2GB, so the behavior might be unexpected

FrankYu-USA commented 1 week ago

please close the ticket, after increase the memory size, issue disappeared:

minikube start --ha --driver=docker --memory=2200mb --container-runtime=containerd --profile ha-demo

prezha commented 1 week ago

excellent, thanks for checking and confirming @FrankYu-USA !

kubernetes / minikube