posts/minikube-vs-kind-vs-k3s

[utterances-bot]

Minikube vs. kind vs. k3s - What should I use?

A little comparison of local Kubernetes tools like Minikube, kind and k3s.

https://brennerm.github.io/posts/minikube-vs-kind-vs-k3s.html

[rampatra]

Nice & brief article 👍

[rampatra]

However, I have one query, as a greenhorn, and that is, can't I use Docker containers in minikube?

On the getting started page (https://minikube.sigs.k8s.io/docs/start/), I see that it supports Docker along with VMs.

[brennerm]

@rampatra We need to differentiate between the minikube driver and the container runtime here. The former defines which environment minikube uses to run the Kubernetes components. That's what the start page is referring to and what you need to have installed on your host system, e.g. Docker or Virtualbox. The latter is referring to the platform that'll execute your containers. Currently minikube supports Docker, cri-o and containerd.

As all of these three support the OCI runtime and image specification they're intercompatible. That means you can execute your locally built Docker images on any minikube instance out there.

Check out the help page of minikube start minikube start --help to see all the available options for the --driver and --container-runtime parameters.

Hope that helps!

[rampatra]

Thanks, makes sense. I think I got confused with the minikube driver and container runtime.

[morph027]

Just discovered https://github.com/rancher/k3d/ which creates k3s in containers so fast i can't even walk to the coffee machine for the waiting coffee :)

[brennerm]

That's exactly the problem. Where are my good old servers that I could start to reinstall and go for lunch in the meantime. Those modern systems are just that more demanding. :wink:

[mecampbellsoup]

@brennerm @rampatra for someone a little bit thick here...

Can you elaborate more on the driver vs. container runtime distinction?

It seems to me like KIND is more or less equivalent to minikube when I configure minikube to use Docker driver and Docker container runtime, no? (Really any container runtime that works is fine, I suppose I'm more focusing on the driver here, i.e. when minikube is not using a full-fledged VM driver but rather Docker.)

[mecampbellsoup]

More context here: https://kubernetes.slack.com/archives/CEKK1KTN2/p1610634695007700

[brennerm]

@mecampbellsoup From the driver perspective I agree, kind is similar to a minikube instance that uses the Docker driver.

When it comes to the container runtime, kind only supports cri-o and containerd.

Essentially it's about differentiating between where the Kubernetes components are running (Docker or VM) and what's the runtime (Docker, containerd or cri-o) that takes care of the containers Kubernetes is managing for you.

[behrangsa]

Can’t Minikube work completely using Docker without any need for VMs?

[brennerm]

@behrangsa Yup, you're right. Wasn't aware of that back then. :)

[suchakra012]

Nice & informative !! On a note, can we use the same tools for running Kubernetes on podman instead on docker in the usual way ?

[brennerm]

@suchakra012 Thanks!

AFAIK minkube and kind can run on podman.

From this issue it seems like k3s' support is currently pending.

[jwuman]

Minikube now supports multi nodes https://minikube.sigs.k8s.io/docs/tutorials/multi_node/

[h3ct0rjs]

Thank you very much.

I've chosen kind is the best option for me, I've used minikube and it take so a long time. Regards, H

[thiscantbeserious]

Running Kubernetes within Docker might sound cool and more efficient without the overhead of a VM until you realize that kind is running a k8s distro single node that can eat away all your resources in a flash, even when running on an octacore - same likely goes for Minikube on Docker.

You basically have to run it in a VM to tune down its resource hunger - which makes compiling in the same VM also slower. So you'll likely want to run two VMs.

k3s is miles ahead when it comes to minimal resource-usage and actually it comes with uninstall & kill-all scripts that you can simply run endlessly (I have done that like a million times). I pair it with k3sup for easy deployment and clustering from my machine.

Even when you run it in a lightweight and thin VM the other two arent even going to come close to how resource-friendly it is. Not even saying that there's also LXC and KVM which are a lot more near hardware as well (havent tried either with it, just saying).

However there is a much better option: Throw old hardware or a few SBCs on a 10$ switch and get to experience a real cluster without solving the troubles of multi-nodes on a single host (including performance bottlenecks).

Yes kinds local development is cool - resources wise it simply sucks from the deployment package itself (this is where rancher put a lot more work into it, and also where it matters).

[h3ct0rjs]

@thiscantbeserious what SBC do you recommend any useful tutorial?

[thiscantbeserious]

Actually following the documentation here is sufficient:

https://github.com/alexellis/k3sup

SBC wise you can add anything - I personally go with ARM64 only now (for the sake of compability) but I also have a few old Odroid HC1/2 (very cheap, Octacore only 2GB Ram tought and ArmHF - makes for a good backup or work offload target tought e.g. DB host for replication - I wouldn't recommend them anymore since software support is quite limited out of the box and Golang runs buggy on ARM32+).

Generally I can recommend anything from Odroid and FriendlyElec - x86-wise I'm personally using a few Odroid H2 (sold out) but even RaspberryPi 4's with 8GB aren't to bad.

An x86 replacement for the Odroid H2 could be the AsRock DeskMini A300 or quite more powerfull the AsRock 4x4 (maybe even to much).

Actually if you want to focus on the best compability and not be bottlenecked by storage IO I'd look for Boards with native PCIe like the RK3399 powered RockPi 4 and NanoPi M4 or likely the best option in terms of storage and cost the Odroid HC4 with 2x SATA Ports over PCIe and 4GB Ram (if you can get it for cheap).

The NanoPi M4 and RockPi 4 can both host an M2 PCIe SSD with their shields, so if space is an concern that's also a good option.

TLDR: Odroid HC4, RockPi 4 4GB, NanoPi M4 4GB are good options for ARM64 and generally price-performance-ratio.

Software-Support from the Vendors mentioned:

RaspberryPi (official Ubuntu Image) > Radaxa (RockPi, lots of different Distros, even OPNSense/FreeBSD) > Hardkernel (Odroid, their Ubuntu 20.04 images) > FriendlyElec (NanoPi, needs to use Ubuntu for ARM64).

If you can, avoid Armbian - their images and Kernels are quite buggy despite being flagged as stable in many cases/combinations nowadays.

I'd personally stick to Ubuntu for the sake of maintainability.

[lesterpotter]

Docker desktop allow you to enable Kubernetes. Could you explain how that K8s relates to the others you describe? What K8s is actually being used when the Docker Kubernetes feature is enabled in preferences? If different from the 3 you describe, then how does it compare? Thanks.

[brennerm]

Hey @lesterpotter, As far as I can tell Docker for Desktop uses kubeadm to setup a single node Kubernetes cluster using Docker containers. So it's pretty similar to kind. More info: https://birthday.play-with-docker.com/kubernetes-docker-desktop/

[h3ct0rjs]

@thiscantbeserious thank you

Héctor F, Regards, Colombia