search

clearlinux / cloud-native-setup

Automation around setting up the cloud-native content (kubernetes) on Clear Linux.
Apache License 2.0
61 stars 41 forks source link

Trying to setup k8s with cri-o and kata-fc runtime on bare metal without Vagrant #303

Closed Zhuul closed 3 years ago

Zhuul commented 4 years ago

Hello, I'm trying to setup a fresh k8s install using cri-o and kata firecracker on bare metal not using Vagrant or anthing else beside "Clearlinux" and https://github.com/clearlinux/cloud-native-setup .

When using "kata-fc runtime" and "devicemapper" along with cri-o (default/overlay2 crio-o setup seems ok) I get this error message from cri-o :

FATA[2020-01-31 19:29:45.041517695Z] devicemapper: Non existing device storage-thinpool

I'm using devicemapper service like :

/sbin/losetup /dev/loop8 /var/lib/crio/devicemapper/disk.img

and my crio.conf file looks like : … storage_driver = "devicemapper" storage_option = [ "dm.basesize=8G", "dm.directlvm_device=/dev/loop8", "dm.directlvm_device_force=true", "dm.override_udev_sync_check=true", "dm.fs=ext4" ] … Should I manualy create a thin pool partition using "dmsetup", "lvcreate", "vgcreate" like explained at https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt OR craft an appropriate /etc/containers/storage.conf file which I'm currently missing ?

I am not sure what to do next since thin-pool partition setup/creation seem to be missing for me... because I'm not using Vagrant with cloud-native-setup ? I don't know.

Anyone with same setup faced this situation already ? Am I missing something ?

Can I just get rid of "devicemapper" storage driver and use "overlay2" instead ?

grahamwhaley commented 4 years ago

/cc @ganeshmaharaj @devimc for any thoughts.

Zhuul commented 4 years ago

Well I changed my mind and I'm going to retry cloud-native-setup using containerd (with Docker along and still without Vagrant) once I solved some "weird" network issue with my Clearlinux host and docker0 ipv6 interface.

Actually I'm looking to make an "ipv6 only" kubernetes installation using cloud-native-setup script (if possible…).

Also containerd / docker could make more sense to me since I'm looking later to deploy Nvidia drivers to use GPU with the cloud and it seems to be easier to do using containerd.

Regarding the install on Clearlinux with containerd... I wanted to know what cgroup-driver should I use for kubelet and other components ? "systemd" or cgroupfs" ? I'm not sure.

Also docker info seems to reconise kata-runtime (but can't reconise version of containerd or runc on my box) and I can tell docker to use "runc" as default but is there a way to mask totally "kata-runtime" from docker ?

Sorry if those questions are off topic… I can close this ticket and reopen another one maybe.

grahamwhaley commented 4 years ago

For the kata-runtime docker bit at least - kata is added as an extra runtime to Docker. If your docker info shows your default runtime as runc, then it should already be benign unless you request to use kata with the --runtime etc.

However, if you do wish to remove it completely from docker, then on my system at least (caveat: which is a bit of a kata dev system, so may not entirely match yours...) the runtime is added in the file /usr/lib/systemd/system/docker.service.d/clearlinux.conf. You can probably edit that file (although, I don't know if that is the correct method for the ClearLinux stateless methodology.

Or, you could remove the kata bundle. a swupd search kata should show you have containers-virt installed, and removing that (swupd bundle-remove containers-virt), should remove kata from the system, and ideally leave you with runc as the default docker runtime.