search

corneliusweig / kubernetes-lxd

A step-by-step guide to get kubernetes running inside an LXC container
295 stars 52 forks source link

Add instructions for lxc on zfs #2

Closed martinhoefling closed 5 years ago

martinhoefling commented 5 years ago

I hope so (/etc/pve/lxc/106.conf) You can probably also configure it via the UI, but I did it directly in the proxmox-LXC config. If you run on bare LXC, this should work as well if you’re zfs backed by directly adding to the lxc config.

Am 05.11.2018 um 22:39 schrieb corneliusweig notifications@github.com:

@corneliusweig commented on this pull request.

In README.md:

@@ -55,6 +55,22 @@ To use it, install lxd and initialize it using lxd init. When prompted, answer

  1. Restart your lxc container. Unfortunately, lxc stop k8s-lxc does not work for me. I need to do lxc exec k8s-lxc reboot.

+### Using docker and kubernetes on zfs backed host systems + +If your host system is backed by ZFS storage (e.g. an option for Proxmox), some adaption need to be made. ZFS currently lacks full namespace support an thus a dataset cannot be reached into a LXC container retaining full control over the child datasets. The easiest solution is to create two volumes for /var/lib/docker and /var/lib/kubelet and format these ext4. +

  • zfs create -V 50G mypool/my-dockervol
  • zfs create -V 5G mypool/my-kubeletvol
  • mkfs.ext4 /dev/zvol/mypool/my-dockervol
  • mkfs.ext4 /dev/zvol/mypool/my-kubeletvol

  • +One then just needs to reach in the two volumes at the right location. The configuration for Proxmox looks like this: I'm no expert with Proxmox, but is it absolutely clear where those settings need to be configured?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

laymonk commented 3 years ago

I hope so (/etc/pve/lxc/106.conf) You can probably also configure it via the UI, but I did it directly in the proxmox-LXC config. If you run on bare LXC, this should work as well if you’re zfs backed by directly adding to the lxc config.

Hi, I am trying to figure out how to use the suggestion below outside Proxmox (in bare LXC) ... 

mp0: /dev/zvol/mypool/my-dockervol,mp=/var/lib/docker,backup=0
mp1: /dev/zvol/mypool/my-kubeletvol,mp=/var/lib/kubelet,backup=0

Is the suggestion to deploy the k8s container root on one of these ext4 filesystems ? If so, why 2 filesystems?

Or that we should mount them into the container at: /var/lib/{docker,kubelet} ?

If so, why not just deploy the entire container on one of these volumes ?

I suspect the intention is to create the k8s containers entirely on non ZFS backing store. If that's the intention, the doc probably needs changing to make it clearer.

And if this assumption is correct, then one could just format one of the volumes with btrfs (or even lvm), create an lxd storage volume and use that for k8s containers (via an lxd profile). I can provide the instructions to do this (it's actually what I have ended up doing, because I just couldn't get the logic of what was being suggested)

corneliusweig commented 3 years ago

I haven't done the proxmox setup myself, so I don't know the answer here. Maybe @martinhoefling knows?

Also I'm no longer actively maintaining these instructions because I have a different setup now. But if you want to make clarifications, PRs are welcome!

martinhoefling commented 3 years ago

Yes, those are separate volumes mounted in /var/lib/docker and /var/lib/kubelet

As far as I recall, reasons were as follows:

Deploying the entire container would make it more difficult to snapshot the setup (I use my standard ZFS scripts for that) and to change size of e.g. the docker volume and I am not sure if I can map further volumes from ZFS into the running system.

laymonk commented 3 years ago

Thanks for the responses, @martinhoefling and @corneliusweig

I think this repo is most useful, and covers quite a bit of the slippery slope of getting lxd working on k8s. Thanks to both of you. I had to jump through considerable hoops to get k8s working well on lxd.

The storage config was also a challenge ... I used btrfs in the end (because just using directory mode didn't appeal to me) ... but I can provide instructions for using either mode.

I can also provide instructions for either installing the entire container on the non-zfs storage pool or mounting only the extra filesystem in the container.

Mind you, lxd provides good snapshot mechanisms that can also be scripted ... and so does btrfs. Personally, I prefer using the container's own snapshot system than the underlying filesystem's ... I also think it's less complicated to locate the entire container in the non-zfs storage, but I guess this is down to personal taste.

corneliusweig commented 3 years ago

@laymonk I'm glad this guide was helpful. My only worry is that it will become outdated, because nobody is maintaining it anymore.

If your storage configuration is of general interest, I'm happy to include it here. However, I want to avoid a recipe with too many optional pieces, because that can easily make it confusing. So if you add an optional section, please clearly mark it so, or move it to a separate file and link to it from the main one.