Swap to Fedora Server

[onedr0p]

Details

Describe the solution you'd like:

I've recently changed from supporting Ubuntu Server to Fedora Server. The reasons behind this are that Fedora moves a faster speed than Ubuntu, which can be viewed as a good thing or bad. That is up to you to decide if you want to make the switch.

As of writing:

• Ubuntu Server Kernel: 5.15
• Fedora Server Kernel: 5.18

I am no fanboy of either of these OSes and will not die on a rock tell you one is better than the other. However, Fedora seems to fit a bit nicer with more important default packages installed and is a a lot easier to manage in the Ansible prepare playbook.

I am curious how Raspberry Pis fit into this, if anyone wants to test.

[mrueg]

Would it be possible to support Ubuntu still at the same time? I've got a slight preference for an apt/deb based environment.

[onedr0p]

I'm not against it, but supporting one platform is much easier than supporting two.

I was always an apt/deb person myself but in this case, for me, I rarely ever SSH into my nodes. They are meant to run k3s and at this level it's only really a difference between apt install vs. dnf install or apt upgrade and dnf upgrade. There's also nmcli vs. netplan but I feel like all these are very minor nuances between the OSs.

I'm curious to hear others take on this too, Fedora has had a lot of flak in the past but it seems to be a formidable alternative to Ubuntu these days as they are pushing the boundaries more on kernel and package support.

Maybe I've been drinking too much of Linus Torvalds Kool aid 🤣

[e3b0c442]

Not saying this isn't a good change, but:

more important default packages installed

doesn't jive with

I rarely ever SSH into my nodes

Honestly, on my k3s nodes I want as little as humanly possible running on them aside from k3s. Every additional service is more memory and/or potential attack surface.

It's also worth noting that a lot of people running a cluster at home like this are running on Raspberry Pi, where there are a few more hurdles to getting Fedora set up compared to Raspberry Pi OS or Ubuntu.

Like you said, at this level it comes down to the package managers and I do think it's worth supporting both.

[onedr0p]

doesn't jive with

Kind of? It means I don't have to SSH into my server to install packages or using Ansible to install certain packages.

Honestly, on my k3s nodes I want as little as humanly possible running on them aside from k3s. Every additional service is more memory and/or potential attack surface.

I 100% agree but using Ubuntu or Fedora Server will have needless packages either way, to truly trim down the bloat you'll want to look into using a OS designed to run containers like Talos.

It's also worth noting that a lot of people running a cluster at home like this are running on Raspberry Pi, where there are a few more hurdles to getting Fedora set up compared to Raspberry Pi OS or Ubuntu.

This is something I considered after the fact, but there is a aarch64 ISO for Fedora Server so I am curious how well that works with the standard config.

Like you said, at this level it comes down to the package managers and I do think it's worth supporting both.

I am open to PRs that can support both, the level of Ansible logic and testing is much for me to do currently.

[willianpaixao]

My two cents is that the support of a given Linux distro is not a toggle, but a slide. For the playbooks to run in let's say RedHat doesn't necessarily needs to fail in Debian. Here are a few solutions:

1. Using ansible.builtin.package which is OS agnostic, as long as the package has the same name across package managers.
2. For tasks beyond package installation, you can fetch the target information during setup time and filter, selecting the right task for the right distro. I've added an example in #396. You can find detailed information and extensive examples in the Ansible docuentation. [1] [2]
3. We can even add tags to specific blocks of tasks to add the possibility of filtering out tasks that would otherwise fail in a different distro. I also added an example in the above mentioned PR. [3]
4. Lastly we could even mix different distros in the same (let's say you have Fedora AND Ubuntu hosts for specific reasons). You can archive this by creating groups and limiting the specific roles or tasks to specific groups. I personally have AMD64 and ARM64 hosts and respective groups to select which roles goes to every host. [4]

That said, all we need is that the tasks in the templates are well targeted (as for which distro they are intended), tagged (which would facilitate sectioning the runs) and documented (to speed up customization for those who have a setup different than the template is meant to). I'm not proposing you "officially" support playbooks for multiple distros (or even architectures), just having the playbooks flexible would be enough.

[onedr0p]

Using ansible.builtin.package which is OS agnostic, as long as the package has the same name across package managers.

Yep, that works, but we would probably need a ubuntu_package: [] and a fedora_packages: [] list of packages for each OS since the names definitely do not carry over.

For tasks beyond package installation, you can fetch the target information during setup time and filter, selecting the right task for the right distro. I've added an example in https://github.com/k8s-at-home/flux-cluster-template/pull/396. You can find detailed information and extensive examples in the Ansible docuentation. [1] [2]

We can even add tags to specific blocks of tasks to add the possibility of filtering out tasks that would otherwise fail in a different distro. I also added an example in the above mentioned PR. [3]

I am aware of the built-in ansible_distribution and ansible_distribution_version variables, it's just that the logic of separating out things is annoying to support because future changes in the OS's across Ubuntu and Fedora, I could easily handle supporting one instead of 2 different OS.

The things that are different on Fedora and Ubuntu include:

• Fedora: Package names, Disabling Selinux, Disabling Swap, Disabling firewalld
• Ubuntu: Package names, Disabling apparmour, Disabling Swap, Disabling / Uninstalling ufw, Uninstall snap and it's preinstalled snaps, installing and configuring unattended-upgrades.

I am only a fan of using tags for medium to large ansible scripts where you need to skip certain tasks that might take too long, I don't see tags being much help here as the playbook runs very quickly and currently is 100% idempotent. It's pretty cumbersome for the user to know what the tags are and how to run playbooks with only certain tags. I would like to keep that simple and not use tags IMO.

4. Lastly we could even mix different distros in the same (let's say you have Fedora AND Ubuntu hosts for specific reasons). You can archive this by creating groups and limiting the specific roles or tasks to specific groups. I personally have AMD64 and ARM64 hosts and respective groups to select which roles goes to every host. [4]

I do not see the need for having different OS's in a cluster and would cringe at the thought of someone doing this (at least in the scope of homelab). :)

[remkolems]

Thanks @onedr0p (and of course all others) for your hard work! The utmost appreciation for your highly opinionated insight(s).

FYI and FWIW I'm new to many facets and aspects of this repo. Unfortunately, I just freshly installed Ubuntu to all my bare-metal nodes, so therefore I'm tied to Ubuntu for the time being. The swap to Fedora was in my case for now an oh-oh moment. As of last week I'm to trying to keep up with this repo, but with one caveat: I'm made the swap completely back to Ubuntu (completely untested until this date and at least for the foreseeable weeks as I'm still grasping/learning all of the aspects of this repo).

My final use-case in due time I want to implement various additions/integrations to this repo (not exhaustive: Rancher UI, pfSense, TrueNAS, SSO/User management, customized OS/kernel optimizations). Goal: to have a hardened Rancher k3s HA-cluster (3 master nodes) based on this repo to manage/visualize/etc. various hardened production clusters. 1) Single main data science cluster (10 nodes) 2) Various NVidia Jetson edge k3s clusters with automated and customized OS (>10+ nodes) + a lot of other IOT data sensors/nodes.

This repo made me reevaluate a lot of minute details. Thanks!! The swap to Fedora is interesting.

My Ubuntu repo can be found at: https://github.com/remkolems/flux-cluster-k3s-ubuntu-template

[brettinternet]

You wouldn't have to use tags to support different OSs in an ansible role. You could use include_tasks where the task name is the distribution name e.g. archlinux.yml, debian.yml or fedora.yml.

---
- name: Perform some OS task
  include_tasks: "{{ ansible_facts['distribution'] | lower }}.yml"

Then, support for OSs could grow over time as tasks are copied for different OS requirements. It would also help delineate OS-specific attributes in OS task files so it's very clear what fedora, debian, etc require individually within each role.

[nachtschatt3n]

Did anybody try to run the new version on a Raspberry Cluster?

[brettinternet]

I run on few raspbian arm64 as well as some archlinux amd64 nodes: https://github.com/brettinternet/homelab. Join us in the k8s-at-home Discord for more generalized support or troubleshooting :)