Container optimized linux OSes - flatcar / talos / bottlerocket

[ncsokas]

Spin up one of each, and figure out the differences.

AWS VM’s

• Time/Hazzle from 0 to working cluster (config/Knowledge/capabilities)
• Scalability of the cluster.
• Upgrade path

‌

Possible distributions/tools to look into:

• Talos
• Kairos
• Flatcar
• Bottlerocket

┆Card is synchronized with this Github issue by Unito

[ncsokas]

Sofus Albertsen commented in Trello:

I know @dominikwawrzkiewicz4 have looked at Talos

[ncsokas]

Michael Ingeman-Nielsen commented in Trello:

@danielredelberger2 and I have looked into this on Friday 19/1 2024.

We looked mainly at Flatcar and Bottlerocket.

Findings

• Flatcar and Bottlerocket are both distributions that give you an immutable OS with the single purpose of running containers.
  • Talos and Kairos seems to be the same, but with the added layer of running a full Kubernetes node instead of just Docker
• Both work in basically the same way:
  • You get a disk image with the preinstalled OS, which is spun up as a VM.
  • You provide a configuration to modify network and other OS setting at startup. It also lets you define services to run on the OS - e.g. a systemd service that pulls and runs an image at startup
  • Once started, the OS partition is read-only. You’re not able to ssh into the box (by default) and no changes can be made to the OS. If you want storage for your container, you mount it in from a storage server. Unsure whether a secondary HDD partition can be mounted.
• Upgrades:
  • There are two /usr partitions: one active (readonly) and one passive. When updating, a new OS image is pulled down and copied into the passive /usr partition. On reboot, the two partitions are swapped.
  • Versioned and atomic upgrades
  • Easy rollback (didn’t try it, though)
• Bottlerocket differences from Flatcar:
  • Tightly tied to AWS ecosystem
  • Availalble as an AWS VM image
  • You use AWS CLI to start/manage the instance
  • You can use other AWS management services tools to control access, securtiy, etc.
  • But conceptually, it’s the same.

Bottom line:

• Use for any static container workload you want to run on-premises or at non-AWS cloud providers
  • e.g. Jenkins master, static jenkins agents, Artifactory, Gitlab etc.
  • Or use it as a remote Docker agent - requires exposing the Docker API, which may be a security concern.
• Consider Bottlerocket if you’re already in AWS. Use AWS tools and terraform to manage it.
• In principle, you could run a K8S node in a container, but use Talos or Kairos instead if that is you need.

Benefits:

• Extremely simple management
• Very limited attack surface
• Atomic upgrades
• Easy rollback to previous versions
• No user management on host OS level
• Configuration as code