Flatcar Container Linux support

[louhisuo]

Description

Flatcar Container Linux (https://www.flatcar.org/) is an interesting immutable Linux OS which suits well to run e.g. Kubernetes. Any change support for Flatcar Linux can be added to Lima as it does not seem to work "out-of-the-box" based on quick testing done.

[afbjorklund]

I think that Flatcar (and the "new" CoreOS) use ignition rather than cloud-init, so I'm not sure it is compatible.

https://www.flatcar.org/docs/latest/installing/vms/qemu/

In theory it would be possible to convert it (user-data), but I'm not sure it is worth it - or if it is enough anyway.

https://www.flatcar.org/docs/latest/provisioning/cl-config/#migrating-from-cloud-configs

[afbjorklund]

Note that you can run Podman Desktop instead of Rancher Desktop, if you want Fedora CoreOS support...

podman machine init

podman machine start

It is "interesting immutable", and has a common ancestor with Flatcar Container Linux (that would be: CoreOS)

---

I personally prefer Lima, because it is smaller (Fedora is like half the size of Fedora CoreOS) and easier to modify.

But if it is possible to run it (Flatcar) with Vagrant, then it should also be possible to run it with Lima - if needed.

If you run the Alpine image with Lima, instead of the Ubuntu image, then it will actually start from a "LiveCD"...

It is not immutable, but it is half way ? The OS disk and the data disk are separated, unlike the other diff disk.

[computeralex92]

A switch to ignition is not needed, as Flatcar is still able to be configured via cloud config: https://github.com/flatcar/coreos-cloudinit/blob/flatcar-master/Documentation/cloud-config.md

It is only not anymore the main focus of development and only a subset of options is supported; but anyways the support is still there. I'm using this in production environments and was able to reuse config for the original CoreOS which was more than 4 years old; without any issue.

[afbjorklund]

Are you able to provide a YAML file, for the "stable" image ?

https://www.flatcar.org/docs/latest/installing/vms/qemu/

Typically the cloud-config user-data looks something like:

#cloud-config
# vim:syntax=yaml

growpart:
  mode: auto
  devices: ['/']

users:
  - name: "anders"
    uid: "1000"
    homedir: "/home/anders.linux"
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh-authorized-keys:
      - ... keys go here ...

write_files:
 - content: |
      #!/bin/sh
      set -eux
      LIMA_CIDATA_MNT="/mnt/lima-cidata"
      LIMA_CIDATA_DEV="/dev/disk/by-label/cidata"
      mkdir -p -m 700 "${LIMA_CIDATA_MNT}"
      mount -o ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0 "${LIMA_CIDATA_DEV}" "${LIMA_CIDATA_MNT}"
      export LIMA_CIDATA_MNT
      exec "${LIMA_CIDATA_MNT}"/boot.sh
   owner: root:root
   path: /var/lib/cloud/scripts/per-boot/00-lima.boot.sh
   permissions: '0755'
# This has no effect on systems using systemd-resolved, but is used
# on e.g. Alpine to set up /etc/resolv.conf on first boot.

manage_resolv_conf: true

resolv_conf:
  nameservers:
  - 192.168.5.3

ca-certs:
  remove_defaults: false
  trusted:

Not sure how well the scripts work, on a read-only filesystem ?

[afbjorklund]

Currently there is no support for compressed images in Lima, so the .bz2 needs to be decompressed first.

• 809

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2

The image boots, but it doesn't set up the ssh login it seems. From the ssh-authorized-keys in cidata.iso

localhost login: core (automatic login)

Flatcar Container Linux by Kinvolk stable 3374.2.5 for QEMU
core@localhost ~ $ 

---

The /dev/cdrom was found, but never mounted anywhere.

Probably because /var/lib/cloud was never created, either.

It seems like cloudinit only runs for some OEMS, or on-demand ?

Otherwise there is a custom shell script, to set up the qemu cmd:

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh

[afbjorklund]

images:
#- location: "https://stable.release.flatcar-linux.net/amd64-usr/3374.2.5/flatcar_production_qemu_image.img.bz2"
#  arch: "x86_64"
#  digest: "sha512:7aa371a6e7013d02f6e3d6031c155420ce617dfe920d4d744950f87faa6b6f83c5e938525fcba5e8d1fb885e647af20fb412c3b56dbaff1a2d3dfcf7815a3145"
- location: "flatcar_production_qemu_image.img"
  arch: "x86_64"
  digest: "sha256:28f570a120f040523972a7cae948eb6ee41cf444f3c7c92f1bc721bf95ee5891"

video:
  display: "default" 

For some reason, Flatcar produces every checksum - except for SHA256...

# MD5 HASH
0a8b08387670cbc62c2d5086622a0956  flatcar_production_qemu_image.img.bz2
# SHA1 HASH
52edb61aacf14e19f23716d20311197fe47a9fa9  flatcar_production_qemu_image.img.bz2
# SHA512 HASH
7aa371a6e7013d02f6e3d6031c155420ce617dfe920d4d744950f87faa6b6f83c5e938525fcba5e8d1fb885e647af20fb412c3b56dbaff1a2d3dfcf7815a3145  flatcar_production_qemu_image.img.bz2

[afbjorklund]

The issue was not the OEM, it was that the labels and paths for user-cloudinit are hardcoded...

• The label of the CD is "config-2", not "cidata"

• The mount path of it is "/media/configdrive", not "/mnt/lima-cidata"

• The name of the file is "openstack/latest/user_data", not "user-data"

---

Once Flatcar finds the cloud-init config, the user is created and the ssh keys are setup for lima.

But the rest of the instance configuration is failing.

[AkihiroSuda]

@afbjorklund Thanks for taking a look, it looks like these files need to be modified? https://github.com/flatcar/coreos-cloudinit/search?q=%2Fmedia%2Fconfigdrive

[afbjorklund]

Maybe they (the locations) can be configured during runtime somehow, env or cmdline ?

But it seems that it only has a partial cloud-init implementation, no /var/lib/cloud etc

[afbjorklund]

Apparently there is a special image for UEFI, noticed when looking at the arm64 variant.

Debug setup:

images:
- location: "https://stable.release.flatcar-linux.net/amd64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "x86_64"
  digest: "sha512:3fe0068f838f8055fbdc374e9d59ab6828d79f1d1d3c0b93fe6b89893ff06d1cbc4acdd19c0c6a54e7462cf86cdd29340eaa2ed6ad3fd114c552f43bce526718"
- location: "https://stable.release.flatcar-linux.net/arm64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "aarch64"
  digest: "sha512:0f4ee08e9a17ece104d477cbc1ab0eda6f410683aea0067779771587c3864df9f3527c1c647158064424b33918fd5bbabf6384e9d71ef4a98a672565d154aba8"

video:
  display: "default"

[afbjorklund]

Using Fedora CoreOS seemed to be slightly better documented, and it was used by Podman Desktop

It also allows for installing various things in /usr/local, by llinking that directory to /var/usrlocal...

EDIT: Needed feature for installing into /opt instead:

• https://github.com/lima-vm/lima/issues/1654

[afbjorklund]

Flatcar doesn't have support for sshfs (or fuse), so it will have to use mountType: "9p" instead.

On a normal operating system you could install it, but that is not possible on an immutable OS.

[afbjorklund]

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

[kouhaidev]

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

Any given reason? This seems interesting.

[afbjorklund]

No interest, duplication of the scripts?

• cloud-init -> ignition.yaml + butane
• boot/* -> ignition.yaml (systemd)

The main user (of CoreOS) is Podman, but they are set on doing their own "machine"... Automatic updates is a nice feature in theory, but they even disabled that feature eventually. Since it meant that the VM would spontaneously reboot, in order to "activate" the new install. https://www.flatcar.org/docs/latest/setup/releases/update-strategies/ (default is reboot in 5m)

[MayCXC]

I'm super interested in this, but I can understand why maintaining ignition.yaml is unappealing.

[afbjorklund]

Most of the cloud-init and ignition/butane is static, so it is actually the OS that is unappealing.

---

I mean it being (mostly) read-only, and requiring a reboot for each (system) installation. Or that the image is much bigger than the regular OS, or requiring special OCI repositories.

Things like that? I'm sure there is some use case where it fits, I just don't like it much myself. But some of the new OS (like Talos) even go all the way, and remove ssh and bash completely...