search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
14.88k stars 579 forks source link

Flatcar Container Linux support #1372

Open louhisuo opened 1 year ago

louhisuo commented 1 year ago

Description

Flatcar Container Linux (https://www.flatcar.org/) is an interesting immutable Linux OS which suits well to run e.g. Kubernetes. Any change support for Flatcar Linux can be added to Lima as it does not seem to work "out-of-the-box" based on quick testing done.

afbjorklund commented 1 year ago

I think that Flatcar (and the "new" CoreOS) use ignition rather than cloud-init, so I'm not sure it is compatible.

https://www.flatcar.org/docs/latest/installing/vms/qemu/

In theory it would be possible to convert it (user-data), but I'm not sure it is worth it - or if it is enough anyway.

https://www.flatcar.org/docs/latest/provisioning/cl-config/#migrating-from-cloud-configs

afbjorklund commented 1 year ago

Note that you can run Podman Desktop instead of Rancher Desktop, if you want Fedora CoreOS support...

podman machine init

podman machine start

It is "interesting immutable", and has a common ancestor with Flatcar Container Linux (that would be: CoreOS)

I personally prefer Lima, because it is smaller (Fedora is like half the size of Fedora CoreOS) and easier to modify.

But if it is possible to run it (Flatcar) with Vagrant, then it should also be possible to run it with Lima - if needed.

If you run the Alpine image with Lima, instead of the Ubuntu image, then it will actually start from a "LiveCD"...

It is not immutable, but it is half way ? The OS disk and the data disk are separated, unlike the other diff disk.

computeralex92 commented 1 year ago

A switch to ignition is not needed, as Flatcar is still able to be configured via cloud config: https://github.com/flatcar/coreos-cloudinit/blob/flatcar-master/Documentation/cloud-config.md

It is only not anymore the main focus of development and only a subset of options is supported; but anyways the support is still there. I'm using this in production environments and was able to reuse config for the original CoreOS which was more than 4 years old; without any issue.

afbjorklund commented 1 year ago

Are you able to provide a YAML file, for the "stable" image ?

https://www.flatcar.org/docs/latest/installing/vms/qemu/

Typically the cloud-config user-data looks something like:

#cloud-config
# vim:syntax=yaml

growpart:
  mode: auto
  devices: ['/']

users:
  - name: "anders"
    uid: "1000"
    homedir: "/home/anders.linux"
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh-authorized-keys:
      - ... keys go here ...

write_files:
 - content: |
      #!/bin/sh
      set -eux
      LIMA_CIDATA_MNT="/mnt/lima-cidata"
      LIMA_CIDATA_DEV="/dev/disk/by-label/cidata"
      mkdir -p -m 700 "${LIMA_CIDATA_MNT}"
      mount -o ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0 "${LIMA_CIDATA_DEV}" "${LIMA_CIDATA_MNT}"
      export LIMA_CIDATA_MNT
      exec "${LIMA_CIDATA_MNT}"/boot.sh
   owner: root:root
   path: /var/lib/cloud/scripts/per-boot/00-lima.boot.sh
   permissions: '0755'
# This has no effect on systems using systemd-resolved, but is used
# on e.g. Alpine to set up /etc/resolv.conf on first boot.

manage_resolv_conf: true

resolv_conf:
  nameservers:
  - 192.168.5.3

ca-certs:
  remove_defaults: false
  trusted:

Not sure how well the scripts work, on a read-only filesystem ?

afbjorklund commented 1 year ago

Currently there is no support for compressed images in Lima, so the .bz2 needs to be decompressed first.

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2

The image boots, but it doesn't set up the ssh login it seems. From the ssh-authorized-keys in cidata.iso

localhost login: core (automatic login)

Flatcar Container Linux by Kinvolk stable 3374.2.5 for QEMU
core@localhost ~ $

The /dev/cdrom was found, but never mounted anywhere.

Probably because /var/lib/cloud was never created, either.

It seems like cloudinit only runs for some OEMS, or on-demand ?

Otherwise there is a custom shell script, to set up the qemu cmd:

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh

afbjorklund commented 1 year ago 
images:
#- location: "https://stable.release.flatcar-linux.net/amd64-usr/3374.2.5/flatcar_production_qemu_image.img.bz2"
#  arch: "x86_64"
#  digest: "sha512:7aa371a6e7013d02f6e3d6031c155420ce617dfe920d4d744950f87faa6b6f83c5e938525fcba5e8d1fb885e647af20fb412c3b56dbaff1a2d3dfcf7815a3145"
- location: "flatcar_production_qemu_image.img"
  arch: "x86_64"
  digest: "sha256:28f570a120f040523972a7cae948eb6ee41cf444f3c7c92f1bc721bf95ee5891"

video:
  display: "default"

For some reason, Flatcar produces every checksum - except for SHA256...

# MD5 HASH
0a8b08387670cbc62c2d5086622a0956  flatcar_production_qemu_image.img.bz2
# SHA1 HASH
52edb61aacf14e19f23716d20311197fe47a9fa9  flatcar_production_qemu_image.img.bz2
# SHA512 HASH
7aa371a6e7013d02f6e3d6031c155420ce617dfe920d4d744950f87faa6b6f83c5e938525fcba5e8d1fb885e647af20fb412c3b56dbaff1a2d3dfcf7815a3145  flatcar_production_qemu_image.img.bz2
afbjorklund commented 1 year ago

The issue was not the OEM, it was that the labels and paths for user-cloudinit are hardcoded...

Once Flatcar finds the cloud-init config, the user is created and the ssh keys are setup for lima.

But the rest of the instance configuration is failing.

AkihiroSuda commented 1 year ago

@afbjorklund Thanks for taking a look, it looks like these files need to be modified? https://github.com/flatcar/coreos-cloudinit/search?q=%2Fmedia%2Fconfigdrive

afbjorklund commented 1 year ago

Maybe they (the locations) can be configured during runtime somehow, env or cmdline ?

But it seems that it only has a partial cloud-init implementation, no /var/lib/cloud etc

afbjorklund commented 1 year ago

Apparently there is a special image for UEFI, noticed when looking at the arm64 variant.

Debug setup:

images:
- location: "https://stable.release.flatcar-linux.net/amd64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "x86_64"
  digest: "sha512:3fe0068f838f8055fbdc374e9d59ab6828d79f1d1d3c0b93fe6b89893ff06d1cbc4acdd19c0c6a54e7462cf86cdd29340eaa2ed6ad3fd114c552f43bce526718"
- location: "https://stable.release.flatcar-linux.net/arm64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "aarch64"
  digest: "sha512:0f4ee08e9a17ece104d477cbc1ab0eda6f410683aea0067779771587c3864df9f3527c1c647158064424b33918fd5bbabf6384e9d71ef4a98a672565d154aba8"

video:
  display: "default"
afbjorklund commented 1 year ago

Using Fedora CoreOS seemed to be slightly better documented, and it was used by Podman Desktop

It also allows for installing various things in /usr/local, by llinking that directory to /var/usrlocal...

EDIT: Needed feature for installing into /opt instead:

afbjorklund commented 1 year ago

Flatcar doesn't have support for sshfs (or fuse), so it will have to use mountType: "9p" instead.

On a normal operating system you could install it, but that is not possible on an immutable OS.

afbjorklund commented 7 months ago

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

kouhaidev commented 7 months ago

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

Any given reason? This seems interesting.

afbjorklund commented 7 months ago

No interest, duplication of the scripts?

The main user (of CoreOS) is Podman, but they are set on doing their own "machine"... Automatic updates is a nice feature in theory, but they even disabled that feature eventually. Since it meant that the VM would spontaneously reboot, in order to "activate" the new install. https://www.flatcar.org/docs/latest/setup/releases/update-strategies/ (default is reboot in 5m)