Migration from k3os to openSUSE MicroOS

[mysticaltech]

Recently Rancher, the creators of k3s and k3os has been bought by SUSE. And in doing so, they've dropped official support for k3os (k3s on the other hand is thriving and has been separated from Rancher).

I went on to contact Jacob Blain Christen, the lead maintainer of k3os, and he told me that he'll continue to do releases on the weekends and that the project could live on if the community maintained it.

However, that is not a stable backing for this project, so I made my own research and concluded that OpenSuse MicroOS, has HUGE backing has it piggybacks on Tumbleweed, a major OpenSuse distro, and has stable and automated transactional updates, as such it's now the best OS to replace k3os.

[mysticaltech]

@sysrich I saw your article here https://rootco.de/2020-12-09-microos-pi-network-monitor/. I know it does not explain how to run MicroOS on Hetzner, but it says that Hetzner has included the ISO, unfortunately running hcloud iso list does not show it anymore. I am left with thinking about doing something like this in rescue mode:

export MICROOS_DISK=" https://downloadcontent.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-SelfInstall.raw.xz"
curl -sL $MICROOS_DISK | xz -d | dd of=/dev/sda status=progress

• I will try it out, but wanted to get your opinion if possible, do you think it will work?
• After that, I just plan to issue a shutdown -r now command to reboot. Do you think networking will work OOTB?
• And does MicroOS support Hetzner user-data?
• If not, I can mount the disk just after dd finishes, but where do embed the ignition or combustion file? Basically, I want to install my SSH public key and k3s, or at least the former, if I can install k3s via SSH later on.

Last but not least, if you have done this before or have any user-data example or general guidance, it would be very much appreciated.

Am trying you move away from k3os as soon as possible, and I know that MicroOS is a much better solution (thanks to your hard work in part I would imagine)! 🙏

[spigell]

Watching for your progress, thanks for sharing! I also stuck with k3os and my bare metal setup. Seeking for similar functionality from another os.

[mnencia]

About installing MicroOS on Hetzner, did you see https://github.com/balta3/hetzner-microos? It could give some useful hints in the direction of an automated install ok MicroOS on hetzner.cloud

[mysticaltech]

Thanks, @mnencia, yes, thanks for sharing, I saw it too. It is interesting. But that method seems simpler https://major.io/2021/08/20/deploy-fedora-coreos-in-hetzner-cloud/.

I believe OpenSuse MicroOS inspired itself a good deal from Fedora CoreOS, as it even supports "ignition", maybe it even shares some code, so that method should work! 🤞

Will keep you folks posted, ASAP. Hopefully this week.

[mysticaltech]

Folks, recap on the advances. Both methods above have failed.

• The one with dd created a read-only filesystem, even for '/boot/writable' that is not supposed to be read-only. Probably because of the sensitivity of btrfs. So we cannot add the ignition file for SSH.
• The one from hetzner-microos, used kexec to launch the tumbleweed yast installer, all manual, not what we want, and does not give a proper MicroOS, just Tumbleweed with transactional updates if you choose so during the setup. So no go!

However, the good news is that MicroOS has images already loaded with k3s, so we have this option https://downloadcontent.opensuse.org/tumbleweed/appliances/iso/openSUSE-MicroOS.x86_64-k3s-SelfInstall.iso

For now, attempts to install it via kexec on the rescue env have failed, because of space limitations. Even when I load it on volumes. But there are still things to try and I have already requested Hetzner support to add it to their list of ISO (as a plan B), hopefully, they'll do so.

Let's see! 🤞 If you can think of something, do not hesitate!

[mnencia]

• The one with dd created a read-only filesystem, even for '/boot/writable' that is not supposed to be read-only. Probably because of the sensitivity of btrfs. So we cannot add the ignition file for SSH.

I think that's by purpose. You can mark the filesystem as rw by running

btrfs property set /mnt ro false

[mysticaltech]

Oh wow, will try ASAP @mnencia! 🙏

[mnencia]

I tried it, but Ignition does not read the configuration, probably because the partition is not labeled "ignition".

What about trying to use cloud-init? It should not require modifying the image, but only defining the right user-data content.

[mysticaltech]

Awesome! Did you create a folder in /boot/writable called ignition, and copy your generated config.ign to it? See https://en.opensuse.org/Portal:MicroOS/Ignition and https://en.opensuse.org/Portal:MicroOS/Design.

That's the format of mine for instance: {"ignition":{"version":"3.0.0"},"passwd":{"users":[{"name":"root","sshAuthorizedKeys":["ssh-ed25519 ___ me@email.com"]}]}}

Also, with the Hetzner console, you can basically "plug a virtual monitor in the server' and see the boot process, which will tell you if the ignition is picked up or not, or what are the errors generated. If it goes too fast, just use screen recording software.

[mysticaltech]

Also, if you are going to retry, do not forget to resize the btrfs after dd, so it takes the whole space left on the disk.

Last but not least, this image is probably best, because it has k3s installed in it already, we just need to see how it's done and how it updates, with transactional updates too or not. If it uses transactional updates then great, if not we can just install the k3s tumbleweed package in the vanilla MicroOS (above), and have it update in a transactional manner along with other OS updates.

https://mirrorcache.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-16.0.0-k3s-SelfInstall-Snapshot20220201.raw.xz

Source: https://mirrorcache.opensuse.org/tumbleweed/appliances/

[mysticaltech]

@mnencia About cloud-init, it could work and is worth a shot too. My initial attempts failed, but I did not try everything.

About ignition, another thing worth a shot is creating ignition/config.ign on the root of an attached volume!

[mnencia]

Creating ignition/config.ign in the root doesn't work. Let's try with /boot/writable/ignition/config.ign

[mysticaltech]

The docs are not great, but https://en.opensuse.org/Portal:MicroOS/Design reveals the proper ignition location! The above should work 🤞

[mnencia]

I tried putting it under /boot/writable, mounted with the command mount -o subvol=/@/boot/writable /dev/sda4 /mnt, but it doesn't work anyway. Probably there is some subtle requirement that is missing.

[sysrich]

As documented here:

https://en.opensuse.org/Portal:MicroOS/Ignition

The only valid location for an ignition configuration is a seperate storage device with a volume name of ignition and a subdirectory named ignition

On Fri 4. Feb 2022 at 11:43, Marco Nenciarini @.***> wrote:

I tried putting it under /boot/writable, mounted with the command mount -o subvol=/@/boot/writable /dev/sda4 /mnt, but it doesn't work anyway. Probably there is some subtle requirement that is missing.

— Reply to this email directly, view it on GitHub https://github.com/kube-hetzner/kube-hetzner/issues/35#issuecomment-1029854089, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJENIDMI73FRHOO7SZSPXLUZOUT7ANCNFSM5NFVEL2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[mysticaltech]

Thank you @sysrich, really appreciate the guidance! 🙏 I obviously had completely missed that! 🤦

Will attempt it with an attached volume. @mnencia If you are going to try this before me, just wanted to share this new finding of mine, the hcloud cli has an add-label command that could be handy here (if you didn't know already).

[phaer]

If it's possible to resize the btrfs filesystem ourselves from hetzners rescue system, we could maybe just add a small, labeled partition at the end? That one wouldn't require an external volume. Going to test that :)

[mnencia]

I think the add-label option in hcloud is about Hetzner metadata. However, Ignition doesn't run even creating a volume like the following

/dev/sdb1: LABEL="ignition" UUID="3dc811c0-9891-4e0f-8b94-1cc38f3967aa" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="ignition" PARTUUID="8fbab7ef-6be0-46b6-b594-53d9847a9e8d"

Looking in the system and initrd content it looks like there is no ignition handling code running in the SelfInstall flavor.

[mnencia]

I've tried with the volume and MicroOS based on Leap 15.3 (Experimental) and it works

Image: https://download.opensuse.org/repositories/openSUSE:/Leap:/Micro:/5.1/images/openSUSE-Leap-Micro.x86_64-Default.raw.xz

[mysticaltech]

@mnencia Great find, and well done. If we can find the Default raw image for MicroOS based on Tumbleweed that works, that's it!

Maybe the ignition logic for these is located somewhere else? And does not support volumes.

@phaer Please let us know if you try with partioniong and it works with the k3s self install image, it would be ideal!

Just FYI folks, when I ran dd myself yesterday, I had to run parted -l to correct a size mismatched error that was showing up on fdisk -l /dev/sda.

Also, I believe it's important to rezize the btrfs to max. Saying all this because it could perhaps influence the ignition seeking mechanisms on Tumbleweed based images?!

[phaer]

Hi, my approach did not work with the k3s self install image, but it did work with the one based on leap 15.3, as recommended by @mnencia.

I did not yet succeed to install k3s on this, as the default repo seems unavailable after boot.

Here's the terraform snippet I used to provision my test server. /root/config.ignwas provisioned beforehand via poseidon/ct provider:

  provisioner "remote-exec" {
    inline = [
      "set -ex",
      "gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0x22C07BA534178CD02EFE22AAB88B2FD43DBDC284",
      "export MIRROR_URL='https://download.opensuse.org/repositories/openSUSE:/Leap:/Micro:/5.1/images'",
      "export IMAGE_NAME='openSUSE-Leap-Micro.x86_64-Default.raw.xz'",
      "export IMAGE_URL=$MIRROR_URL/$IMAGE_NAME",
      "curl --progress-bar -L -o $IMAGE_NAME $IMAGE_URL",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256 $IMAGE_URL.sha256",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256.asc $IMAGE_URL.sha256.asc",
      "gpg --verify $IMAGE_NAME.sha256.asc",
      "sha256sum -c $IMAGE_NAME.sha256",
      "cat $IMAGE_NAME | xz -d | dd of=/dev/sda status=progress",
      "sgdisk -e /dev/sda",
      "partprobe /dev/sda",
      "parted -s /dev/sda resizepart 3 99%",
      "parted -s /dev/sda mkpart primary ext2 99% 100%",
      "mount /dev/sda3 /mnt/ && btrfs filesystem resize max /mnt && umount /mnt",
      "mke2fs -L ignition /dev/sda4",
      "mount /dev/sda4 /mnt",
      "mkdir /mnt/ignition",
      "cp /root/config.ign /mnt/ignition/config.ign",
      "umount /mnt",
      "shutdown -r +1",
      "sleep 1",
      "exit 0"
    ]

[mnencia]

This works and the resulting system is executing ignition

MICROOS_DISK="https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2"
curl -sL "${MICROOS_DISK}" -o /tmp/MicroOS.qcow2
qemu-img convert -p -f qcow2 -O host_device /tmp/MicroOS.qcow2 /dev/sda

[mysticaltech]

Wonderful! You made my day @mnencia! :-) Well done everyone. Now serious business can begin!

[mysticaltech]

Now the next steps are very clear:

• Check the partition size, resize to max if necessary (like @phaer did above)
• Verify networking, especially the private network interface, which needs to work well before anything else can happen. It should receive DHCP, and maybe we need to add routes to the Hetzner GW to it like done in the current code.
• Then, we have to run k3s for each of the three configurations, starting with the control plane.

I will definitely look at this as soon as I can but if you continue the work, please do not hesitate to share your success and failures, so we advance faster!

The last step will be to configure the cluster with kured for automatic safe rebooting after upgrades, and also transactional-updates, which luckily for us has full support for it.

[mnencia]

This list of commands works for me.

I had to remove the gpg and sha256sum verification because they do not match.

[
      "set -ex",
      "gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0x22C07BA534178CD02EFE22AAB88B2FD43DBDC284",
      "MIRROR_URL='https://mirrorcache.opensuse.org/tumbleweed/appliances/'",
      "IMAGE_NAME='openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2'",
      "IMAGE_URL=$MIRROR_URL/$IMAGE_NAME",
      "curl --progress-bar -L -o $IMAGE_NAME $IMAGE_URL",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256 $IMAGE_URL.sha256",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256.asc $IMAGE_URL.sha256.asc",
      # "gpg --verify $IMAGE_NAME.sha256.asc", # TODO: this doesn't match
      # "sha256sum -c $IMAGE_NAME.sha256", # TODO: this doesn't match
      "qemu-img convert -p -f qcow2 -O host_device $IMAGE_NAME /dev/sda",
      "sgdisk -e /dev/sda",
      "partprobe /dev/sda",
      "parted -s /dev/sda resizepart 4 99%",
      "parted -s /dev/sda mkpart primary ext2 99% 100%",
      "mount /dev/sda4 /mnt/ && btrfs filesystem resize max /mnt && umount /mnt",
      "mke2fs -L ignition /dev/sda5",
      "mount /dev/sda5 /mnt",
      "mkdir /mnt/ignition",
      "cp /root/config.ign /mnt/ignition/config.ign",
      "umount /mnt",
      "shutdown -r +1",
      "sleep 1",
      "exit 0"
]

[mysticaltech]

Beautiful! Yes, image verification is not a big issue for now.

[phaer]

I had to remove the gpg and sha256sum verification because they do not match.

Interesting, I had the same problem when I tried that image, but just failed to reproduce it. Could be that those keys were just updated, or maybe there's inconsistent state between mirrors and we got different redirects.

@mnencia Could you try if verification works if you use the following mirror for all 3 files (qcow, sha256sum, gpg signature)? If so, I would report that issue upstream.

MIRROR_URL='http://mirror.easyname.at/opensuse/tumbleweed/appliances/'

It is of course most important to get it to work at all, so great that it works without verification as a first step! :tada:

[mnencia]

@phaer You are right. Using the URL you provided works perfectly. Probably the mirrorcache URL was redirecting the download to different mirrors with different images.

[mysticaltech]

Why not just do this without mirrors folks? Just use https://download.opensuse.org/opensuse/tumbleweed/appliances/. That's a personal preference at least.

[mnencia]

Because it means to use the mirrors anyway

$ curl -I https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-16.0.0-k3s-kvm-and-xen-Snapshot20220202.qcow2
HTTP/2 302
location: https://mirrorcache.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-16.0.0-k3s-kvm-and-xen-Snapshot20220202.qcow2
content-type: text/html; charset=iso-8859-1
date: Fri, 04 Feb 2022 21:24:08 GMT
server: Apache

[mysticaltech]

Ok I see, yeah makes sense!

[mysticaltech]

Turns out we can simplify even more just by using the metalink and aria2 (thank you get.opensuse.org, I wish we had found that before).

It takes care of mirrors and even does automatic checksum checks, and corrections if I understand correctly. Basically, it just gets the job done without headaches, just by issuing:

aria2c https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-kvm-and-xen.qcow2.meta4

[mysticaltech]

Ok, that should work:

[
      "set -ex",
      "apt install -y aria2",
      "aria2c https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2.meta4",
      "qemu-img convert -p -f qcow2 -O host_device $(ls -a | grep MicroOS | grep -v meta4) /dev/sda",
      "sgdisk -e /dev/sda",
      "partprobe /dev/sda",
      "parted -s /dev/sda resizepart 4 99%",
      "parted -s /dev/sda mkpart primary ext2 99% 100%",
      "mount /dev/sda4 /mnt/ && btrfs filesystem resize max /mnt && umount /mnt",
      "mke2fs -L ignition /dev/sda5",
      "mount /dev/sda5 /mnt",
      "mkdir /mnt/ignition",
      "cp /root/config.ign /mnt/ignition/config.ign",
      "umount /mnt",
      "shutdown -r +1",
      "sleep 1",
      "exit 0"
]

[mnencia]

Wouldn't be better to use the -o option to use a fixed destination name?

[mysticaltech]

No, unfortunately, it doesn't work for metalinks and torrents, because the name is dynamic.

But anyways, got it working and it's beautiful, thank you so much for the amazing work today, that is art! :) Will push a staging branch as soon as I have something solid.

[mnencia]

What about using

TMP_DIR=$(mktemp -d)
aria2c --dir=$TMP_DIR --follow-metalink=mem https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2.meta4

It is safer to isolate the download in a dedicated directory

[mysticaltech]

Love the --follow-metalink=mem definitely cleaner! 🙏

About the temp dir, I just feel it's overkill because we are on the rescue env anyway. But to make it safer, I made the grep a lot more selective with a proper regex that matches the full thing, without room for doubt!

[mysticaltech]

Have created a new branch called staging and have pushed the initial work in there.

Please note that agents.tf, servers.tf, and output.tf are stored in the temp folder for the moment. I am adding to the root folder only the necessary files needed to reach the next step.

Please do not hesitate to submit PR there.

For now, the master instance is created, we are able to access it with SSH, and k3s is there so this is great.

But we have to find a way to configure eth1 because it comes DOWN. If you have any ideas, please shoot. I believe we may need to add a simple combustion file, in the extra partition, renaming the label to "combustion" (they allow both ignition and combustion, so that is great).

[mysticaltech]

So ip link set eth1 up does turn the interface UP. But no DHCP happens, and I do not know how to provoke it. Maybe we have to install new packages?! Seems unlikely.

Also wicked ifup all returns the following, without eth1, as if it does not have it in its config!

[mysticaltech]

Indeed, it's not configured. That article I believe says how to copy the configuration file. Apparently etc is writable so config should persist, no need for "combustion" for now.

[mysticaltech]

Done for today - Remember, if you advance, please start by pulling the staging branch, and when you are done open a PR there - Or just shoot ideas here!

Everything is welcome to move this needle forward, and thanks one more time for the great job today @phaer and @mnencia ! You folks saved the day 🙏

[mnencia]

Added eth1 and hostname in #40

[mysticaltech]

Wonderful, thank you @mnencia, simple and elegant solution! 🙏

[mysticaltech]

Alright, folks, initial logic is done and has been pushed to the staging branch, more testing is required now.

The key to unlocking this last part was finding out how k3s is installed on MicroOS, thanks to https://build.opensuse.org/package/show/openSUSE:Factory/k3s.

Also, made use of the config.yaml option to configure k3s, as explained in k3s configuration file.

What remains is installing Kured on the cluster itself and configuring it if needed. For that, someone needs to install Kubic (also a MicroOS derivative, that uses kubeadm), as it it already configured to work with transactional-udpates and dump this file /usr/share/kured/kured-<version>.yaml as explained in here, to see what the config params are.

Any PRs, comments, or tweaks are always welcome! 🙏

[phaer]

@mysticaltech Your last link, about Kured's configuration links to k3s docs a second time, I think thats a mistake?

Just installed Kubic in a VM, but /usr/share/kured/kured-<version>.yaml does not exist after boot. find / -iname '*kured*' only yields the following file:

/usr/share/k8s-yaml/kured/kured.yaml

```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kured rules: # Allow kured to read spec.unschedulable # Allow kubectl to drain/uncordon # # NB: These permissions are tightly coupled to the bundled version of kubectl; the ones below # match https://github.com/kubernetes/kubernetes/blob/v1.19.4/staging/src/k8s.io/kubectl/pkg/cmd/drain/drain.go # - apiGroups: [""] resources: ["nodes"] verbs: ["get", "patch"] - apiGroups: [""] resources: ["pods"] verbs: ["list","delete","get"] - apiGroups: ["apps"] resources: ["daemonsets"] verbs: ["get"] - apiGroups: [""] resources: ["pods/eviction"] verbs: ["create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kured roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kured subjects: - kind: ServiceAccount name: kured namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: kube-system name: kured rules: # Allow kured to lock/unlock itself - apiGroups: ["apps"] resources: ["daemonsets"] resourceNames: ["kured"] verbs: ["update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: kube-system name: kured subjects: - kind: ServiceAccount namespace: kube-system name: kured roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kured --- apiVersion: v1 kind: ServiceAccount metadata: name: kured namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kured # Must match `--ds-name` namespace: kube-system # Must match `--ds-namespace` spec: selector: matchLabels: name: kured updateStrategy: type: RollingUpdate template: metadata: labels: name: kured spec: serviceAccountName: kured tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule hostPID: true # Facilitate entering the host mount namespace via init restartPolicy: Always containers: - name: kured image: registry.opensuse.org/kubic/kured:1.9.1 # If you find yourself here wondering why there is no # :latest tag on Docker Hub,see the FAQ in the README imagePullPolicy: Always securityContext: privileged: true # Give permission to nsenter /proc/1/ns/mnt env: # Pass in the name of the node on which this pod is scheduled # for use with drain/uncordon operations and lock acquisition - name: KURED_NODE_ID valueFrom: fieldRef: fieldPath: spec.nodeName command: - /usr/bin/kured # - --force-reboot=false # - --drain-grace-period=-1 # - --skip-wait-for-delete-timeout=0 # - --drain-timeout=0 # - --period=1h # - --ds-namespace=kube-system # - --ds-name=kured # - --lock-annotation=weave.works/kured-node-lock # - --lock-ttl=0 # - --prometheus-url=http://prometheus.monitoring.svc.cluster.local # - --alert-filter-regexp=^RebootRequired$ # - --alert-firing-only=false # - --reboot-sentinel=/var/run/reboot-required # - --prefer-no-schedule-taint="" # - --reboot-sentinel-command="" # - --slack-hook-url=https://hooks.slack.com/... # - --slack-username=prod # - --slack-channel=alerting # - --notify-url="" # See also shoutrrr url format # - --message-template-drain=Draining node %s # - --message-template-drain=Rebooting node %s # - --blocking-pod-selector=runtime=long,cost=expensive # - --blocking-pod-selector=name=temperamental # - --blocking-pod-selector=... # - --reboot-days=sun,mon,tue,wed,thu,fri,sat # - --reboot-delay=90s # - --start-time=0:00 # - --end-time=23:59:59 # - --time-zone=UTC # - --annotate-nodes=false # - --lock-release-delay=30m # - --log-format=text ```

I can't really investigate further atm, but that does look more like an example than a working config.

[mysticaltech]

Yes, now corrected! Thanks for catching @phaer :)

[mnencia]

I tried running kured using the manifest provided by @phaer, and it looks working:

time="2022-02-06T19:04:44Z" level=info msg="Binding node-id command flag to environment variable: KURED_NODE_ID"
time="2022-02-06T19:04:44Z" level=info msg="Kubernetes Reboot Daemon: 1.9.1"
time="2022-02-06T19:04:44Z" level=info msg="Node ID: k3s-control-plane-0"
time="2022-02-06T19:04:44Z" level=info msg="Lock Annotation: kube-system/kured:weave.works/kured-node-lock"
time="2022-02-06T19:04:44Z" level=info msg="Lock TTL not set, lock will remain until being released"
time="2022-02-06T19:04:44Z" level=info msg="Lock release delay not set, lock will be released immediately after rebooting"
time="2022-02-06T19:04:44Z" level=info msg="PreferNoSchedule taint: "
time="2022-02-06T19:04:44Z" level=info msg="Blocking Pod Selectors: []"
time="2022-02-06T19:04:44Z" level=info msg="Reboot schedule: SunMonTueWedThuFriSat between 00:00 and 23:59 UTC"
time="2022-02-06T19:04:44Z" level=info msg="Reboot check command: [test -f /var/run/reboot-required] every 1h0m0s"
time="2022-02-06T19:04:44Z" level=info msg="Reboot command: [/usr/bin/systemctl reboot]"

I've checked and the transactional-update system looks already configured correctly:

k3s-control-plane-0:~ # cat /etc/transactional-update.conf
REBOOT_METHOD=kured

static:~ # systemctl status transactional-update.timer
● transactional-update.timer - Daily update of the system
     Loaded: loaded (/usr/lib/systemd/system/transactional-update.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Sun 2022-02-06 17:55:00 UTC; 1h 23min ago
    Trigger: Mon 2022-02-07 00:35:47 UTC; 5h 17min left
   Triggers: ● transactional-update.service
       Docs: man:transactional-update(8)

Feb 06 17:55:00 static systemd[1]: Started Daily update of the system.

k3s-control-plane-0:~ # cat /usr/lib/systemd/system/transactional-update.timer
[Unit]
Description=Daily update of the system
Documentation=man:transactional-update(8)
After=network.target local-fs.target

[Timer]
OnCalendar=daily
AccuracySec=1m
RandomizedDelaySec=2h
#Persistent=true

[Install]
WantedBy=timers.target

[mnencia]

The recipe to build the kured.yaml file is here.

mkdir -p %{buildroot}%{_datadir}/k8s-yaml/kured
cat kured-rbac.yaml kured-ds.yaml > %{buildroot}%{_datadir}/k8s-yaml/kured/kured.yaml
chmod 644  %{buildroot}%{_datadir}/k8s-yaml/kured/kured.yaml
sed -i -e 's|image: .*|image: registry.opensuse.org/kubic/kured:%{version}|g' %{buildroot}%{_datadir}/k8s-yaml/kured/kured.yaml

It's the concatenation of kured-rbac.yaml and kured-ds.yaml, with the image repointed from docker.io/weaveworks/kured to registry.opensuse.org/kubic/kured.

[mnencia]

That is precisely the exact content of https://github.com/weaveworks/kured/releases/download/1.9.1/kured-1.9.1-dockerhub.yaml, but the imagePullPolicy and the image fields in the daemonset spec. Using the GitHub release file directly is fine, but if we want to have the same behavior as Kubic, we can easily accomplish it using kustomize.

[mysticaltech]

Great find folks, so that basically confirms that MicroOS, and Kubic do create the default "indicator" file /var/run/reboot-required when it needs to reboot. And Kured picks this up, and issues the standard systemctl reboot command when it's ready. They kept it simple! :)

Will just go with the Github release directly for now, as I do not see the default behavior changing anytime soon.