Questions from Gitter about building/deployments

[pauldotknopf]

I'm posting it here so that it can be easier to expand upon.

From Gitter:

Hello,

I write to you, because Darch a project of yours caught my interest.

Lately I think of hopping to another distribution from Solus, mainly because of missing packages. However even as I was a Gentoo and a Slackware user in the past I just dread installing another Linux distribution. Mainly because of laziness. I'm thinking about straight up Ubuntu and Void, so I'm happy to see that both are supported by Darch.

Darch is a great idea! It uses familiar tools, but enables statelessness. I probably should have written this mail after I tried it, but I want to ask you a few things.

You mention NixOS. but have you heard about OSTree project and Silverblue Fedora? I wonder what do you think about their approach. It certainly is more involved. That's one of the things I like in your idea - just use what one already uses.

How do updates work? I understand that one can take an image from Docker Hub for example, build on one's computer or on some CI. How do you manage to push new images? Do you just have a scheduled job and check for updates daily or so, build and push them? Then on Darch system one has to manually pull for updates and install if available? My father uses a Linux system and I think that it would be easier for me to maintain and install new software for him with Darch. However it would be best if updates would be transparent.

I also use Chromebooks personally. There are many things that I don't like about them, but one feature is amazing. Updates are really easy and it makes the system maintenance free. Images are quite small and releases not so frequent so it does not get in the way. Also when it can't boot from new image it automatically boots from the previous working one and marks new as bad. I was wondering how far could Darch go with being ChromeOS-like in those respects? Mainly update in background, two partition scheme and reboot to previous working version if something is wrong.

Darch images most of the time for a desktop use would be probably quite big. Is some kind of binary delta possible?

I was dabbling a bit with my own Linux distribution - Tomatoaster [1]. I don't have much, mainly building a minimal rootfs image without root, that could boot to Firefox on qemu. I'm using Void tools to build it, but a result would not be a Void system. Main feature is to have it ChromeOS-like updates. I planned to check the access order of files while booting to Firefox to optimally order files in squashfs image. On SSD it would probably not be noticeable, but on HDD, eMMC or streaming from network it could be helpful.

[pauldotknopf]

Lately I think of hopping to another distribution from Solus, mainly because of missing packages. However even as I was a Gentoo and a Slackware user in the past I just dread installing another Linux distribution. Mainly because of laziness. I'm thinking about straight up Ubuntu and Void, so I'm happy to see that both are supported by Darch.

This is similar to my position. I've been on Arch for a few years, but I just don't really care about bleeding edge. If I need latest bits of something, I'll compile it. Ubuntu is my home now. It isn't "hipster", but it fucking works and is well supported. Shoot me. ;)

You mention NixOS. but have you heard about OSTree project and Silverblue Fedora? I wonder what do you think about their approach. It certainly is more involved. That's one of the things I like in your idea - just use what one already uses.

I'm not familiar with Silverblue, but I am familiar with OSTree. I work on embedded Linux software. I was tasked with implementing an update mechanism that never bricks and is super reliable. OSTree was a more complicated setup, but it would have served the purpose. It is basically Git, but optimized for binaries. A tmpfs overlay could easily be added over it. In the end though, at work, I decided to just roll my own initramfs scripts that mounted a squashfs image with a tmpfs overlay. It turned out to be really straight-forward. When I was done, I said "I want this, but for my Desktop, using Docker." So, I built Darch.

How do updates work? I understand that one can take an image from Docker Hub for example, build on one's computer or on some CI.

Yup. Every time I push a commit, Travis builds my images. See here.

How do you manage to push new images?

See here.

Then on Darch system one has to manually pull for updates and install if available?

My dotfiles contain a function that I run called update-dev-image. See here.

My father uses a Linux system and I think that it would be easier for me to maintain and install new software for him with Darch. However it would be best if updates would be transparent.

The update-dev-image command mentioned previously could be on a cron job, and if an update is detected, show a notification, prompting "reboot for updates".

I also use Chromebooks personally. There are many things that I don't like about them, but one feature is amazing. Updates are really easy and it makes the system maintenance free. Images are quite small and releases not so frequent so it does not get in the way. Also when it can't boot from new image it automatically boots from the previous working one and marks new as bad. I was wondering how far could Darch go with being ChromeOS-like in those respects? Mainly update in background, two partition scheme and reboot to previous working version if something is wrong.

For work, I did a "dual boot" approach, but instead of dual partitions, it's a dual set of kernel/initramfs/squash images on a single partition. I customized grub to increment a counter until a succesful boot updated grubenv, indicating a succesful boot. If the counter reaches N, it will fall back to the previous image, before the update. I could share those grub sequences if you'd like. However, I don't think it would be a good idea to bake it into Darch, because doing so would take over the entire grub menu/workflow. I'd like Darch images to just be "discovered", as if they are another Linux distro you can select. End users can choose to create custom grub.cfg files if they'd like.

Darch images most of the time for a desktop use would be probably quite big. Is some kind of binary delta possible?

Docker image layers are delta'd. But yes, all the layers combined, your images can get pretty big. My Ubuntu install is roughly 3GB. I could trim a lot out, but I'm comfortable at the moment.

I was dabbling a bit with my own Linux distribution - Tomatoaster [1]. I don't have much, mainly building a minimal rootfs image without root, that could boot to Firefox on qemu. I'm using Void tools to build it, but a result would not be a Void system. Main feature is to have it ChromeOS-like updates. I planned to check the access order of files while booting to Firefox to optimally order files in squashfs image. On SSD it would probably not be noticeable, but on HDD, eMMC or streaming from network it could be helpful.

All said and done, it sounds like you want Darch with some kind of automatic updates in the backgrounds so that non-techies can treat it as if it is like ChromeOS. Like I said, this is do-able now with just some custom grub.cfg stuff. Let me know if you want me to share my scripts with you. I'd have to distill them down a little.

[hadrianw]

Thank you for your response.

It would be splendid if you could share your grub customization. Thank you for that.

I was thinking a bit about updates. I could set up rsync or other sync utility to prepare new images directly on target computer. So it could mount current rootfs somewhere with a fresh tmpfs overlay. Sync on that and then squash the result. That would be a trade off: download size and simplicity vs compute time and complexity.

Docker image layers are delta'd. But yes, all the layers combined, your images can get pretty big. My Ubuntu install is roughly 3GB. I could trim a lot out, but I'm comfortable at the moment.

Does layering in this context mean that for example there is a base image and upon this is next desktop layer and then another one for development? Each additional layer as a delta of all the previous ones? So when downloading the development image one downloads all three which are then combined to form a single image?

If that's the case it could be possible to stack update layers at the end. So first one has: base0, desktop0, dev0. Then after an update: base0, desktop0, dev0, base1, desktop1, dev1. When one has version 0 only delta layers from version 1 would be required.

Oh. But I forgot about inheritance. So the update should be a single delta based on the last image. Squashing base0, desktop0 and dev0 to single image dev1 inheriting from dev0.

This could be better fleshed out, but it could be probably easier bolted on current implementation.

[hadrianw]

Regarding updates of images themselves. I meant for package update. Do you just have a cron job to build a new image every day and if there is a change you publish the new image? Or maybe you detect package upgrades and then automatically build new images?

[pauldotknopf]

I was thinking a bit about updates. I could set up rsync or other sync utility to prepare new images directly on target computer. So it could mount current rootfs somewhere with a fresh tmpfs overlay. Sync on that and then squash the result. That would be a trade off: download size and simplicity vs compute time and complexity.

This is something I've thought about implementing. A darch stage export and darch stage import command, used share the squash files, kernel and initramfs, removing the need from using Docker. You could rsync a "mydarchimage.tar.gz" to your target and import it without the need of containerd (docker).

Does layering in this context mean that for example there is a base image and upon this is next desktop layer and then another one for development? Each additional layer as a delta of all the previous ones? So when downloading the development image one downloads all three which are then combined to form a single image?

Yes. Each new layer only contains the differences.

If that's the case it could be possible to stack update layers at the end. So first one has: base0, desktop0, dev0. Then after an update: base0, desktop0, dev0, base1, desktop1, dev1. When one has version 0 only delta layers from version 1 would be required.

Yup. You could have a base image that rarely changes, and a top-level image that you keep rebuilding that simply does apt-get update. Your target will see the only new layer is the apt-get update layer and will only download the deltas.

Oh. But I forgot about inheritance. So the update should be a single delta based on the last image. Squashing base0, desktop0 and dev0 to single image dev1 inheriting from dev0.

This could be better fleshed out, but it could be probably easier bolted on current implementation.

Yup, this all sounds do-able.

[pauldotknopf]

Regarding updates of images themselves. I meant for package update. Do you just have a cron job to build a new image every day and if there is a change you publish the new image? Or maybe you detect package upgrades and then automatically build new images?

I don't have a cronjob. My builds only trigger when I push an update to my recipes. After I push, I give it an hour to build, then I run update-dev-image and reboot.

[pauldotknopf]

Here are my update scripts.

Note: Only use this as a reference. It will definately not work with Darch, but will give you an idea on how to do it.

grub.cfg

load_env
source /efi/boot/load_boot_vars
source /efi/boot/check-boot-integrity
# We run load_boot_vars again because check-boot-integrity
# may have changed boot version (A or B), so we need new
# kernel/initrd paths.
source /efi/boot/load_boot_vars

if [ -e "$BOOT_ROOT_GRUB_CONF" ]; then
    source "$BOOT_ROOT_GRUB_CONF"
fi

default=evo4k
timeout=0
menuentry 'system'{
linux $BOOT_ROOT_KERNEL rootwait rootfstype=ext4 rootimage=$BOOT_ROOT_IMAGE_NAME root=/dev/disk/by-partuuid/9d69c3d4-4175-4a46-baba-64f95bcea861 console=ttyS0,115200 console=none intel_idle.max_cstate=1 quiet
initrd $BOOT_ROOT_INITRD
}

load_boot_vars

#!/bin/bash

if [ -z "$BOOT_VERSION" ]; then
    echo "BOOT_VERSION didn't exist, setting it to \"A\"."
    BOOT_VERSION="A"
fi

echo "BOOT_VERSION: $BOOT_VERSION"

if [ "$BOOT_VERSION" = "A" ]; then
    BOOT_ROOT_IMAGE_NAME=boot1/rootfs.img
    BOOT_ROOT_INITRD=($BOOT_ROOT_DRIVE)/boot1/initrd
    BOOT_ROOT_KERNEL=($BOOT_ROOT_DRIVE)/boot1/bzImage
    BOOT_ROOT_GRUB_CONF=($BOOT_ROOT_DRIVE)/boot1/grub-conf
else
    BOOT_ROOT_IMAGE_NAME=boot2/rootfs.img
    BOOT_ROOT_INITRD=($BOOT_ROOT_DRIVE)/boot2/initrd
    BOOT_ROOT_KERNEL=($BOOT_ROOT_DRIVE)/boot2/bzImage
    BOOT_ROOT_GRUB_CONF=($BOOT_ROOT_DRIVE)/boot2/grub-conf
fi

echo "BOOT_ROOT_IMAGE_NAME: $BOOT_ROOT_IMAGE_NAME"
echo "BOOT_ROOT_INITRD: $BOOT_ROOT_INITRD"
echo "BOOT_ROOT_KERNEL: $BOOT_ROOT_KERNEL"
echo "BOOT_ROOT_GRUB_CONF: $BOOT_ROOT_GRUB_CONF"

check-boot-integrity

#!/bin/bash

if [ -z "$BOOT_COUNT" ]; then
    echo "BOOT_COUNT didn't exist, setting it to \"0\"."
    BOOT_COUNT="0"
fi

if [ -z "$BOOT_VERIFIED" ]; then
    echo "BOOT_VERIFIED didn't exist, setting it to \"true\"."
    BOOT_VERIFIED="true"
fi

echo "BOOT_COUNT: $BOOT_COUNT"
echo "BOOT_VERIFIED: $BOOT_VERIFIED"

if [ "$BOOT_VERIFIED" != "true" ]; then
    echo "Boot hasn't been verified. Increasing boot counter."

    if [ "$BOOT_COUNT" == "0" ]; then
        BOOT_COUNT="1"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "1" ]; then
        BOOT_COUNT="2"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "2" ]; then
        BOOT_COUNT="3"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "3" ]; then
        BOOT_COUNT="4"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "4" ]; then
        BOOT_COUNT="5"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "5" ]; then
        BOOT_COUNT="6"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "6" ]; then
        BOOT_COUNT="7"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "7" ]; then
        BOOT_COUNT="8"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "8" ]; then
        BOOT_COUNT="9"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "9" ]; then
        BOOT_COUNT="10"
        save_env BOOT_COUNT
    elif [ "$BOOT_COUNT" == "10" ]; then
        echo "Reached 10 boots, falling back to other boot partition."
        if [ "$BOOT_VERSION" = "A" ]; then
            BOOT_VERSION="B"
            BOOT_VERIFIED="true"
            BOOT_COUNT="0"
        else
            BOOT_VERSION="A"
            BOOT_VERIFIED="true"
            BOOT_COUNT="0"
        fi
        # Use to append kernel parameter so that applications will not if this was first boot after we fell back
        # because of a failed update.
        BOOT_FELL_BACK="true"
        save_env BOOT_VERSION
        save_env BOOT_VERIFIED
        save_env BOOT_COUNT
    else
        echo "Invalid boot count $BOOT_COUNT will be reset to 0."
        BOOT_COUNT="0"
        save_env BOOT_COUNT
    fi
fi

updatecommit

This script is to be called on boot if you are certain the current boot is valid. If this script is not called, the update will be rolled back after 10 boots.

#!/bin/bash

# Load all the grub variables into bash session.
while read x; do
eval "$x"
done << EOF
$(grub-editenv /boot/EFI/BOOT/grubenv list)
EOF

if [ "$BOOT_VERIFIED" != "true" ]; then
    # Let grub know this boot was verified!
    grub-editenv /boot/EFI/BOOT/grubenv set BOOT_VERIFIED=true
    grub-editenv /boot/EFI/BOOT/grubenv set BOOT_COUNT=0
else
    echo "Already verified!"
fi

[pauldotknopf]

Some things to note about my scripts.

1. I'm pivoting on an initramfs/kernel/sqaush pair. You'd likely want to pivot between two Darch grub menu entries.
2. Grub doesn't support arithmatic, so I manually implemented the incrementing.

[pauldotknopf]

It would be nice if you could develop something that isn't tied to Darch, but could be used by Darch.

You could create a grub plugin that will support swapping between 2 menu entries. You'd also have a bash script that would support "persisting" a change between a menu entry 1 or 2.

[hadrianw]

Thanks for all the information! I will try to implement it in the new year. I have to experiment a bit with grub. I'll give you a tip when I will have something.