Explore ways to use the persistent partition for data that shouldn't be "reset"

[jimmykarily]

Because of the limitations in size when booting UKI images, we are considering putting a big chunk of the data (e.g. big binaries, cached container images etc) on the persistent partition. The problem with this approach is that on reset, we format the persistent partition.

To work around this issue, we could use an additional partition where the data will also be written. After reset a cloud-config will be responsible to write the data back to the persistent partition.

An altetnative would be to use the additional partition as a non-resettable persistent partition and avoid any copies but this needs immucore to know about that partition and mount it properly on the final booted system. We could even use the oem partition for that (which doesn't get reset).

Let's try the first approach above which need less coding and if that doesn't work we consider the other options.

[jimmykarily]

Here is an example of how to copy the files after installation to the additional partition:

#cloud-config
users:
- name: kairos
  passwd: kairos

stages:
  after-install:
      - commands:
        - echo "Copying files to persistent path"
        - /usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
        - mount /dev/mapper/persistent /usr/local
        - mkdir /usr/local/.state/opt.bind
        - cp -rfv /run/initramfs/live/data/* /usr/local/.state/opt.bind

[jimmykarily]

I created a uki image with:

docker run --rm -v $PWD/data:/data -v $PWD/unpacked:/unpacked -v $PWD/build:/result -v $PWD/e2e/assets/keys/:/keys quay.io/kairos/osbuilder-tools:v0.200.5 build-uki dir:/unpacked --overlay-iso /data --output-dir /result --keys /keys --output-type iso --boot-branding "FedoraStandardMaster"

with one file in the overlay-iso directory:

~/workspace/kairos/enki (main)*$ cat data/data/this-should-never-reset 
This file should be here after reset

Then I installed with a config like this:

#cloud-config

install:
  device: "/dev/vda"
  auto: true
  partitions:
    oem:
      size: 4000
      fs: ext4

users:
- name: "kairos"
  passwd: "kairos"

stages:
  after-install:
      - commands:
        - echo "Copying files to oem and persistent"
        - /usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
        - /usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
        - mount /dev/mapper/persistent /usr/local
        - mount /dev/mapper/oem /oem
        - mkdir -p /usr/local/.state/opt.bind
        - mkdir -p /oem/opt.bind
        - cp -rfv /run/initramfs/live/data/* /usr/local/.state/opt.bind
        - cp -rfv /run/initramfs/live/data/* /oem/opt.bind
        - umount /dev/mapper/persistent
        - umount /dev/mapper/oem
        # Also re-encrypt the partitions?
  kairos-uki-reset.after:
      - commands:
        - |
          /bin/bash <<'EOF'
          #!/bin/bash

          set -e
          echo "Copying files from oem to persistent"
          # /oem was mounted in my tests. Let's umount it to be sure.
          umount /oem || true
          # Close all encrypted partitions
          for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
          /usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
          /usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
          mount /dev/mapper/persistent /usr/local
          mount /dev/mapper/oem /oem
          mkdir -p /usr/local/.state/opt.bind
          cp -rfv /oem/opt.bind/* /usr/local/.state/opt.bind
          umount /dev/mapper/persistent
          umount /dev/mapper/oem
          cryptsetup close /dev/mapper/persistent
          cryptsetup close /dev/mapper/oem

          EOF

I just can't get the stage for after reset right. This one doesn't trigger automatically but when I run the script manually, the file gets copied just fine. I just have to find the right "hook" to run the script.

[jimmykarily]

Strange, after-reset should work: https://github.com/kairos-io/kairos-agent/blob/0ae9c04eb4e1fe373d5f9d71baa94f8f8e4ef39b/internal/agent/reset.go#L87

[jimmykarily]

Probably the UkiResetAction needs some calls like this: https://github.com/kairos-io/kairos-agent/blob/0ae9c04eb4e1fe373d5f9d71baa94f8f8e4ef39b/pkg/action/reset.go#L234

Somewhere here? https://github.com/kairos-io/kairos-agent/blob/0ae9c04eb4e1fe373d5f9d71baa94f8f8e4ef39b/pkg/uki/reset.go#L83

[jimmykarily]

With this change in the kairos-agent, this config now works:

#cloud-config

install:
  device: "/dev/vda"
  auto: true
  partitions:
    oem:
      size: 4000
      fs: ext4

users:
- name: "kairos"
  passwd: "kairos"

stages:
  after-install:
      - commands:
        - echo "Copying files to oem and persistent"
        - /usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
        - /usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
        - mount /dev/mapper/persistent /usr/local
        - mount /dev/mapper/oem /oem
        - mkdir -p /usr/local/.state/opt.bind
        - mkdir -p /oem/opt.bind
        - cp -rfv /run/initramfs/live/data/* /usr/local/.state/opt.bind
        - cp -rfv /run/initramfs/live/data/* /oem/opt.bind
        - umount /dev/mapper/persistent
        - umount /dev/mapper/oem
        - cryptsetup close /dev/mapper/persistent
        - cryptsetup close /dev/mapper/oem

  after-reset:
      - commands:
        - |
          /bin/bash <<'EOF'
          #!/bin/bash

          set -e
          echo "Copying files from oem to persistent"
          # /oem was mounted in my tests. Let's umount it to be sure.
          umount /oem || true
          # Close all encrypted partitions
          for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
          /usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
          /usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
          mount /dev/mapper/persistent /usr/local
          mount /dev/mapper/oem /oem
          mkdir -p /usr/local/.state/opt.bind
          cp -rfv /oem/opt.bind/* /usr/local/.state/opt.bind
          umount /dev/mapper/persistent
          umount /dev/mapper/oem
          cryptsetup close /dev/mapper/persistent
          cryptsetup close /dev/mapper/oem

          EOF

[jimmykarily]

Fix is now on master, the snippet should work. I'll give it another test and then we can close it.

[mudler]

let's also update our docs with the snippet above, the Notes section looks a good candidate for now: https://kairos.io/docs/installation/trustedboot/#notes

[jimmykarily]

Docs merged, closing.