guide: formatting oracle cloud boot volumes as btrfs

[chayleaf]

Important things to keep in mind:

• As of now, you can't reattach a volume as a boot volume once you format it! Once you detach and format it, you can only create a new instance with that volume. If you do try to attach it, it will show "attaching" for a few minutes, then detach automatically.
• RHEL-based ARM distros have an old kernel, which has a different page size, meaning btrfs volumes which work there won't work on new kernels. Also, you can't pivot_root into Nix on their RHEL-based image, and chroot doesn't work in any case as it breaks Nix sandboxing which can't be disabled. Furthermore, their RHEL-based images use a complicated partition scheme with 3 partitions and LVM. Basically, use Ubuntu.
• Don't forget to set partition types and labels properly
• If your system fails in initrd, open your VM's cloud console to find out what the issue is
• You will need to use multiple VMs. Make sure you create all of them in the same AD, or they won't be able to interact with each other! They also must all have the same architecture as the VM you will end up using with the btrfs volume.

With that in mind, rough operation order:

1. Create a vm1 with Ubuntu
2. Create a second vm2 (Ubuntu as well)
3. Terminate vm2, keep its boot volume
4. Attach vm2 boot volume to vm1
5. Resize the vm2 efi partition by moving the ext partition further and recreating the efi partition (optional, the default of 100mb is just barely enough), btrfs-convert the vm2 root partition. Dont forget to label the new partitions properly (UEFI and cloud-root or something like that)
6. Chroot into vm2 boot volume, patch fstab to change ext4 into btrfs, reinstall grub (don't mistake the drive!). Now you are done with vm1, detach vm2 boot drive from it, if you don't think you'll need more troubleshooting just terminate it alongside its boot volume.
7. Create vm3 from vm2's boot drive. You should now have Ubuntu with btrfs - this will be your final instance.
8. Boot vm3, run nixos-infect. I had some config file issues which I didn't really feel like investigating, and by the time you reach this step you must be considerably tired too - so instead of running it as is, just copy the functions into your shell and run them one by one, in particular, check the generated drive configutation before running the actual infect script, and make sure the bootloader was successfully installed before rebooting.
9. Hopefully, nixos should now be installed on btrfs! I recommend setting a grub configuration limit, since you can't choose an old generation at boot either way, and your efi partition is fairly small

There's some leeway for errors (for example, the VM booted up even when I didn't mark the EFI partition type as EFI) but it still took me around 15 hours of trial and error to end up with this order

[ivan]

If you need to change the partitions before installing NixOS, why not emergency-kexec instead of nixos-infect? :)

(One might have to change the modules a bit, but you get an in-memory NixOS and all the block devices are writable.)

[kradalby]

• Resize the vm2 efi partition by moving the ext partition further and recreating the efi partition (optional, the default of 100mb is just barely enough), btrfs-convert the vm2 root partition. Dont forget to label the new partitions properly (UEFI and cloud-root or something like that)

Could you elaborate on this one? I end up with a disk that isnt bootable when I try to resize it. I am roughly following this: https://superuser.com/questions/1230741/how-to-resize-the-efi-system-partition

And I end up with the following diff from blkid:

Before (working):

/dev/sda1: UUID="B022-8CF1" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="c0954268-e4e2-409c-a9f4-de31d8b5e89a SEC_TYPE="msdos" 

After

/dev/sda1: UUID="FC00-0B91" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System" PARTUUID="281e1ab9-cf5d-4c9e-8438-4260c6b0f29d"

Any tips @pavlukivan?

[chayleaf]

Interesting, mine is:

/dev/sda15: LABEL_FATBOOT="UEFI" LABEL="UEFI" UUID="78FD-AD17" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="6adc9979-75db-a447-b6dd-7878d792f77b"

Perhaps they updated the Ubuntu image? Try using an older one if you're using the latest one, or try using a newer one if you're using an old one.

I can at least see your volume has a different PARTLABEL and no SEC_TYPE, maybe it has something to do with that. EFI is always a bit finicky, try making it be as similar to the old partition as possible I guess... @kradalby

[kradalby]

Thanks @pavlukivan, I actually managed yesterday, but I didn't have time to post an update. I'll try to remember when I get back, I have a nice set of commands now (I used them with the Oracle 8 image).

[chayleaf]

ah, that's why I recommended to use the Ubuntu image instead, it has a simpler partition table

[kradalby]

Understandable, I used the Oracle image because it had "almost" the layout I wanted, I just wanted to expand the ESP.

My working cloud-init file for Oracle Linux 8.5 to NixOS with a larger (1gb) ESP:

https://gist.github.com/kradalby/0957a5d4fbb9018d0b700b118f5c5341