sysroot on lvm+btrfs - Githubissues

fredleb commented 4 years ago

I have the following setup:

/dev/sda2: ext4 on luks that contains key files to the sysXX partitions to be mapped to /dev/mapper/keys /dev/sda3: luks sys00 /dev/sdb1: luks sys01 /dev/sdc1: luks sys02 I want the user to only give the password at bootup for the "keys" partition. The cryptfstab looks like this: ``` keys UUID=ed67ffb1-4bdb-434a-b011-4c9147f05ad7 none luks sys00 UUID=ed5d6f31-8015-4904-8ca6-e4f0d952f025 /root/keys/luks00.keyfile luks sys01 UUID=f2a34e45-8bdd-433d-b73a-4dd3336be28a /root/keys/luks01.keyfile luks sys02 UUID=5f81c88b-a5c7-471e-981e-3b5b5b406ee6 /root/keys/luks02.keyfile luks ``` The fstab looks like this: ``` /dev/mapper/keys /root/keys auto x-systemd.device-timeout=9999h 0 1 UUID=52a7433c-3914-422c-8d71-c016badb8c81 /sysroot auto x-systemd.device-timeout=9999h,noatime 0 2 ```

I got that setup working before this project became FHS compliant with an ugly hack (a call to mount) in initrd-shell.sh.

That was evil.

Now I am trying to get it to work with systemv.

My idea is that I can achieve the same thing using a systemd service that looks like this:

``` [Unit] Description=Initrd Key Mounter Service ConditionPathExists=/etc/crypttab ConditionPathExists=/etc/fstab ConditionPathExists=/etc/initrd-release DefaultDependencies=no Before=initrd-sysroot-mount.service Requires=dev-mapper-keys.device [Service] Type=oneshot ExecStart=mount /dev/mapper/keys /root/keys [Install] WantedBy=sysinit.target ```

The problem however is that this fails to build with the following error

==> ERROR: unit not found: dev-mapper-keys.device

My question is:

is that a bug ?
or am I going in the wrong direction ?

Thanks

fredleb commented 4 years ago

That what I think I tried first but I could not find how to express it in a systemd way. If I add:

Requires=dev-mapper-root00vg-root.device
After=dev-mapper-root00vg-root.device
Requires=dev-mapper-root01vg-root.device
After=dev-mapper-root01vg-root.device
Requires=dev-mapper-root02vg-root.device
After=dev-mapper-root02vg-root.device

to sysroot.mount it's not happy:

==> ERROR: can not find service unit: dev-mapper-root00vg-root.device
==> ERROR: can not find service unit: dev-mapper-root01vg-root.device
==> ERROR: can not find service unit: dev-mapper-root02vg-root.device

On the booted system however I can see the device units (although with the escaped names... could that be the problem ?)

``` dev-disk-by\x2did-dm\x2dname\x2dkeys.device loaded active plugged /dev/disk/by-id/dm-name-keys > dev-disk-by\x2did-dm\x2dname\x2droot00vg\x2droot.device loaded active plugged /dev/disk/by-id/dm-name-root00vg-root > dev-disk-by\x2did-dm\x2dname\x2droot01vg\x2droot.device loaded active plugged /dev/disk/by-id/dm-name-root01vg-root > dev-disk-by\x2did-dm\x2dname\x2droot02vg\x2droot.device loaded active plugged /dev/disk/by-id/dm-name-root02vg-root > dev-disk-by\x2did-dm\x2dname\x2dsys00.device loaded active plugged /dev/disk/by-id/dm-name-sys00 > dev-disk-by\x2did-dm\x2dname\x2dsys01.device loaded active plugged /dev/disk/by-id/dm-name-sys01 > dev-disk-by\x2did-dm\x2dname\x2dsys02.device loaded active plugged /dev/disk/by-id/dm-name-sys02 ```

Andrei-Pozolotin commented 4 years ago

forward dependency is a mistake (device units are transient, there are no devices at mkinitcpio build time):

Requires=dev-mapper-root00vg-root.device
After=dev-mapper-root00vg-root.device

have you tried reverse dependency?

fredleb commented 4 years ago

How ?

Andrei-Pozolotin commented 4 years ago

like so:

mkdir -p dev-mapper-root00vg-root.device.d
echo ""  > override.conf
echo "[Install]"  >> override.conf
echo "RequiredBy: ... "  >> override.conf

Andrei-Pozolotin commented 4 years ago

should be possible to say it once, like dev-mapper-@.device.d not sure of syntax

fredleb commented 4 years ago

done but the directories and files are not being pulled in the initramfs... I guess I need to override an already included service too and add them via an InitrdPath

Andrei-Pozolotin commented 4 years ago

or have some fake service do [Install]/WantedBy: them basically all these hacks just to have system up, will "make it right" later

Andrei-Pozolotin commented 4 years ago

meanwhile, next idea to bypass btrfs fsid uuid:

here it basically says https://unix.stackexchange.com/questions/120907/arch-does-not-mount-btrfs-array-on-boot

that this is a "blocking call": ExecStart=/usr/bin/btrfs device scan

then we could inject more dependencies:

btrfs-device-scan.service -> sysroot.mount
crypttab lvm-group -> lvm-root scan -> btrfs-device-scan.service

fredleb commented 4 years ago

On the virtual machine it looks better, I'll try on the real machine some time this week. Thanks, enjoy your week-end !

fredleb commented 4 years ago

Hi.

I have noticed a couple of things that you might be interested in:

using arch-chroot, when building the image
- I get a note: "replacing initramfs unit file: /usr/lib/systemd/system/initrd-debug-progs.service" (but only that one)
- crypttab is not copied in the image.
using systemd-nspawn with exactly the same setup, crypttab is copied to the image but then I get two (correct) errors that were not showing up using arch-chroot:
- "==> ERROR: can not find service unit: root-keys.mount" (which is true because the correct name is different)
- "==> ERROR: command failure (1): dropbearconvert openssh dropbear /etc/ssh/ssh_host_ecdsa_key /etc/dropbear/dropbear_ecdsa_host_key" (the file is indeed missing)
- I get the other notes "replacing xxx" as expected

I am not going to spend any time on that, I'm just giving you the info.

Andrei-Pozolotin commented 4 years ago

I'm just giving you the info

thank you, please keep doing that

using arch-chroot ... crypttab is not copied in the image

we will ignore chroot issues untill someone makes a real case for it there are in fact 2 different paths for "bad/chroot" vs "good/systemd" container environments, as seen here: plain_unit_concat() vs smart_unit_concat() https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/src/mkinitcpio-install.sh

ERROR: command failure (1): dropbearconvert ... the file is indeed missing

that is a well known behavior "feature" of dropbear: https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/src/initrd-dropbear.service

# note:
# dropbear-convert needs host keys in pem format
# to regenerate host keys use: `ssh-keygen -A -m PEM`

Anty0 commented 4 years ago

Hi @Andrei-Pozolotin, I have managed to find some time to look into my issue on my own finally. :slightly_smiling_face:

Looking though systemd-analyze plot I have noticed initrd-cleanup.service stops sysroot.mount executing systemctl --no-block isolate initrd-switch-root.target. Since initrd-switch-root.target wants initrd-root-fs.target I would expect it would to keep sysroot.mount started. :thinking:

I tryied to fix it and I was able to fix (workaround?) my issue by adding x-systemd.wanted-by=initrd-root-fs.target option to /sysroot in /etc/mkinitcpio-systemd-tool/config/fstab. Hope this info can help also someone else. :slightly_smiling_face:

My current (working) setup:

- Partitioning: `Full disk luks2 -> LVM -> {root,var,home}` - `lsblk -f` ``` NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT sda └─sda1 crypto_LUKS 2 d4844303-338e-4ed2-9110-26b170116828 └─dataluks LVM2_member LVM2 001 809nvj-w92D-0uLd-Ct6P-eOjG-wGVv-ItPlRC └─vgdata-data ext4 1.0 453bfe4c-7c10-46d4-8d5e-50339ecefddb 970,7G 68% /mnt/data sr0 zram0 [SWAP] nvme1n1 └─nvme1n1p1 crypto_LUKS 2 18ceaf63-f1d8-4c75-b60a-0157ef155384 └─antyluks LVM2_member LVM2 001 VONNVM-HceN-StGb-a43h-AtO6-Fcgi-lyUPu0 ├─vgmain-root ext4 1.0 36737474-7501-4d2c-8eed-9fd34dd5d6ce 88,9G 32% / ├─vgmain-var ext4 1.0 2150fa6e-3183-40af-a4b4-9c8209f7ac53 93,4G 29% /var └─vgmain-home ext4 1.0 8eb2ab27-d2ae-47bb-a73a-1c4a38c2bc04 1,1T 23% /home nvme0n1 # (Windows drive) ├─nvme0n1p1 ntfs Recovery D6366CA6366C8977 ├─nvme0n1p2 vfat FAT32 566D-6F12 9,1M 91% /esp ├─nvme0n1p3 ├─nvme0n1p4 ntfs 82DE7398DE7382E9 └─nvme0n1p5 ntfs Data 24641664641638CE ``` - `/etc/mkinitcpio-systemd-tool/config/fstab` ``` diff 23a24,26 > > # /dev/mapper/vgmain-root > UUID=36737474-7501-4d2c-8eed-9fd34dd5d6ce /sysroot ext4 ro,noatime,async,x-systemd.device-timeout=9999h,x-systemd.wanted-by=initrd-root-fs.target 0 1 ``` - `/usr/lib/systemd/system/initrd-cryptsetup.service` ``` 52a53 > InitrdBinary=/usr/bin/cryptsetup ``` - Kernel cmd line: `rd.luks.name=18ceaf63-f1d8-4c75-b60a-0157ef155384=antyluks scsi_mod.use_blk_mq=1 fbcon=scrollback:1024k sysrq_always_enabled=1` - Enabled initrd services: `initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service initrd-debug-progs.service`

Andrei-Pozolotin commented 4 years ago

@Anty0 thank you for the update

can you please document your case as "consistent story" on wiki https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-LVM

with step-by-step instructions which are easier to follow for a new user

fredleb commented 4 years ago

Tested on system and it works. Do you want me to write a wiki page about it? My setup is quite complex it seems...Where should I do that?

Sent from Yahoo Mail on Android

On Fri, 10 Apr 2020 at 16:23, Andrei Pozolotinnotifications@github.com wrote:

@Anty0 thank you for the update

can you please document your case as "consistent story" on wiki https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-LVM

with step-by-step instructions which are easier to follow for a new user

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Andrei-Pozolotin commented 4 years ago

@fredleb

Do you want me to write a wiki page about it?

yes, please: https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-Btrfs

again, with step-by-step instructions which are easier to follow for a new user

Andrei-Pozolotin commented 4 years ago

@Anty0 @fredleb : if you guys think "lvm support" can be generalized for others in a form of a initrd-lvm-xxx.{mount,service} units, please share your ideas

fredleb commented 4 years ago

Hi, Not sure how it would be possible... (without adding stuff to systemd itself)

For my setup I am using a bit of luck : it might not work with a big number of devices.Each LVM LV that is found and attached in /dev/mapper fulfills a "RequiredBy" of sysroot.mount and that's good enough for 3 devices, but in reality an extra step is taking place between the two that is needed: the discovery of the BTRFS FS on the LVM LV... Since the BTRFS UUID is shared among devices, I can't really use that UUID to trigger anything... Unless there were a way to say that a device is required 3 times (which would mean hard coding a number somewhere and I would not like that)... I don't think this is workeable...

Another way of doing would be to have a the LVM attachement trigger a service that checks if the BTRFS raid is mountable. That means it would fail the first time, the second time too (although it would be mountable in degraded mode) but succeed the third time (or xth time, where x is the number of devices taking part into that raid)... I tried that but I could not get sysroot.mount to wait for anything... It was ignoring any After or Requires and was mounting as soon as the What condition was workeable... not sure if it wanted or a bug... Also: there is no tool to tell you if a volume is mountable or not... 2 possibilities : really try to mount it, or parse the output of "BTRFS device info xxxx" for "missing device"... both not nice.For this one however I thought that a cleaner solution would be to have a service indeed check for the BTRFS mountability of sysroot and then simply add a new simlink in /dev/mapper/mountable_sysroot (via udev ?) and have sysroot.mount use this path as its What condition. I did not try it... I'm not sure there is a way to fool udev into doing that somehow... I'll start writing the doc tomorrow. Cheers,Fred

On Friday, 10 April 2020, 17:53:50 CEST, Andrei Pozolotin notifications@github.com wrote:

@Anty0 @fredleb : if you guys think "lvm support" can be generalized for others in a form of a initrd-lvm-xxx.{mount,service} units, please share your ideas

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Anty0 commented 4 years ago

can you please document your case as "consistent story" on wiki

Sure, I'll try. :) In the end It'll probably need some language fixes, as my english isn't really great, but I'll do my best.

Anty0 commented 4 years ago

if you guys think "lvm support" can be generalized for others in a form of a initrd-lvm-xxx.{mount,service} units, please share your ideas

I'm not sure right now, but I'll try to look into creating something like this.

fredleb commented 4 years ago

V0.1 of doc is ready but I have no idea how to upload it...

Andrei-Pozolotin commented 4 years ago

I enabled public-edit on wiki, this should work now https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case%3A-Sysroot-on-Btrfs/_edit

fredleb commented 4 years ago

Well, it's gonna have to wait a bit: it's broken again... more on this when I find the problem...

fredleb commented 4 years ago

The new problem is that the root password reenabling done by initrd-dropbear.service is overwritten by the initrd-shell.service. I mean initrd-shell.service overwrite /etc/shadow but does not enable the root password...

fredleb commented 4 years ago

Solved with a drop-in to initrd-shell.service that calls the enable-root-login again. But that's just luck I guess: if any unit comes after that and overwrites the /etc/shadow again, it will be broken again.

Andrei-Pozolotin commented 4 years ago

Solved with a drop-in

so, would you rather have:

move this snippet initrd-dropbear.service -> initrd-shell.service

#[X-SystemdTool]
#InitrdBuild=/usr/lib/mkinitcpio-systemd-tool/initrd-build.sh command=do_root_login_enable

document that one needs 2 overrides: initrd-shell.service: activate

#[X-SystemdTool]
#InitrdBuild=/usr/lib/mkinitcpio-systemd-tool/initrd-build.sh command=do_root_login_enable

initrd-dropbear.service: activate

#[Service] 
#ExecStart=
#ExecStart=/bin/dropbear -j -k -F -p ${SSHD_PORT}

fredleb commented 4 years ago

So I have something that works ok. I have written doc for others to make it better: https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-Btrfs

fredleb commented 4 years ago

Solved with a drop-in

so, would you rather have:

1. move this snippet `initrd-dropbear.service` -> `initrd-shell.service`

#[X-SystemdTool]
#InitrdBuild=/usr/lib/mkinitcpio-systemd-tool/initrd-build.sh command=do_root_login_enable

1. document that one needs 2 overrides:
   `initrd-shell.service`: activate

#[X-SystemdTool]
#InitrdBuild=/usr/lib/mkinitcpio-systemd-tool/initrd-build.sh command=do_root_login_enable

initrd-dropbear.service: activate

#[Service] 
#ExecStart=
#ExecStart=/bin/dropbear -j -k -F -p ${SSHD_PORT}

Well if my drop-in is not only luck, it is ifine then. But otherwise I don't really know. Obviously the least intrusive solution is the best.

Creating a copy of initrd-shell.service in /etc is a bit of a pain when the next update comes...

Andrei-Pozolotin commented 4 years ago

@fredleb Frederic:

a. thank you for the magnum opus, "whole story" sounds much more impressive :-)

b. it seems there is still one (original?) problem: lack of explicit dependency on btrfs volume discovery / scan completion https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Device_scanning

for example, have you tested/concidered the following dependency pattern:

/etc/systemd/system/dev-mapper-rootXXvg-root.device.d/override.conf

[Install]
#   vvv
RequiredBy=initrd-btrfs-scan.service

initrd-btrfs-scan.service

[Unit]
Description=Initrd btrfs scan
#   vvv
Before=sysroot.mount
DefaultDependencies=false

[Service]
Type=oneshot
RemainAfterExit=true
#   vvv
ExecStart=/usr/bin/btrfs device scan

[Install]
RequiredBy=sysroot.mount

/etc/systemd/system/sysroot.mount

[Unit]
Requires=root-keys-source.mount
After=root-keys-source.mount
#   vvv
After=initrd-btrfs-scan.service
Before=initrd-root-fs.target
ConditionPathExists=/etc/initrd-release
DefaultDependencies=false

[Mount]
What=/dev/disk/by-uuid/c081646d-1234-1234-1234-2d821fbea470
Where=/sysroot
Type=btrfs
Options=noatime,subvol=/@

[Install]
WantedBy=initrd-root-fs.target

random-archer / mkinitcpio-systemd-tool

sysroot on lvm+btrfs #36