sysroot on lvm+btrfs

[fredleb]

I have the following setup:

/dev/sda2: ext4 on luks that contains key files to the sysXX partitions to be mapped to /dev/mapper/keys /dev/sda3: luks sys00 /dev/sdb1: luks sys01 /dev/sdc1: luks sys02 I want the user to only give the password at bootup for the "keys" partition. The cryptfstab looks like this: ``` keys UUID=ed67ffb1-4bdb-434a-b011-4c9147f05ad7 none luks sys00 UUID=ed5d6f31-8015-4904-8ca6-e4f0d952f025 /root/keys/luks00.keyfile luks sys01 UUID=f2a34e45-8bdd-433d-b73a-4dd3336be28a /root/keys/luks01.keyfile luks sys02 UUID=5f81c88b-a5c7-471e-981e-3b5b5b406ee6 /root/keys/luks02.keyfile luks ``` The fstab looks like this: ``` /dev/mapper/keys /root/keys auto x-systemd.device-timeout=9999h 0 1 UUID=52a7433c-3914-422c-8d71-c016badb8c81 /sysroot auto x-systemd.device-timeout=9999h,noatime 0 2 ```

I got that setup working before this project became FHS compliant with an ugly hack (a call to mount) in initrd-shell.sh.

That was evil.

Now I am trying to get it to work with systemv.

My idea is that I can achieve the same thing using a systemd service that looks like this:

``` [Unit] Description=Initrd Key Mounter Service ConditionPathExists=/etc/crypttab ConditionPathExists=/etc/fstab ConditionPathExists=/etc/initrd-release DefaultDependencies=no Before=initrd-sysroot-mount.service Requires=dev-mapper-keys.device [Service] Type=oneshot ExecStart=mount /dev/mapper/keys /root/keys [Install] WantedBy=sysinit.target ```

The problem however is that this fails to build with the following error

==> ERROR: unit not found: dev-mapper-keys.device

My question is:

• is that a bug ?
• or am I going in the wrong direction ?

Thanks

[Andrei-Pozolotin]

> is that a bug ? no: ``` Requires=dev-mapper-keys.device ``` will result in ``` ERROR: unit not found: dev-mapper-keys.device ``` since it is looking for actual unit file in `/usr/lib/systemd/system/dev-mapper-keys.device`, etc. > am I going in the wrong direction ? yes, let me think,

[Andrei-Pozolotin]

please try the following and report back:

**`[initramfs]/etc/crypttab`** ``` # root UUID=52a7433c-3914-422c-8d71-c016badb8c81 none luks keys UUID=ed67ffb1-4bdb-434a-b011-4c9147f05ad7 none luks,nofail ``` **`[initramfs]/etc/fstab`** ``` # /dev/mapper/root /sysroot auto x-systemd.device-timeout=9999h 0 1 /dev/mapper/keys /sysroot/root/keys auto x-systemd.device-timeout=9999h 0 2 ``` **`[real-root]/etc/crypttab`** ``` # sys00 UUID=ed5d6f31-8015-4904-8ca6-e4f0d952f025 /root/keys/luks00.keyfile luks,nofail ``` **`[real-root]/etc/fstab`** ``` # /dev/mapper/root / auto defaults,nofail,noatime 0 1 /dev/mapper/sys00 /mnt/sys00 auto defaults,nofail,noatime 0 2 ``` **add to `initrd-cryptsetup.service`** ``` # provide folder for /root/keys mount InitrdPath=/sysroot/root/keys/ create=yes ``` **add to `[real-root]`** ``` mkdir -p /root/keys mkdir -p /mnt/sys00 ```

[fredleb]

Thanks for your answer.

I haven't tried yet but I'm pretty sure it won't work like that: I left out a couple of details thinking it would make things simpler but...

``` NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT loop0 squashfs 4.0 sda |-sda1 |-sda2 ext4 1.0 d51deb1b-b348-4d9c-8a0c-ed256a73c7c7 303.8M 30% /boot `-sda3 crypto_LUKS 1 ed5d6f31-8015-4904-8ca6-e4f0d952f025 `-sys00 LVM2_member LVM2 001 CtPzrD-XXVK-D8B7-NJcO-ulhL-TCaR-RTuEo3 `-root00vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 sdb |-sdb1 |-sdb2 ext4 1.0 e0012f06-307c-46f6-be62-4f43bbaebbf8 /root/keys_clear `-sdb3 crypto_LUKS 1 f2a34e45-8bdd-433d-b73a-4dd3336be28a `-sys01 LVM2_member LVM2 001 opvZYc-TYWt-Up6S-nBlW-KrKE-0cUc-QeiFWu `-root01vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 sdc |-sdc1 |-sdc2 `-sdc3 crypto_LUKS 1 5f81c88b-a5c7-471e-981e-3b5b5b406ee6 `-sys02 LVM2_member LVM2 001 yLpKvu-9SjM-dQWh-5zMZ-hXzD-dcwv-95vyjZ `-root02vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 10.5G 15% / sdd `-sdd1 crypto_LUKS 1 ed67ffb1-4bdb-434a-b011-4c9147f05ad7 `-keys00 ext4 1.0 619600b7-aabd-4987-8b53-249f1aac10a3 /root/keys sr0 iso9660 ARCH_202003 2020-03-01-09-32-57-00 ```

Now this is the real thing: the root filesystem is inside an lvm container...

(/root/keys_clear is just a unencrypted copy of /keys with extra scripts to unlock and mount all volumes so I can arch-chroot in the system from the arch live CD) This used to work in a virtual machine on a beefy system but only thanks to an explicit "mount" of the /dev/mapper/keys to /keys in initrd-shell.sh, immediatly after the keys volume is decrypted. On the real system (with the ugly hack) it was successfully mounting the "keys" volume and decrypting the sysXX volumes but because not all 3 LVM volume were scanned fast enough to make all 3 BTRFS (in RAID 5) devices available, the BTRFS was being found unreadable when the mount /sysroot was attempted... I would have tried again to find a solution by myself but really I am stuck now.

I have basically 2 problems to solve here:

• decrypt via password input and mount "keys" to decrypt by keyfile several system volumes
• wait for all LVM devices taking part in "sysroot" to be available before attempting to mount it (but that probably deserves its own ticket)

But maybe I am missing a point so I will try your solution later today.

Oh yeah and to make things simple: this is a headless remote server of course... so I need dropbear to give the password to "keys" (but that's not an issue)...

[Andrei-Pozolotin]

Now this is the real thing: the root filesystem is inside an lvm

oh, man, you are cheating with your symptoms :-)

1. ok, then your setup is conceptually similar to "sysroot-on-overlay" https://github.com/jdub/openwrt-systemd/tree/master/systemd/files/etc/systemd/system

2. so next thing to try is to provide your own sysroot.mount and root-keys.mount with inter-dependency on lvm2-pvscan.service or whatever gets your array ready https://git.archlinux.org/svntogit/packages.git/tree/trunk/sd-lvm2_install?h=packages/lvm2

3. remember Mount units may either be configured via unit files, or via /etc/fstab https://www.freedesktop.org/software/systemd/man/systemd.mount.html

4. finally, try this in vm first

[fredleb]

Phase 1: success !

It took me a while to figure out a couple of things but I got working the first phase.

So what works:

• One password opens the luks encrypted partition (from here on "keys") containing key files to system luks encrypted partitions (from here on "sysXX") for the root filesystem that are opened automagically from the initramfs.

My [initramfs]/etc/crypttab (really /etc/mkinitcpio-systemd-tool/config/crypttab) looks like this:

``` keys UUID=ed67ffb1-4bdb-434a-b011-4c9147f05ad7 none luks sys00 UUID=ed5d6f31-8015-4904-8ca6-e4f0d952f025 /root/keys/luks00.keyfile luks sys01 UUID=f2a34e45-8bdd-433d-b73a-4dd3336be28a /root/keys/luks01.keyfile luks sys02 UUID=5f81c88b-a5c7-471e-981e-3b5b5b406ee6 /root/keys/luks02.keyfile luks ``` - "keys" is mounted to [initramfs]/root/keys. This is done by having an enabled explicit mount unit (the name **MUST** be the same as the "Where" path)

root-keys.mount:

``` [Unit] Before=sysroot.mount ConditionPathExists=/etc/crypttab ConditionPathExists=/etc/initrd-release DefaultDependencies=false Requires=initrd-cryptsetup.service [Mount] What=/dev/disk/by-uuid/e0012f06-307c-46f6-be62-4f43bbaebbf8 Where=/root/keys Type=ext4 Options=noatime,x-systemd.device-timeout=9999h [Install] WantedBy=sysinit.target [X-SystemdTool] InitrdPath=/root/keys/ create=yes ``` This unit draws initrd-cryptsetup.service and /etc/crypttab in the initramfs.

• Then the sysXX luks containers are opened and the root fs (BTRFS raid 5) is mounted at /sysroot by another enabled explicit mount unit

sysroot.mount:

``` [Unit] Requires=root-keys.mount Before=initrd-root-fs.target After=root-keys.mount ConditionPathExists=/etc/crypttab ConditionPathExists=/etc/initrd-release DefaultDependencies=false [Mount] What=/dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81 Where=/sysroot Type=btrfs Options=noatime,x-systemd.device-timeout=9999h [Install] WantedBy=initrd-root-fs.target ```

Note: because these are the 2 file systems that I need to mount to boot, and they are mounted via explicit mount units, my [initramfs]/etc/fstab (really /etc/mkinitcpio-systemd-tool/config/fstab) is now empty.

And taaadaaa, it boots.

In the next comment, I will write what does not work and a probably unrelated failure that I can see.

[fredleb]

I see a problem after I enter the password

but I guess this is an arch-linux problem and not related to this tool.  But it does not seem to do much damage because the system boots anyway. So... I'll ignore that for now. Feel free to comment though !

[Andrei-Pozolotin]

switch to console, journalctl -b ?

[fredleb]

Nothing in it... weird...

[Andrei-Pozolotin]

looks like still timing problem, where is lvm array dependency in the units?

[fredleb]

Yeap... I'm looking into how to solve that now...

[Andrei-Pozolotin]

this lvm dependency issue is already solved here:

https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_LVM ``` If the root file system resides in a logical volume, the root= kernel parameter must be pointed to the mapped device, e.g /dev/vg-name/lv-name. ``` need to reverse engineer what udev rules produce and how they activate `lvm2-pvscan.service` https://git.archlinux.org/svntogit/packages.git/tree/trunk/sd-lvm2_install?h=packages/lvm2 sounds like you need one more inter-mount-unit for the `/dev/vg-name/lv-name` :-)

[fredleb]

Hmmm... several problems with that path...

- I don't use the root= kernel parameter (because it could cause the mounting to timeout when the user does not give the password fast enough or when using dropbear (my case)) - My raid 5 is not managed by LVM but by the BTRFS filesystem itself. LVM sees 3 independant logical devices but BTRFS puts them together to find its files. So I need `/dev/mapper/vg00root` and `/dev/mapper/vg01root` and `/dev/mapper/vg01root` ready before I can mount the /sysroot... One easy way would be of course to write a bash script that just waits with a loop for all 3 devices to be visible and have it in a service unit before sysroot.mount... But then I have to maintain a list or hard code stuff in it... I am afraid that when one disk needs replacement, I will have to get in there again and do ugly things...

[Andrei-Pozolotin]

> I don't use the root= kernel parameter you do: it is called `/sysroot` in `fstab` or `sysroot.mount` >easy way would be of course to write a bash script you already had bash script, now is a chance to find "the right way" i.e. better understand how dependency on `lvm2-pvscan.service` works

[fredleb]

Is it evil to have a service unit polling ?

BTRFS tools allow me to know if a mountable FS is missing any device... I could use that to avoid hard coding any thing by simply polling it until it says it foudn all devices...

[Andrei-Pozolotin]

Is it evil to have a service unit polling ?

no, we already use similar hacks:

https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-network.service#L18 ``` #### required to recover from emergency service ExecStart=/bin/sh -c "echo '%N: enable network devices'" ExecStart=/bin/sh -c "for dev in $(ls /sys/class/net) ; do ip link set $dev up ; done" ```

[Andrei-Pozolotin]

but @fredleb Frederic, remember: you have one more lvm client: @Anty0 :-)

[fredleb]

Yeah, right.

Plus: - I find polling still half evil - My solution would involve parsing the output of `btrfs filesystem show /sysroot` and that's another evil too... So I will look into more LVM stuff. Thanks for your help. Let me know if I need to give more details for documentation purpose.

[fredleb]

Now I am lost again. I have modified my sysroot.mount to:

sysroot.mount

``` [Unit] #Requires=root-keys.mount #After=root-keys.mount Before=initrd-root-fs.target ConditionPathExists=/etc/crypttab ConditionPathExists=/etc/initrd-release DefaultDependencies=false Requires=wait-for-sysroot.service After=wait-for-sysroot.service [Mount] What=/dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81 Where=/sysroot Type=btrfs Options=noatime,x-systemd.device-timeout=9999h [Install] WantedBy=initrd-root-fs.target ```

And added a service wait-for-sysroot.service that I hope will wait until all crypto stuff is done and then make sure all LVM stuff is activated:

wait-for-sysroot.service

``` [Unit] Description=LVM activation for sysroot ConditionPathExists=/etc/initrd-release DefaultDependencies=no #Requires=initrd-root-device.target #After=initrd-root-device.target #Requires=root-keys.mount #After=root-keys.mount Requires=initrd-cryptsetup.service After=initrd-cryptsetup.service Before=sysroot.mount [Service] Type=oneshot ExecStart=pvscan --cache -aay [Install] #RequiredBy=sysroot.mount <- this is ignored by mkinitcpio #WantedBy=initrd-root-fs.target #WantedBy=initrd-root-device.target WantedBy=sysinit.target ```

Notes:

• I have tried all the commented combinations and every time the sysroot.mount is done BEFORE this service
• probably bug or missing feature for mkinitcpio-systemd-tool: [Install] RequiredBy=some.mount is ignored by mkinitcpio.

The above combination gives:


[Andrei-Pozolotin]

with all the stuff that is commented out, it is hard to match the boot graph to the actual config state at the time, so I guess you will want to experiment more and produce set of pairs config -> graph :-)

meanwhile, general observations:

probably bug : [Install] RequiredBy=some.mount is ignored

what do you mean, exactly (1, 2a, 2b)?

1. when `[Install]` section is processed by `systemctl enable` it is translated into bunch of symlinks inside `/etc/systemd/system` 2. when `mkinitcpio-hook` runs, it supposed to 2a. replicate symlinks inside initrd 2b. pull dependencies of enabled units into initrd

remember: you can look / navigate inside /boot/initramfs-linux.img

after `mkinitcpio` but before `reboot`, say by using: * visually [mc](https://www.archlinux.org/packages/community/x86_64/mc/) * or scripted [lsinitcpio](https://wiki.archlinux.org/index.php/Mkinitcpio#Extracting_the_image) so then please take a look: does it all "look good": * expected units/binaries are present * expected configs are present / enabled / linked * etc, etc

RequiredBy=sysroot.mount <- this is ignored by mkinitcpio

every XxxxBy entry is a reverse dependency, which can be expressed as an override on the opposite side, use it to "fix the bug"

https://www.freedesktop.org/software/systemd/man/systemd.unit.html "Forward" property | "Reverse" property | Where used -- | -- | -- Before= | After= | [Unit] section After= | Before= Requires= | RequiredBy= | [Unit] section | [Install] section Wants= | WantedBy= | [Unit] section | [Install] section PartOf= | ConsistsOf= | [Unit] section | an automatic property BindsTo= | BoundBy= | [Unit] section | an automatic property Requisite= | RequisiteOf= | [Unit] section | an automatic property Triggers= | TriggeredBy= | Automatic properties, see notes below Conflicts= | ConflictedBy= | [Unit] section | an automatic property PropagatesReloadTo= | ReloadPropagatedFrom= | [Unit] section ReloadPropagatedFrom= | PropagatesReloadTo= Following= | n/a | An automatic property |  

The above combination gives:

if you literally mean (all the units with commented out stuff -> graph in the previous comment) then the config and the graph do match as expected :-)

Now I am lost again.

please don't be! :-) it is apparent by now that more patience will be required

you may also want to play with stronger dependency via BindsTo, similar to: initrd-sysroot-mount.service

[fredleb]

I can confirm that wait-for-sysroot.serivce containing a "Install/RequiredBy=sysroot.mount" creates a dir "sysroot.mount.requires" with a proper symlink to the service when enabled but that the mkinitcpio run is NOT copying the dir and link to the initramfs.

I check if I can fix it. Or is it wanted that way ?

[Andrei-Pozolotin]

please:

1. provide absolute paths
2. in both initramfs and real-root
3. check all locations: /etc /run /usr/lib

[fredleb]

I created wait-for-sysroot.service in [real]/etc/systemd/system with:

[Install]
RequiredBy=sysroot.mount

Enabling it I see a [real]/etc/systemd/system/sysroot.mount.requires as expected with a symlink to my service in it. after a daemon-reload (not sure I need that) and mkinitcpio -p linux, the directory and the link are missing in [initramfs]/etc/systemd/system. ( The .wants are there as expected) I check in [initramfs]/usr/lib too and there is nothing (that would have been weird). no /run in initramfs: I checked from inside the real system...

Or did you mean something else ?

[Andrei-Pozolotin]

I see. I am making a unit test for that https://github.com/random-archer/mkinitcpio-systemd-tool/tree/master/tool/image

[Andrei-Pozolotin]

just noticed: ExecStart=pvscan --cache -aay should fail, need abs path

[Andrei-Pozolotin]

@fredleb please review: #43

[fredleb]

I have tried with forward and reverse dependency to sysroot.mount and in both cases I get the exact opposite of what I want... I strongly suspect systemd itself now...

Is there anyway to force systemd:

• to trace the exact path of the unit it executes (in case it overrides mine maybe ?)
• to force systemd to keep the generated units ?
• to have some kind of dry run (so I don't have to rescue my system so often) ?

Is there any difference between the systemd executable for initramfs and for the real system ?

I'm gonna get a beer now.

[Andrei-Pozolotin]

I strongly suspect systemd itself now...

that is by design, next stage is melancholy - see systemd as tragedy ;-)

I get the exact opposite of what I want...

so the solution then is simple: configure units to the exact opposite to what you have now :-)

to trace the exact path of the unit it executes

• enable logging, collect humongous log dump, then grep them

• systemd-analyze dot with filtering, though it can not do much analyze.c

force systemd to keep the generated units

you mean, like transient mounts? there is a systemd dbus api which can receive triggered signals with unit state snapshot, but I do not know any user-ready tools that expose that

to have some kind of dry run

sure, just run in QEMU or VirtualBox

it is hard to debug initramfs-linux.img in a "simple container" like docker or systemd-nspawn, since the is no cgroups virtualization for udevd and you need real hardware emulation/virtualization

difference between the systemd executable for initramfs and for the real system

no: extract and compare

initramfs-linux.img/init   vs    1570680 Mar 27 14:44 /usr/lib/systemd/systemd

I'm gonna get a beer now.

me too!

in summary:

1. basically, available tools (systemd.log_level, systemd analyze) are fine when you go from a working state to a broken one, so you can compare before and after

2. they are useless when you start in a broken state, when you are swamped with deluge of mindless details

3. what worked in the past was to fire up a vm with simple config and then keep building it up until it breaks

remember: there is nothing wrong with systemd, it is just a tragedy

[fredleb]

So I tried a couple of things and now that my virtual machine is stuck and won't boot, I think that I will need to have a look at the systemd code. Somehow, I have the feeling that mount units disregard Before and Requires constraints as soon as the device in What becomes available. If that's the case, I guess it's a bug ? I am giving up for today...

[Andrei-Pozolotin]

I have the feeling that mount units disregard

1. over last 5 years that changed a couple of times in systemd, I do not bother "to understand" anymore, I just look for workarounds

2. take a look on the story around #10, #12, initrd-sysroot-mount.service to see if that rings a bell with what you feel

3. here is systemd-analyze from a small working test system, based on recommended activation

``` systemctl enable initrd-cryptsetup.path systemctl enable initrd-tinysshd.service systemctl enable initrd-debug-progs.service systemctl enable initrd-sysroot-mount.service ``` with `systemd-analyze` legend: ``` Color legend: black = Requires dark blue = Requisite dark grey = Wants red = Conflicts green = After ``` **`systemd-analyze dot --order initrd* | dot -T png > systemd-order.png`**  **`systemd-analyze dot --require initrd* | dot -T png > systemd-require.png`**  **`systemd-analyze dot initrd* | dot -T png > systemd-total.png`** 

where both dot --order and dot --require sort of make sense, individually, but dot <total> - not so much

one reason could be that -.mount and sysroot.mount are in fact, same entity, just on different sides of switch-root

4. if you take a look on dracut bootup it seems they are performing a hackerty in rootfs-generator.sh similar to initrd-sysroot-mount.service for use by 98dracut-systemd

[Andrei-Pozolotin]

FYI: now #48 brings basic qemu support for testing

1. core functions: * [arkon_config.py](https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/tool/module/arkon_config.py) * [produce_media.py](https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/tool/module/produce_media.py) 2. test unit: * [test/cryptsetup/verify.py](https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/tool/image/test/cryptsetup/verify.py) 3. azure build * [mkinitcpio-systemd-tool](https://dev.azure.com/random-archer/mkinitcpio-systemd-tool/_build/results?buildId=158&view=logs&j=7aab14fc-15ff-5f71-d20e-2c51e4e60603)

[Andrei-Pozolotin]

FYI: another thing to try: https://github.com/systemd/systemd/blob/master/NEWS

``` CHANGES WITH 245: ``` ``` * The /etc/fstab support in systemd now supports two new mount options x-systemd.{required,wanted}-by=, for explicitly configuring the units that the specified mount shall be pulled in by, in place of the usual local-fs.target/remote-fs.target. ``` ``` * The support for /etc/crypttab gained a new x-initrd.attach option. If set, the specified encrypted volume is unlocked already in the initrd. This concept corresponds to the x-initrd.mount option in /etc/fstab. ```

[fredleb]

now my crypttab is no longer copied in the initramfs for some reason... What should be the minimal conditions to have it pulled ?

[Andrei-Pozolotin]

the minimal conditions

things are relocated now:

[README.md](https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/README.md) ``` edit /etc/mkinitcpio-systemd-tool/config/crypttab edit /etc/mkinitcpio-systemd-tool/config/fstab systemctl enable initrd-cryptsetup.path systemctl enable initrd-tinysshd.service systemctl enable initrd-debug-progs.service systemctl enable initrd-sysroot-mount.service ```

[fredleb]

Back. So I restarted from pretty much scratch and now I see that enabling initrd-cryptsetup.path is not actually pulling initrd-cryptsetup.service at all...

I added manually a symlink to initrd-cryptsetup.service in etc/systemd/system/sysinit.target.wants/ but that does not change anything.

[Andrei-Pozolotin]

1. with latest release?

2. we have unit-test for that :-)

   • azure/mkinitcpio-systemd-tool
   • tool/image/test/cryptsetup/verify.py

[fredleb]

Yeah latest release. I actually reinstalled all arch packages...

[Andrei-Pozolotin]

actually pulling initrd-cryptsetup.service

you search like this, right?

mkdir -p /tmp/initrd
cd /tmp/initrd
lsinitcpio -x /boot/initramfs-linux.img
ls -las /tmp/initrd/usr/lib/systemd/system/initrd-cryptsetup.service
cat     /tmp/initrd/usr/lib/systemd/system/initrd-cryptsetup.service
ls -las /tmp/initrd/etc/systemd/system/sysinit.target.wants/initrd-cryptsetup.path

symlink to initrd-cryptsetup.service in etc/systemd/system/sysinit.target.wants/

this should not be done, see src/initrd-cryptsetup.path

# note:
# this is a twin unit for initrd-cryptsetup.service
# enable only initrd-cryptsetup.path, initrd-cryptsetup.service is activated on demand

[fredleb]

Nearly: lsinitcpio -l /boot/initramfs-linux.img | grep "crypt"

``` etc/systemd/system/sysinit.target.wants/initrd-cryptsetup.path usr/lib/libcrypt.so.1 usr/lib/libcrypto.so.1.1 usr/lib/libcryptsetup.so.12 usr/lib/libgcrypt.so.20 usr/lib/modules/5.5.13-arch2-1/kernel/amlogic-gxl-crypto.ko usr/lib/modules/5.5.13-arch2-1/kernel/ccp-crypto.ko usr/lib/modules/5.5.13-arch2-1/kernel/cryptd.ko usr/lib/modules/5.5.13-arch2-1/kernel/crypto_engine.ko usr/lib/modules/5.5.13-arch2-1/kernel/crypto_safexcel.ko usr/lib/modules/5.5.13-arch2-1/kernel/crypto_simd.ko usr/lib/modules/5.5.13-arch2-1/kernel/crypto_user.ko usr/lib/modules/5.5.13-arch2-1/kernel/dm-crypt.ko usr/lib/modules/5.5.13-arch2-1/kernel/fcrypt.ko usr/lib/modules/5.5.13-arch2-1/kernel/pcrypt.ko usr/lib/modules/5.5.13-arch2-1/kernel/tcrypt.ko usr/lib/modules/5.5.13-arch2-1/kernel/virtio_crypto.ko usr/lib/systemd/system-generators/systemd-cryptsetup-generator usr/lib/systemd/system/cryptsetup.target usr/lib/systemd/system/initrd-cryptsetup.path usr/lib/systemd/system/sysinit.target.wants/cryptsetup.target usr/lib/systemd/systemd-cryptsetup ```

[fredleb]

Could it be cause by the fact that I am in a chrooted env ?

I boot my broken system via the archiso, and then arch-chroot in the decrypted and mounted system...

But that was working 10 days ago...

[Andrei-Pozolotin]

lsinitcpio -l /boot/initramfs-linux.img | grep "crypt"

do you see problems during mkinicpio -v ... > build.log ?

Could it be cause by the fact that I am in a chrooted env ?

try with systemd-nspawn instead: wiki/System-Recovery

[fredleb]

No problem in the log. I think I was the problem again... I ran a mkinitcpio from systemd-spawn without success and then out of despair I reinstalled the mkinitcpio-systemd-tool, rebuilt, and here they are. So I guess I had put my finger in the /usr/lib/systemd/system stuff Pffff...... sorry to waste your time.. and mine... I'm gonna take a break now but I hope to solve this LVM stuff this week-end. Thanks for your help !

[Andrei-Pozolotin]

I think I was the problem again...

seems like low beer level, yep, definitely :-)

[fredleb]

Here is a detailed log of the problem...

I have trimmed it and replaced the \x2d with - for readability.

``` Apr 02 11:24:10 chaos kernel: Btrfs loaded, crc32c=crc32c-intel Apr 02 11:24:10 chaos kernel: BTRFS: device fsid 52a7433c-3914-422c-8d71-c016badb8c81 devid 4 transid 9713 /dev/dm-2 scanned by systemd-udevd (251) Apr 02 11:24:10 chaos lvm[353]: 1 logical volume(s) in volume group "root00vg" now active Apr 02 11:24:10 chaos lvm[377]: 1 logical volume(s) in volume group "root01vg" now active Apr 02 11:24:10 chaos kernel: BTRFS: device fsid 52a7433c-3914-422c-8d71-c016badb8c81 devid 5 transid 9713 /dev/dm-5 scanned by systemd-udevd (288) Apr 02 11:24:10 chaos kernel: BTRFS: device fsid 52a7433c-3914-422c-8d71-c016badb8c81 devid 3 transid 9713 /dev/dm-3 scanned by systemd-udevd (245) Apr 02 11:24:10 chaos lvm[362]: 1 logical volume(s) in volume group "root02vg" now active Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-name-root02vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-uuid-52a7433c-3914-422c-8d71-c016badb8c81.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-uuid-52a7433c-3914-422c-8d71-c016badb8c81.device: Job 53 dev-disk-by-uuid-52a7433c-3914-422c-8d71-c016badb8c81.device/start finished, result=done -> trigger mount /sysroot Apr 02 11:24:10 chaos systemd[1]: Found device /dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81. Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root02vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-uuid-LVM-XxuxxjLUhhnevLBq0hqTjd11oa1ZKtDfw5GVbdGDJAuaD9Ig2Ln8sXimZvnxMzZz.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-root02vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-dm-3.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: sys-devices-virtual-block-dm-3.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: ConditionPathExists=/etc/initrd-release succeeded. Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: ConditionPathExists=/etc/crypttab succeeded. Apr 02 11:24:10 chaos systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No such file or directory Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: About to execute: /usr/bin/mount /dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81 /sysroot -t btrfs -o noatime,x-systemd.device-timeout=9999h Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Forked /usr/bin/mount as 397 Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Changed dead -> mounting Apr 02 11:24:10 chaos systemd[1]: Mounting /sysroot... Apr 02 11:24:10 chaos systemd[397]: sysroot.mount: Executing: /usr/bin/mount /dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81 /sysroot -t btrfs -o noatime,x-systemd.device-timeout=9999h Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Got notification message from PID 208 (FDSTORE=1) Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Added fd 47 (n/a) to fd store. Apr 02 11:24:10 chaos kernel: BTRFS info (device dm-3): disk space caching is enabled Apr 02 11:24:10 chaos kernel: BTRFS info (device dm-3): has skinny extents Apr 02 11:24:10 chaos systemd[1]: blockdev@dev-mapper-root02vg-root.target: Failed to load configuration: No such file or directory Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Changed mounting -> mounting-done Apr 02 11:24:10 chaos systemd[1]: Received SIGCHLD from PID 397 (mount). Apr 02 11:24:10 chaos systemd[1]: Child 397 (mount) died (code=exited, status=0/SUCCESS) Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Child 397 belongs to sysroot.mount. Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Mount process exited, code=exited, status=0/SUCCESS Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Changed mounting-done -> mounted Apr 02 11:24:10 chaos systemd[1]: sysroot.mount: Job 52 sysroot.mount/start finished, result=done Apr 02 11:24:10 chaos systemd[1]: Mounted /sysroot. Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 47 (stored), closing. Apr 02 11:24:10 chaos systemd[1]: dev-root01vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-uuid-LVM-5bbu07nRcueHARaTs0CcY2KXY0TiTMzAgN0WIF9nYmgDGRvJQ64crKFPIDBpOjyY.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-name-root01vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root01vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-dm-5.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: sys-devices-virtual-block-dm-5.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: initrd-root-fs.target: ConditionPathExists=/etc/initrd-release succeeded. Apr 02 11:24:10 chaos systemd[1]: initrd-root-fs.target changed dead -> active Apr 02 11:24:10 chaos systemd[1]: initrd-root-fs.target: Job 50 initrd-root-fs.target/start finished, result=done Apr 02 11:24:10 chaos systemd[1]: Reached target Initrd Root File System. Apr 02 11:24:10 chaos systemd[1]: initrd-parse-etc.service: ConditionPathExists=/etc/initrd-release succeeded. Apr 02 11:24:10 chaos systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No such file or directory Apr 02 11:24:10 chaos systemd[1]: initrd-parse-etc.service: About to execute: /usr/local/bin/systemctl daemon-reload Apr 02 11:24:10 chaos systemd[1]: initrd-parse-etc.service: Forked /usr/local/bin/systemctl as 419 Apr 02 11:24:10 chaos systemd[1]: initrd-parse-etc.service: Changed dead -> start-pre Apr 02 11:24:10 chaos systemd[1]: Starting Reload Configuration from the Real Root... Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: ConditionPathExists=/etc/initrd-release succeeded. Apr 02 11:24:10 chaos systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No such file or directory Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Passing 0 fds to service Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: About to execute: /bin/true Apr 02 11:24:10 chaos systemd[419]: initrd-parse-etc.service: Executing: /usr/local/bin/systemctl daemon-reload Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Forked /bin/true as 421 Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Changed dead -> start Apr 02 11:24:10 chaos systemd[1]: Starting Initrd /sysroot binder... Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Got notification message from PID 208 (FDSTORE=1) Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Added fd 47 (n/a) to fd store. Apr 02 11:24:10 chaos systemd[421]: initrd-sysroot-mount.service: Executing: /bin/true -> too late Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-uuid-52a7433c-3914-422c-8d71-c016badb8c81.device: Device dev-disk-by-uuid-52a7433c-3914-422c-8d71-c016badb8c81.device appeared twice with different sysfs paths /sys/devices/virtual/block/dm-3 and /sys/devices/virtual/block/dm-2, ignoring the latter. Apr 02 11:24:10 chaos systemctl[419]: Bus n/a: changing state UNSET → OPENING Apr 02 11:24:10 chaos systemctl[419]: Bus n/a: changing state OPENING → AUTHENTICATING Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-uuid-LVM-I8Z99zHFIxmRYrnv5SEstf48RoQS25kyr1NkdsfaaihdTa5kcx987ihuCc0i3Ido.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-root00vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root00vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-disk-by-id-dm-name-root00vg-root.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: dev-dm-2.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: sys-devices-virtual-block-dm-2.device: Changed dead -> plugged Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Got notification message from PID 208 (FDSTORE=1) Apr 02 11:24:10 chaos systemd[1]: systemd-journald.service: Added fd 48 (n/a) to fd store. Apr 02 11:24:10 chaos systemd[1]: Received SIGCHLD from PID 421 (true). Apr 02 11:24:10 chaos systemd[1]: Child 421 (true) died (code=exited, status=0/SUCCESS) Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Child 421 belongs to initrd-sysroot-mount.service. Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Main process exited, code=exited, status=0/SUCCESS Apr 02 11:24:10 chaos audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=initrd-sysroot-mount comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success' Apr 02 11:24:10 chaos audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lvm2-pvscan@254:0 comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success' Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Changed start -> exited Apr 02 11:24:10 chaos systemd[1]: initrd-sysroot-mount.service: Job 54 initrd-sysroot-mount.service/start finished, result=done Apr 02 11:24:10 chaos systemd[1]: Finished Initrd /sysroot binder. ```

Clearly what I see here is that the 3 identical uuids are confusing systemd... So the problem is not with your tool but with systemd and btrfs. I have no idea how to solve that. I would imagine that systemd would be the one responsible for handling that and not udev. What would you recommend I go from there ? Should I write a ticket in systemd or a question in superuser.com ?

[Andrei-Pozolotin]

it says Apr 02 - you sure looking at the current problem?

[fredleb]

Yeah. It's a virtual machine, I did not bother with loads of config because it is only there to reproduce this problem on something faster than the real machine.

[Andrei-Pozolotin]

assuming you had it working in the past, this symptom could be:

• a "new config problem"
• just normal for btrfs, so: "ignoring the latter"

[Andrei-Pozolotin]

perhaps wrong level uuid in crypttab?

/usr/bin/mount /dev/disk/by-uuid/52a7433c-3914-422c-8d71-c016badb8c81 /sysroot -t btrfs -o noatime,x-systemd.device-timeout=9999h

this make it look like "higher level block device uuid"

BTRFS: device fsid 52a7433c-3914-422c-8d71-c016badb8c81

[fredleb]

hmmm...

It never really worked properly.

I mean it boots, but there are errors that don't crash the booting on the virtual machine but do on the real one. The real one has slow mechanical HDDs whereas the virtual ones are all on the same ssd of my workstation.

What works:

• asking for password
• decrypting the "keys" partition and mounting it
• decrypting the "sys" partitions

But the first time the shared uuid triggers the mount, at least one of the 3 partitions is missing...

[fredleb]

UUID is correct.

``` NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT sda |-sda1 |-sda2 ext4 1.0 d51deb1b-b348-4d9c-8a0c-ed256a73c7c7 304.3M 30% /boot `-sda3 crypto_LUKS 1 ed5d6f31-8015-4904-8ca6-e4f0d952f025 `-sys00 LVM2_member LVM2 001 CtPzrD-XXVK-D8B7-NJcO-ulhL-TCaR-RTuEo3 `-root00vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 sdb |-sdb1 |-sdb2 ext4 1.0 e0012f06-307c-46f6-be62-4f43bbaebbf8 `-sdb3 crypto_LUKS 1 f2a34e45-8bdd-433d-b73a-4dd3336be28a `-sys01 LVM2_member LVM2 001 opvZYc-TYWt-Up6S-nBlW-KrKE-0cUc-QeiFWu `-root01vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 sdc |-sdc1 |-sdc2 `-sdc3 crypto_LUKS 1 5f81c88b-a5c7-471e-981e-3b5b5b406ee6 `-sys02 LVM2_member LVM2 001 yLpKvu-9SjM-dQWh-5zMZ-hXzD-dcwv-95vyjZ `-root02vg-root btrfs 52a7433c-3914-422c-8d71-c016badb8c81 9.2G 22% / ```

[Andrei-Pozolotin]

• the uuid id definitely "higher level", and udev/systemd definitely must handle that, only how? :-)

• these guys look healthy:

  Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root00vg-root.device: Changed dead -> plugged
  Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root01vg-root.device: Changed dead -> plugged
  Apr 02 11:24:10 chaos systemd[1]: dev-mapper-root02vg-root.device: Changed dead -> plugged

  perhaps go back to explicit unit dependency [all-3-of: dev-mapper-rootXX] -> sysroot ?

• basically, looks like missing one more level of indirection in fstab