Since systemd-251.3 mdadm doesn't start at boot time

[s9gf4ult]

Describe the bug

It started when I tried to update the system by updating the nixpkgs after which I could not boot the system.

My layout is next:

nvme0n1               259:0    0  1,8T  0 disk
└─nvme0n1p1           259:4    0  1,8T  0 part
  └─md127               9:127  0  3,6T  0 raid0
    └─encroot         254:0    0  3,6T  0 crypt
      ├─fasta-root    254:2    0  3,5T  0 lvm   /nix/store
      │                                         /
      └─fasta-swap    254:3    0  129G  0 lvm
nvme1n1               259:1    0  1,8T  0 disk
├─nvme1n1p1           259:2    0    5G  0 part  /boot
└─nvme1n1p2           259:3    0  1,8T  0 part
  └─md127               9:127  0  3,6T  0 raid0
    └─encroot         254:0    0  3,6T  0 crypt
      ├─fasta-root    254:2    0  3,5T  0 lvm   /nix/store
      │                                         /
      └─fasta-swap    254:3    0  129G  0 lvm

The encroot is an encrypted pv occupying the md127 the raid0 array configured with mdadm. The fasta/root is my root partition with ext4 on it.

So, to boot the system initrd scripts must first initialize the array md127 with mdadm, then decrypt encroot asking me to type the password, then initialize the fasta vg and mount the fasta/root as root file system.

This had been perfectly working before I updated nixpkgs at some point. After update the systemd printed the message Waiting the device /... at early stage of boot (no root partition mounted). This device by UUID is definitely md127 which didn't initialized.

I decided to bisect the nixpkgs to figure out which commit caused the problem, and here is bisect log:

git bisect start
# status: waiting for both good and bad commits
# good: [ce49cb7792a7ffd65ef352dda1110a4e4a204eac] neovim: pass the --clean flag to only use wrapped rc
git bisect good ce49cb7792a7ffd65ef352dda1110a4e4a204eac
# status: waiting for bad commit, 1 good commit known
# bad: [36cc29d837e7232e3176e4651e8e117a6f231793] Merge pull request #185621 from tjni/fix-3mux
git bisect bad 36cc29d837e7232e3176e4651e8e117a6f231793
# bad: [efb7405702e62f6edbdcfcacd2ce3e56aa9de40a] Merge pull request #183986 from r-ryantm/auto-update/catch2
git bisect bad efb7405702e62f6edbdcfcacd2ce3e56aa9de40a
# bad: [361072ab382e527f47fc4170df6dc82d1c691190] nrfutil: 6.1.3 -> 6.1.6
git bisect bad 361072ab382e527f47fc4170df6dc82d1c691190
# bad: [5857574d45925585baffde730369414319228a84] Merge pull request #183307 from Djabx/autoUpgrabe-boot
git bisect bad 5857574d45925585baffde730369414319228a84
# bad: [4b9ec37bffd0d5cca19b4f574c7a9b271d44ee13] vim-vint: set meta.mainProgram
git bisect bad 4b9ec37bffd0d5cca19b4f574c7a9b271d44ee13
# skip: [64d382d608158fc7c739ad9a0d2bf665c95ca5a4] python3Packages.tensorboard_plugin_profile: 2.5.0 -> 2.8.0
git bisect skip 64d382d608158fc7c739ad9a0d2bf665c95ca5a4
# bad: [50d51b452225c312635dcdca58b62b75f6169fde] python3Packages.pymdown-extensions: 9.4 -> 9.5
git bisect bad 50d51b452225c312635dcdca58b62b75f6169fde
# bad: [4775efd1f48bd2df2e41aeea2b4137f6c2b51daf] python3Packages.chardet: 4.0.0 -> 5.0.0
git bisect bad 4775efd1f48bd2df2e41aeea2b4137f6c2b51daf
# bad: [bbe8931b36199d505042195cc86b24a7d776f78a] pipewire: 0.3.55 -> 0.3.56
git bisect bad bbe8931b36199d505042195cc86b24a7d776f78a
# good: [0f4858cb394553bfbae7254e3b0e550c783f6f8e] Merge pull request #181359 from SuperSandro2000/python310Packages.zipp
git bisect good 0f4858cb394553bfbae7254e3b0e550c783f6f8e
# good: [a8cdd8a284bab0458b7411af1b1d86abe7ecb273] Merge pull request #181170 from trofi/update-kmod
git bisect good a8cdd8a284bab0458b7411af1b1d86abe7ecb273
# bad: [1ae70f1c80a17547fe18c5f6086ddc9f275d94b3] Merge staging-next into staging
git bisect bad 1ae70f1c80a17547fe18c5f6086ddc9f275d94b3
# good: [111abd87ca2376ba3b31f6ab347ae3974e5c53af] Merge pull request #168590 from amjoseph-nixpkgs/stdenv-disallowedReferences
git bisect good 111abd87ca2376ba3b31f6ab347ae3974e5c53af
# bad: [3eb1aa774944136d52f7f7f6a1c0e17e0dfcdd96] Merge pull request #179681 from Mic92/systemd
git bisect bad 3eb1aa774944136d52f7f7f6a1c0e17e0dfcdd96
# bad: [a14d1a2e7e8c19087897bc646a37f50096b736b1] systemd: 250.4 -> 251.3
git bisect bad a14d1a2e7e8c19087897bc646a37f50096b736b1
# first bad commit: [a14d1a2e7e8c19087897bc646a37f50096b736b1] systemd: 250.4 -> 251.3

The same system configuration was tested, only the nixpkgs commit was changing.

Reboot followed each step of bisect, so it took several days to finish :(

Finally, the problem is that systemd doesn't initialize the array with mdadm in which the whole system lives.

Steps To Reproduce

Steps to reproduce the behavior:

1. Checkout any commit from a14d1a2e7e8c19087897bc646a37f50096b736b1 to 32096899af23d49010bd8cf6a91695888d9d9e73 of the nixpkgs
2. Try to build the system with storage layout as shown above

Expected behavior

At early stage of boot (initrd loaded, but lot root partibion) systemd scripts initialize the array md127 with mdadm, then decrypt encroot asking me to type the password, then initialize the fasta vg and mount the fasta/root as root file system.

Additional context

This does not happen if checkout some commit before the a14d1a2e7e8c19087897bc646a37f50096b736b1 of nixpkgs, with exactly same system configuration.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.56, NixOS, 22.11 (Raccoon), 22.11.git.ce49cb7792a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.3`
 - nixpkgs: `/etc/nixos/nixpkgs`

[alexbakker]

I'm seeing this as well after updating to the 22.11 beta. Your setup appears to be very similar to mine, with the main difference being that I don't use LVM. Did you ever figure out a workaround for this by any chance?

(And if not, would this qualify as a release blocker? I imagine anyone using a combination of LUKS and RAID is going to get bitten by this)

[johnalotoski]

Just discovered I can't boot one of my machines with RAID and LUKS as well after updating to 22.11. Setup shown here.

[mmarx]

This seems to be the same issue as #199551. As a workaround, you can refer to the device using the /dev/disk/by-id/md-name-… links.

[johnalotoski]

That workaround works for me @mmarx! Thanks very much for the tip!

[alexbakker]

@mmarx That works, thanks! I also tested with /dev/disk/by-id/md-uuid-... and that actually works as well.

[kravemir]

Just discovered I can't boot one of my machines with RAID and LUKS as well after updating to 22.11.

The same case for me, after I re-booted my headless NAS upgraded to NixOS 22.11. Which sucks - no display, no keyboard - needed to move the machine to investigate.

This seems to be the same issue as #199551. As a workaround, you can refer to the device using the /dev/disk/by-id/md-name-… links.

The workaround - configuration rework - works for me too, but I have a bit more complex disk setup with multiple raids for the NAS.

Luckily, the system successfully booted to previous generation - boot entry. So, I could spin up the system.

---

Multiple mdadm RAIDs

Drives configuration:

``` - NVMe drive: - LUKS - LVM - `/` - system - 2 pieces of HDD: - mdadm in RAID1 - LUKS - LVM - `ext4` mounted on `/storage/hdddata` - 2 pieces of SSD: - mdadm in RAID1 - LUKS - LVM - `ext4` mounted on `/storage/ssddata` ```

I could prepare a fix for the configuration. However I didn't want to mess up - mix up IDs of drives. So, I've used following commands:

# figure out UUIDs for md127 and md126 virtual RAID devices
lsblk -f

# figure out mdadm's UUIDs for md127 and md126
ls -lh /dev/disk/by-id/md-*

Then, I validated, that my pairing is correct by checking new .../md-uuid... locations are mapped to correct .../by-uuid/... locations by putting them into same ls -lh ... command:

❯ ls -lh /dev/disk/by-uuid/8ca24d0f-6140-49f0-b8d7-e8810f0b60b6 /dev/disk/by-id/md-uuid-b536ed2e\:1762be89\:58616e54\:d1dac3ba 
lrwxrwxrwx 1 root root 11 Jan 14 11:22 /dev/disk/by-id/md-uuid-b536ed2e:1762be89:58616e54:d1dac3ba -> ../../md127
lrwxrwxrwx 1 root root 11 Jan 14 11:22 /dev/disk/by-uuid/8ca24d0f-6140-49f0-b8d7-e8810f0b60b6 -> ../../md127

❯ ls -lh /dev/disk/by-uuid/7d0120f7-6f50-428e-9aa5-71a5fc44ff7c /dev/disk/by-id/md-uuid-1cc099f4\:94a3981d\:b89715c3\:7a17952f 
lrwxrwxrwx 1 root root 11 Jan 14 11:22 /dev/disk/by-id/md-uuid-1cc099f4:94a3981d:b89715c3:7a17952f -> ../../md126
lrwxrwxrwx 1 root root 11 Jan 14 11:22 /dev/disk/by-uuid/7d0120f7-6f50-428e-9aa5-71a5fc44ff7c -> ../../md126

Note - hint: after I was done, I figured out, that I could have also used LABEL column from lsblk -f for matching IDs.

[Gonzih]

Config example of using LABEL for mdadm config:

  boot.initrd.services.swraid.enable = true;
  boot.initrd.services.swraid.mdadmConf =
    "ARRAY /dev/md0 metadata=1.2 name=partitionlabel LABEL=partitionlabel";
  boot.initrd.luks.devices."devicename" = {
    device = "/dev/disk/by-id/md-name-partitionlabel";
  }

Though I will leave this here in case anyone needs more details.

[tyrion]

@Gonzih No need to specify swraid.mdadmConf, at least in my case, as the automatic scan is able to correctly assemble the array. The only issue seems to be the missing /dev/disk/by-uuid, which is fixed by using the workaround suggested by @mmarx.

[beardhatcode]

I also seem to have this problem