mkinitcpio-hook is broken after move to "community"

[Chief-Detektor]

Hi! Thanks for this cool project! It worked really great until last update.

`mkinitcpio -p linux` completes with the following error: ``` ==> ERROR: Missing any units in /etc/systemd/system with entry ConditionPathExists=/etc/initrd-release ``` After reboot sshd is not reachable from initrd and I have to wait to the timeout to be able to access the rescure shell. From there I can unencrypt my root partition and via `systemctl default` continue the boot process as usual. Something else I noticed besides the missing ability to access via ssh is that I am also unable to use ctrl+alt+Fx keys to get an other tty how I did before to be able to unlock the root partition during the timeout. Do I need to change my config? I could not see any *.pacnew config that would have pointed me to update my configs and also I could not see anything obvious in the last couple commits.

Regards!

[ml-]

tl;dr:

• edit configuration in /etc/mkinitcpio-systemd-tool/
• systemctl enable initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service

I guess the AUR package went away too quickly alongside it's comments:

> dvzrv commented: > > Hey all! I've [made this package FHS compliant](https://github.com/random-archer/mkinitcpio-systemd-tool/pull/30) for the sake of moving it to the [community] > repsository. > > This means, that when this package moves to [community] it *will not* > ship the systemd units in `/etc/systemd/system` anymore and it *will > not* have those files in the backup array either (modification and > enabling of units works as before)! > The crypttab and fstab for the initrd are then to be found in > `/etc/mkinitcpio-systemd-tool/config` and the network configuration is > found in `/etc/mkinitcpio-systemd-tool/network`. > > *Please backup your modifications to the above files* and direct any generally applicable > modifications towards [upstream](https://github.com/random-archer/mkinitcpio-systemd-tool) in the future. >

[Andrei-Pozolotin]

watch for v22

1. please watch for https://github.com/random-archer/mkinitcpio-systemd-tool/releases/tag/v22 2. in `community` https://www.archlinux.org/packages/community/any/mkinitcpio-systemd-tool/ 3. and report back if that resolves the issue

[dvzrv]

It might also be worthwhile creating a short FAQ or extending the wiki for setup explanations. Those can then be linked to from the Documentation directive in the units.

[ShapeShifter499]

I'm also getting this error every time I run mkinitcpio -P

``` # Warning: initrd-tinysshd.service changed on disk, the version systemd has loaded is outdated. # This output shows the current version of the unit's original fragment and drop-in files. # If fragments or drop-ins were added or removed, they are not properly reflected in this output. # Run 'systemctl daemon-reload' to reload units. ``` I like to have tinyssh on a different ssh port so I thought after this upgrade I could use "systemd edit initrd-tinyssh.service" and add the following, where is my different port number. But doing this seems to trigger the above error. ``` [Service] ExecStart=/usr/bin/busybox tcpsvd -v 0 /usr/bin/tinysshd -v /etc/tinyssh/sshkeydir ```

[ml-]

I'm also getting this error every time I run mkinitcpio -P

> ``` > # Warning: initrd-tinysshd.service changed on disk, the version systemd has loaded is outdated. > # This output shows the current version of the unit's original fragment and drop-in files. > # If fragments or drop-ins were added or removed, they are not properly reflected in this output. > # Run 'systemctl daemon-reload' to reload units. > ``` > > I like to have tinyssh on a different ssh port so I thought after this upgrade I could use "systemd edit initrd-tinyssh.service" and add the following, where is my different port number. But doing this seems to trigger the above error. > > ``` > [Service] > ExecStart=/usr/bin/busybox tcpsvd -v 0 /usr/bin/tinysshd -v /etc/tinyssh/sshkeydir > ```

It works since 22-1:

- `systemctl daemon-reload` - Remove all the leftovers of the AUR package in `/etc/systemd/system` - Configure everything in `/etc/mkinitcpio-systemd-tool` - **enable service** `systemctl enable initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service` - Create dropin `systemctl edit initrd-tinyssh.service` - Rerun mkinitcpio `mkinitcpio -p linux{,-lts,zen,hardened,whatever}` I use it with another port and I can confirm that it works since yesterday. Also make sure to reset `ExecStart=` before adding the new one, otherwise you will have tinysshd listening on both ports.

[Chief-Detektor]

Thanks for the fast replies! Can confirm that it works on 22-1 with the steps described by @ml- above.

I did all execpt `Create dropin systemctl edit initrd-tinyssh.service` What is the purpose of this step? Now since this package is in community I suggest that you display an install note on upgrade including the steps above just in case somebody else runs into this issue when upgrading from the old aur package. Thanks for the fast reply and help. You guys are awesome!

[ml-]

I did all execpt Create dropin systemctl edit initrd-tinyssh.service What is the purpose of this step?

It is not required if you want to keep the default setup. It's just in case you want to adjust systemd units without creating a full copy.

For example if you want to change the default port of `initrd-tinyssh.service`, invoke `systemctl edit initrd-tinyssh.service` which will open an editor. Then you can override existing directives like this: ```systemd [Service] # empty ExecStart to remove previous value! See man systemd.service(5) ExecStart= # new command which listens on port 22022 ExecStart=/usr/bin/busybox tcpsvd -v 0 22022 /usr/bin/tinysshd -v /etc/tinyssh/sshkeydir ``` It is nice because it's just an override/addition to an existing unit and the original unit file remains untouched and can be seamlessly updated in the next package update. As already mentioned, this is not required, it's just very nice for users adjusting their systemd units.

[Andrei-Pozolotin]

meanwhile, will use a "scary warning": https://github.com/random-archer/mkinitcpio-systemd-tool/commit/903e5485172176428cfaa2ed7bf3e823083a35c2

[dvzrv]

There's now a warning message in the package upon upgrade from <=22.

[Chief-Detektor]

@ml- Thanks for the explanation! The warning sure will help others to cope with the migration from aur to community version! That's cool!

[ShapeShifter499]

Ok new issue. I booted into a arch iso to try to recover the system and I get this error running mkinitcpio.

``` ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default' -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img ==> Starting build: 5.5.10-arch1-1 -> Running build hook: [base] -> Running build hook: [autodetect] -> Running build hook: [modconf] -> Running build hook: [block] -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] -> Running build hook: [systemd] -> Running build hook: [systemd-tool] Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img ==> Image generation successful ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback' -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect ==> Starting build: 5.5.10-arch1-1 -> Running build hook: [base] -> Running build hook: [modconf] -> Running build hook: [block] ==> WARNING: Possibly missing firmware for module: wd719x ==> WARNING: Possibly missing firmware for module: aic94xx -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] -> Running build hook: [systemd] -> Running build hook: [systemd-tool] Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat Running in chroot, ignoring request: cat ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img ==> Image generation successful ```

[dvzrv]

@ShapeShifter499 I can't reproduce that.

Could you paste the modifications on top of any of the `initrd-*` units and scripts, that you did? Maybe there are still leftovers from the AUR package in place. You could start by identifying the enabled units and then post their entire modifications: `systemctl cat `

[ShapeShifter499]

@dvzrv

@ShapeShifter499 I can't reproduce that.

Before I do, did you try this from inside of a chroot (arch-chroot) from a running arch linux iso?

[LubosKolouch]

Yes, same issue here if I build initrd from chroot.

[dvzrv]

Ah... I see.

I guess `systemctl cat ` is not allowed to be run in a chroot. That's pretty meh... Also, no documentation on this AFAICS

[dvzrv]

Potentially relevant systemd issues and commits:

https://github.com/systemd/systemd/issues/5446, https://github.com/systemd/systemd/commit/295ee9845c801300298d01256eb5a9e3ff117ae0 I guess running systemctl commands within chroot is more or less breaking or not really supported (and I understand why you don't want to be able to e.g. start a service in a chroot). However, something like `systemctl cat` should still work IMHO. Maybe worth reporting?

[dvzrv]

I guess we either distinguish between the use-cases

(e.g. `none`, `systemd-nspawn`, `chroot`) based on `systemd-detect-virt` or revert back to an implementation, that basically re-implements the configuration overlay system of systemd. Sorry for poking for that change in the pull request @ml- :-/ I didn't know it wouldn't work in a chroot.

[dvzrv]

@ShapeShifter499 That all being said, I just found something probably of value.

There is [mention of](https://github.com/systemd/systemd/commit/295ee9845c801300298d01256eb5a9e3ff117ae0#diff-0e08ed2dbc97131dbf8deb9eb7af4163L561) `SYSTEMD_IGNORE_CHROOT`. Could you try exporting this (e.g. `export SYSTEMD_IGNORE_CHROOT=1`) and try again? Not entirely sure it will work without dbus connection tho

[ShapeShifter499]

@dvzrv it's no good I get the following errors.

``` root@archiso ~ # arch-chroot /mnt [root@archiso /]# SYSTEMD_IGNORE_CHROOT=1 [root@archiso /]# export SYSTEMD_IGNORE_CHROOT=1 [root@archiso /]# mkinitcpio -P ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default' -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img ==> Starting build: 5.5.10-arch1-1 -> Running build hook: [base] -> Running build hook: [autodetect] -> Running build hook: [modconf] -> Running build hook: [block] -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] -> Running build hook: [systemd] -> Running build hook: [systemd-tool] Failed to connect to bus: No data available Failed to connect to bus: No data available Failed to connect to bus: No data available Failed to connect to bus: No data available ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img ==> Image generation successful ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback' -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect ==> Starting build: 5.5.10-arch1-1 -> Running build hook: [base] -> Running build hook: [modconf] -> Running build hook: [block] ==> WARNING: Possibly missing firmware for module: wd719x ==> WARNING: Possibly missing firmware for module: aic94xx -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] -> Running build hook: [systemd] -> Running build hook: [systemd-tool] Failed to connect to bus: No data available Failed to connect to bus: No data available Failed to connect to bus: No data available Failed to connect to bus: No data available ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img ==> Image generation successful [root@archiso /]# ```

[dvzrv]

@ShapeShifter499 hang in there. I'm working on a fix.

[dvzrv]

@ShapeShifter499 please try again with #34 applied.

[ShapeShifter499]

@dvzrv manually applied your workaround and now 'mkinitcpio -P' completes as normal in chroot. System boots as expected, prompting for encryption key via tinyssh ssh server again.

Thank you

[ShapeShifter499]

@ml- Thank you for your help too, I forgot about needing to add a empty variable before replacing it in 'systemctl edit' or in the override.conf file

[ml-]

@ShapeShifter499 I'm very sorry that I didn't consider this case. Should've taken into account that running mkinitcpio inside a chroot is a common thing to do.

[Andrei-Pozolotin]

additional workarounds for "chroot issue" could be:

* to start `systemd` inside `chroot` * try SYSTEMCTL_INSTALL_CLIENT_SIDE=1 see https://github.com/systemd/systemd/blob/master/docs/ENVIRONMENT.md

[Andrei-Pozolotin]

@dvzrv no dice with systemctl cat, must roll our own as you did

[ml-]

I'm going to play around with systemd and chroot later.

`SYSTEMCTL_INSTALL_CLIENT_SIDE` sounds interesting. Maybe it will be possible to use systemd functionality after all without re-implementing it in Bash

[dvzrv]

Meanwhile I think we should release a new version

[Andrei-Pozolotin]

yes, I was praying to you whole day https://github.com/random-archer/mkinitcpio-systemd-tool/pull/34#issuecomment-602049901 since only you are the master of the community :-)

[vaminakov]

> **tl;dr:** > > * edit configuration in `/etc/mkinitcpio-systemd-tool/` > > * `systemctl enable initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service` > >

Followed this notes, seems all works. System boot, cryptsetup successfully opened container... But dropbear doesnt work (sure i'm fixed /xxx/initrd-release and enable state)

[ml-]

I guess the /xxx can now be removed in general.

[vlna]

My system has a completely fresh Arch installation. I'm not sure what I'm doing wrong.

I don't know how to enter password. I see menu (a,s,r,q) after ssh but `a` does nothing (it waits a few seconds). Physical console doesn't work at all. ``` Mar 23 18:49:48 w-16gb shell[253]: console/ssh info : init Mar 23 18:49:48 w-16gb shell[257]: console/ssh info : user menu Mar 23 18:52:42 w-16gb shell[265]: console/ssh info : custom agent try #1 Mar 23 18:52:48 w-16gb shell[327]: console/ssh warn : missing request #1 Mar 23 18:52:51 w-16gb shell[330]: console/ssh info : custom agent try #1 Mar 23 18:52:57 w-16gb shell[392]: console/ssh warn : missing request #1 Mar 23 18:53:49 w-16gb shell[394]: console/ssh info : run sub shell ``` My crypttab is ``` data UUID=a939a60d-6ad6-461a-a997-0f18ef7cb838 none luks,timeout=180 ``` and blkid returns ``` /dev/sdb3: UUID="a939a60d-6ad6-461a-a997-0f18ef7cb838" TYPE="crypto_LUKS" ``` My fstab hasn't valid lines

[Andrei-Pozolotin]

@vlna Vladimír:

1. if you trace your log output in [initrd-shell.sh](https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-shell.sh) you can understand what is going on i.e. `CTRL+F` for log message `custom agent try #1`, etc 2. since you do have some console available, try to open `cryptsetup` disk manually: https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption ``` cryptsetup open /dev/sdb3 data ``` but then, ` /dev/sdb3 data` is not your root disk, where is your root disk? 3. > Physical console doesn't work at all missing usb / kbd driver? did it work right before this change? can you connect another/external/different ps2 or usb kbd device? 4. > My fstab hasn't valid lines what do you mean? it is empty? it shows default content? it is not what you have configured? did you apply changes in `/etc/mkinitcpio-systemd-tool/`? https://github.com/random-archer/mkinitcpio-systemd-tool/issues/32#issuecomment-601661569 show result of `cat fstab`, `cat crypttab`, when in initramfs `console/ssh`

[vlna]

1. I'm not sure what you mean 2. cryptsetup is missing in the image (I have linux-lts + fallback only) - it looks like root cause. Package cryptsetup-2.3.1-1 is installed. `/dev/sbd3` is not root disk and I want to unlock it only to mount and/or to symlink it later. root is `/dev/sda2`, boot is `/dev/sda1` 3. It was working before changes and it is working when I boot from USB to rebuild initramfs 4. `fstab` has all lines commented out with # initramfs is built without errors. I removed `systemd` and `systemd-tool` from fallback initramfs to get easy way to boot full system

``` [root@archiso etc]# mkinitcpio -P ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default' -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img ==> Starting build: 5.4.26-1-lts -> Running build hook: [base] -> Running build hook: [udev] -> Running build hook: [autodetect] -> Running build hook: [modconf] -> Running build hook: [block] -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] -> Running build hook: [systemd] -> Running build hook: [systemd-tool] convert openssh to tinysshd host key ed25519 ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img ==> Image generation successful ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback' -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect -S systemd -S systemd-tool ==> Starting build: 5.4.26-1-lts -> Running build hook: [base] -> Running build hook: [udev] -> Running build hook: [modconf] -> Running build hook: [block] ==> WARNING: Possibly missing firmware for module: aic94xx ==> WARNING: Possibly missing firmware for module: wd719x -> Running build hook: [filesystems] -> Running build hook: [keyboard] -> Running build hook: [fsck] ==> Generating module dependencies ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts-fallback.img ==> Image generation successful ```

[Andrei-Pozolotin]

@vlna Vladimír: it seems there is no real problem, only misunderstanding :-) please spend more time and study again both readme.md and this whole thread issues/32

[vaminakov]

I guess the /xxx can now be removed in general.

Sure I did. Or you mean need to remove full block ConditionPathExists= ?

[Andrei-Pozolotin]

@vvl-rulez Vladislav:

But dropbear doesnt work

please report specifically:

1. what do you mean, exactly? collect logs from `initramfs` boot: * `journalctl -b -t shell` * `ps | grep ...` * etc,.. 2. please try `tinysshd` instead, and report back: https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-tinysshd.service 3. you can look / navigate inside `/boot/initramfs-linux.img` after `mkinitcpio` but before `reboot`, say using [`mc`](https://www.archlinux.org/packages/community/x86_64/mc/), so take a look: does it all "look good": * expected binaries are present * expected configs are present / enabled * etc, etc.

[Anty0]

Hi, Thank you for this tool. :) Unfortunately I have some issues after last update I have failed to fix.

I have managed to successfully fix my configuration after last update, but after reboot I got similar error as described in issue #10 . I can mount sysroot manually and resume boot. I'm not sure where the real issue is. I didn't have this problem before. (Sorry for my English...)

My configuration (package files diff):

- Kernel cmd line: - `rd.luks.name=18ceaf63-f1d8-4c75-b60a-0157ef155384=antyluks scsi_mod.use_blk_mq=1 fbcon=scrollback:1024k sysrq_always_enabled=1` - `/etc/mkinitcpio-systemd-tool/config/fstab`: ```diff 21a22,24 > > # /dev/mapper/vgmain-root > UUID=36737474-7501-4d2c-8eed-9fd34dd5d6ce /sysroot ext4 ro,noatime,async,x-systemd.device-timeout=9999h 0 1 ``` - `/usr/lib/systemd/system/initrd-cryptsetup.service`: ```diff 44a45 > InitrdBinary=/usr/bin/cryptsetup ``` - `/usr/lib/systemd/system/initrd-emergency.service`: ```diff 13c13 < ConditionPathExists=/etc/xxx/initrd-release --- > ConditionPathExists=/etc/initrd-release 35c35 < InitrdService=disable --- > InitrdService=enable 38c38 < #InitrdPath=/etc/systemd/system/emergency.service source=/etc/systemd/system/initrd-emergency.service --- > InitrdPath=/etc/systemd/system/emergency.service source=/usr/lib/systemd/system/initrd-emergency.service ``` - Executed commands after update: - `systemctl daemon-reload` - `systemctl enable initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service`

[Andrei-Pozolotin]

@Anty0 apparently your have "sysroot-on-lvm"? in this case follow @fredleb on #36

[chocobytes]

But dropbear doesnt work

> **tl;dr:** > ``` > * edit configuration in `/etc/mkinitcpio-systemd-tool/` > > * `systemctl enable initrd-cryptsetup.path initrd-network.service initrd-shell.service initrd-tinysshd.service` > ``` Followed this notes, seems all works. System boot, cryptsetup successfully opened container... But dropbear doesnt work (sure i'm fixed /xxx/initrd-release and enable state)

I was having the same issue.

I was having the same issue. Quick and dirty fix -- in addition to running "systemctl enable initrd-dropbear.service" and fixing ConditionPathExists like you did also change in the [X-SystemdTool] section InitrdService from "disable" to "enable" and uncomment InitrdBuild. It's worth nothing directly editing package service unit files is not recommended but placing an edited copy under /etc/systemd/system/ folder to override doesn't work here so I'm not sure what the "proper" thing to do would be.

[Andrei-Pozolotin]

@chocobytes @vvl-rulez

in addition to running "systemctl enable initrd-dropbear.service"

remember that dropbear and tinysshd are mutually exclusive, unless ran on separate ports, so it is needed to enable one / disable another

[Andrei-Pozolotin]

@kolcon @ShapeShifter499 please confirm "chroot issue" is resolved after https://github.com/random-archer/mkinitcpio-systemd-tool/releases/tag/v23

[Anty0]

@Andrei-Pozolotin

Thanks for quick response. :) Based on the issue I'll try to replace my `fstab` entry with `sysroot.mount`. It'll have dependency on lvm service. So I don't have to use `x-systemd.device-timeout` "hack" anymore.

Do I understand it correctly?

[Andrei-Pozolotin]

@Anty0

Do I understand it correctly?

No. You just wait for @fredleb to test multiple setups in his vm and then give you a proper solution :-)

If you can build virtualbox vm yourself and verify proper solution please do so. Meanwhile continue to use your manual boot workaround, and continue lvm-related conversation in issue #36 .

[Anty0]

Oh, got it, sorry. Thanks. :)

[vaminakov]

@Andrei-Pozolotin I solved the problem.

> 1. what do you mean, exactly? collect logs from `initramfs` boot: I solved the problem. It was that systemd's overwrite unit's config doesnt work. I mean /xxx block in initrd-dropbear.service (so to make it works, need edit /usr/lib/systemd/system/initrd-dropbear.service, systemctl edit... not enough). Maybe it's time to remove it from code and give people to make a choise between dropbear and tinyssh? More, they now disabling/enabling by systemd, not code in unit files. > 2. please try `tinysshd` instead, and report back: I cant use tinyssh, cause I need to use rsa keys. My hardware in some steps have only RSA implementation... p.s. only in some cases. usually I use tinyssh. > remember that `dropbear` and `tinysshd` are mutually exclusive, > unless ran on separate ports, so it is needed to enable one / disable another It is obviously :)))

[Andrei-Pozolotin]

@vvl-rulez

I solved the problem

ok, great.

> Maybe it's time to remove it from code yes, pending in #33 > give people to make a choice between dropbear and tinyssh how do you see that done? * enable both on different ports? * disable both by default, and always require user config action? * or ... ?

[vaminakov]

@Andrei-Pozolotin

* disable both by default, and always require user config action?

I think that will be best variant. Already user need to do"systemctl enable initrd-***", that's enough.

* enable both on different ports?

More services, MORE!!! :))) Let's add openssh, and mkinitcpio-systemd-tool's self realization :) Dropbear covers most cases of technical limits for crypto, tinyssh for other cases (more tiny, faster, easy to config, etc).

[Andrei-Pozolotin]

ok, after #33 all services will be disabled by default after install of systemd-tool

[Andrei-Pozolotin]

to all: you are invited to code review #38 for #33