Systemd 247 support

[kris7t]

Since updating to systemd 247.1-1, my mkinitcpio became completely unusable.

I'm running mkinitcpio-systemd-tool in a cloud server to remotely unlock the LUKS encrypted disk via a dropbear SSH server. Previously, with up to systemd 246.6-1, I was able to SSH into the mkinitcpio environment to provide my passphrase, and I could also interact with the environment through the web console of my server.

Since systemd 247.1-1, SSH connections became impossible, the server does not respond to connection attempts (both ssh and netcat just hangs when trying to connect). The web console is also unusable: while I am able to interact with the server when it's still in the GRUB menu, once the initramfs is loaded, it's not responding to key presses (hence it's impossible to provide the LUKS passphrase even over the web console). However, the server is not frozen: the cursor is still blinking, and after the password prompt times out, it tries to start the recovery shell (which fails, because root login is disabled).

Downgrading to systemd 246.6-1 solved the issue for now, but it would be better if mkinitcpio-systemd-tool could be fixed. However, debugging the problem is extremely hard (since I have no way to interact with the server in the failed state).

[dvzrv]

@kris7t Hm, I can not reproduce this with a local hardware setup using a pc-engines APU. I am using tinysshd though. Maybe you can try that? It may be dropbear related.

[kris7t]

@dvzrv I tried switching from dropbear to tinyssh, but I'm getting the same phenomenon on systemd 247. However, both dropbear and tinyssh run fine on systemd 246.

[ArchangeGabriel]

I know this is not really helpful, but no issue for me either with tinyssh on systemd 247. Maybe something specific to your network configuration?

[kris7t]

@ArchangeGabriel I was using the default initrd-network.network and it was working so far. Do you have any clue what should I check about the network configuration?

[ArchangeGabriel]

So dhcp? I’m using a fixed IP instead, maybe that is the difference between us. If you’re able to try a fixed IP, we could at least eliminate that possibility:

[Match]
Name=eth0

[Network]
Address=<IP>
Gateway=<GW_IP>

[dvzrv]

@kris7t this is just an idea, but maybe your initrd now uses unique interface names?

The default only does DHCP for non-unique interface names: https://github.com/random-archer/mkinitcpio-systemd-tool/blob/29ff01a854193f0cbc9710fe036734a43b54ed7c/src/initrd-network.network#L11

You could extend that by

Name=en*

(or a specific interface name)

I actually use a unique interface name (enp1s0) and non-unique interface name eth0 for the match.

[kris7t]

@dvzrv I changed that to Name=eth* en*, with no avail (264 still works, but 247 doesn't). But I'll try just Name=en* some time later.

[kris7t]

After enabling systemd.journald.forward_to_console in the kernel command line, I noticed the systemd-networkd has failed to add the default route and complained about the gateway being unreachable: eth0: Could not set route: Network is unreachable.

Turns out Hetzner is giving out /32 IP addresses, but the gateway is always 172.31.1.1, which is obviously not on the subnet. Previously, this was working fine, but now I had to set up the routes manually and set GatewayOnLink=yes to allow the default route to be added by following https://serverfault.com/a/948351:

[Match]
Name=eth0

[Network]
Address=<IPv6 address>/64
Address=<IPv4 address>/32
DNS=2a01:4f8:0:1::add:1010
DNS=2a01:4f8:0:1::add:9999
DNS=2a01:4f8:0:1::add:9898
DNS=213.133.98.98
DNS=213.133.99.99
DNS=213.133.100.100

[Route]
Gateway=fe80::1

[Route]
Destination=172.31.1.1
Scope=link

[Route]
Gateway=172.31.1.1
GatewayOnLink=yes

This is a bit puzzling, since up until systemd 246, systemd-netword was able to accept this gateway over DHCP and correctly add the default route (and a link-local route to the gateway). But now I had to revert to a static network configuration for both mkinitcpio and my normal networkd setup (which became identical to the one above, but with eth0 replaced with enp0s1).

Anyways, I guess we can close this issue, since it's not actually caused by mkinitcpio-systemd-tool, but rather by my VPS provider's idiosyncratic network setup and some systemd 247 DHCP changes.

Thanks for all the help and the suggestions to look at the network configuration!

[ArchangeGabriel]

Bug in upstream systemd: https://bugs.archlinux.org/task/68862, https://github.com/systemd/systemd/issues/17803.

Should be fixed in a future release, since the fix has already been merged.