Networking does not work with New Arch Installation

[CarstenFeuls]

I have got a Problem with a new Arch Installation. The Network does not start at boot. The Solution for me was to add this users to /etc/passwd. systemd-journal-gateway:x:191:191:systemd-journal-gateway:/:/usr/bin/nologin systemd-timesync:x:192:192:systemd-timesync:/:/usr/bin/nologin systemd-network:x:193:193:systemd-network:/:/usr/bin/nologin systemd-bus-proxy:x:194:194:systemd-bus-proxy:/:/usr/bin/nologin systemd-resolve:x:195:195:systemd-resolve:/:/usr/bin/nologin

and the depending group also to /etc/group.

Before this doing the users are Dynamic user. systemd-resolve::61662:61662:Dynamic User:/:/sbin/nologin systemd-network::63822:63822:Dynamic User:/:/sbin/nologin

I dont know where this users are defined. I think we need to include the file were the dynamic user are defined into the Initialramdisk.

[ShapeShifter499]

I have noticed this too, some Arch Linux upgrade broke something.

[ShapeShifter499]

@Andrei-Pozolotin any ideas?

[ShapeShifter499]

@CarstenFeuls Are you adding these to the rootfs or the initramfs?

[CarstenFeuls]

@ShapeShifter499 I have added the systemd users and groups in /etc off my host system not in the initalramdisk. The problem is i dont known where the missing users are defined in systemd. I think we only need to include the files where the dynamic users are defined.

[ShapeShifter499]

On my system the dynamic users 'systemd-resolve' and 'systemd-network' appear to be defined in the unit files themselves.

http://0pointer.net/blog/dynamic-users-with-systemd.html https://www.freedesktop.org/software/systemd/man/systemd.exec.html

[ShapeShifter499]

@CarstenFeuls Why does this even work?

https://github.com/random-archer/mkinitcpio-systemd-tool/blob/c270db352889e76b5e980660e825d2144ee718d4/initrd-shell.service#L40 AND https://github.com/random-archer/mkinitcpio-systemd-tool/blob/1e90560cc3a2a64cd4425650cbbc85414db739df/initrd-build.sh#L19 suggests those entries you are adding to /etc/passwd would be removed when the /etc/passwd file is copied to the initramfs.

My issue is that my system no longer gets a IP after power on, I cannot ssh in to remotely unlock my device with the scripts in mkinitcpio-systemd-tool. If adding users /etc/passwd and /etc/group work, I'm not sure why. I do know something is working because I can access the serial and it does get to the 'secret>' prompt for unlocking the rootfs encryption.

[ShapeShifter499]

Ok I completely removed the mkinitcpio-systemd-tool package and it's included files THEN I installed it again. I made some edits to the package that I now see was stupid. It is working for me as is again, no edits to /etc/passwd were necessary for my system.

[ml-]

@ShapeShifter499 Long story short: There is no issue with it? Going to upgrade my server now and see if I run into some sort of problems.

[vauvenal5]

@ml- Did upgrading the server fix the issue for you?

[ml-]

@vauvenal5 You got me wrong. I didn't have this issue, but was afraid of upgrading the server because of it. (Info: my server was offline for a long time before)

I just did a full pacman upgrade, everything worked fine after the reboot. I could enter the secret as usual.

[ShapeShifter499]

@ml- no issues here. Everything is working fine for me, glad it's also working for you too.

[vauvenal5]

I had the same behaviour like reported by @CarstenFeuls. More precisely the systemd-networkd and systemd-resolved services did not start because of the missing users. After manually adding them like suggested above everything worked fine.

[ShapeShifter499]

@vauvenal5 On Arch Linux? I just noticed that my /etc/passwd already has the below users. It doesn't include every user mentioned by @CarstenFeuls but it seems to just work. I'm wondering if there isn't maybe a bug somewhere in a package that left some users from being properly added in Arch Linux.

systemd-journal-remote:x:982:982:systemd Journal Remote:/:/sbin/nologin
systemd-network:x:981:981:systemd Network Management:/:/sbin/nologin
systemd-resolve:x:980:980:systemd Resolver:/:/sbin/nologin
systemd-coredump:x:979:979:systemd Core Dumper:/:/sbin/nologin

[vauvenal5]

Yes on a fresh Arch Linux install. I only added the two users that where mentioned in the logs namely systemd-network and systemd-resolve and the according groups. After this it worked. I mean if it is not considered to be a bug it should maybe be mentioned somewhere that adding those users might be necessary. When known it is not a big deal to add them manually.

[ShapeShifter499]

@vauvenal5 I still don't quite understand why this works though. Isn't the scripts setup by mkinitcpio-systemd-tool removing 'systemd' users from /etc/passwd when the file is added to initramfs?

[Ghabry]

I just installed a fresh Arch on a remote system so I can't really see what is going wrong without a remote KVM but adding the two missing users manually to the rootfs passwd solved the network issues for me.

At least the network is up, the ssh server rejects my key "Permission denied (publickey)" but that must be a config error on my side :)

EDIT: Found the reason about ssh: ed25519 keys are NOT supported.

[vauvenal5]

@ShapeShifter499 sry for the late reply. To be honest I haven't looked further into it yet, due to working fix and time constraints.

[suiryc]

Playing with a minimal (console only) ArchLinux fresh install in a VM, I encountered the same issue:

sept. 09 13:46:23 archlinux systemd-networkd[212]: Cannot resolve user name systemd-network: No such process
sept. 09 13:46:23 archlinux systemd-resolved[290]: Cannot resolve user name systemd-resolve: No such process

There seem to be issues with dynamic users in systemd as mentioned in systemd ticket #9503 pointing to debian ticket #902971. Apparently next systemd release should revert to creating those users again, at least until dynamic users work properly.

As for other people here, to make it work I had to manually create the two users (and groups) mentioned in the logs. I actually had to somehow write them in /etc/passwd and /etc/group because useradd complained that those users already existed ...

[ShapeShifter499]

I moved away from Arch Linux ARM to a x86_64 Arch Linux system and I had to make the aforementioned modifications to '/etc/passwd' and '/etc/group'. Does it matter what user and group ID I used for the 'systemd-network' and 'systemd-resolve' users and groups? It would seem that it doesn't as long as I didn't make them the same as a any other user or group ID.

[suiryc]

@ShapeShifter499 As far as I know, what matters is indeed that the chosen IDs are not already used.

[ShapeShifter499]

@suiryc alright cool.

But it seems whoever was maintaining the Arch Linux dropbear package has removed 'dss' and 'dsa' from the supported keys in the latest update. I'm now experiencing issue #21

[suiryc]

It may not be really the case.

dropbearconvert parses the content of the openssl key to determine its type (even though you give it through CLI): https://github.com/mkj/dropbear/blob/d740dc548924f2faf0934e5f9a4b83d2b5d6902d/keyimport.c#L394 So for example if it contains BEGIN RSA PRIVATE KEY, it determines it's rsa, etc. If it does not find a matching type, it fails. When I tested, for some reason (changes in latest openssl version ?) the generated keys did not contain the same text (and thus dropbearconvert could not convert them). I just used old keys to workaround it, but I presume one could try editing the text (at least so that dropbearconvert can recognize the key type).

[ShapeShifter499]

@suiryc nope, that didn't work. I tried changing "-----BEGIN OPENSSH PRIVATE KEY-----" to "-----BEGIN RSA PRIVATE KEY-----" AND "-----END OPENSSH PRIVATE KEY-----" to "-----END RSA PRIVATE KEY-----" I got the below error.

[root@archiso /]# dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key /etc/dropbear/dropbear_rsa_host_key
Error: ASN.1 decoding failure
Error reading key from '/etc/ssh/ssh_host_rsa_key'
[root@archiso /]#

[ShapeShifter499]

@suiryc Going forward, it would seem the only options for me would be. To shoehorn OpenSSH in place of Dropbear for the remote decryption. Or generate separate ssh host keys and have OpenSSH and Dropbear listening on different ports so that my local SSH client won't complain about mismatching host keys.

[ShapeShifter499]

I was seeing errors in my systemd units, specifically with shadow.service. The fix for the errors below is to run 'pwconv' and 'grpconv' to update the added entries from above made in '/etc/passwd' and '/etc/group' to the shadow files '/etc/shadow' and '/etc/gshadow'

[root@kumo ~]# systemctl status shadow
● shadow.service - Verify integrity of password and group files
   Loaded: loaded (/usr/lib/systemd/system/shadow.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-10-25 15:32:56 PDT; 2s ago
  Process: 1179 ExecStart=/bin/sh -c /usr/bin/pwck -r || r=1; /usr/bin/grpck -r && exit $r (code=exited, status=2)
 Main PID: 1179 (code=exited, status=2)

Oct 25 15:32:56 kumo systemd[1]: Started Verify integrity of password and group files.
Oct 25 15:32:56 kumo sh[1179]: no matching group file entry in /etc/gshadow
Oct 25 15:32:56 kumo sh[1179]: add group 'systemd-network' in /etc/gshadow? No
Oct 25 15:32:56 kumo sh[1179]: no matching group file entry in /etc/gshadow
Oct 25 15:32:56 kumo sh[1179]: add group 'systemd-resolve' in /etc/gshadow? No
Oct 25 15:32:56 kumo sh[1179]: grpck: no changes
Oct 25 15:32:56 kumo systemd[1]: shadow.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 25 15:32:56 kumo systemd[1]: shadow.service: Failed with result 'exit-code'.

[alkahan]

By default openssh keys are in “RFC4716” format. To convert your openssh keys, you need to generate them in the PEM format.

ssh-keygen -A -m PEM

You can now convert them with dropbearconvert

[ShapeShifter499]

@alkahan is PEM format insecure?

[alkahan]

PEM is not more or less secure than RFC4716. This is just another way to store the data (private or public keys). The is no security question here.

[Andrei-Pozolotin]

please confirm if there are still issues at v17 https://github.com/random-archer/mkinitcpio-systemd-tool/releases/tag/v17

[ShapeShifter499]

Seems resolved for me here. But I'm not sure what others think.

[Andrei-Pozolotin]

assume resolved