ssh keys sometimes not present

[rajil]

I am doing a build multiple times to tweak my system. Sometimes, i have noticed that the ssh keys in /etc/ssh/ are not generated. In one of the tries, there wasnt any key at all and therefore i wasnt able to login into the RPI.

For instance in my current build, i do see these messages

Creating config file /etc/ssh/sshd_config with new version
Creating SSH2 RSA key; this may take some time ...
2048 SHA256:Xt90dVJCurKQ5BD5Zvq7Z4B5pcQicvgaulKra/jo/tI root@debian (RSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:RmIOfx6tdG3XO+3LkqzCka5vzOAdyhGGHdv0tyNAKIs root@debian (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:RGoRqA6nhropKepp/nceXvPCGF/vnTHsaxSSZpn9flE root@debian (ED25519)
qemu: Unsupported syscall: 384
Created symlink /etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service.

However, after boot the RPI i dont see the ECDSA and ED25519 keys,

# ls -la /etc/ssh/
total 568
drwxr-xr-x  2 root root   4096 Oct 16 01:35 .
drwxr-xr-x 82 root root   4096 Oct 16 02:03 ..
-rw-r--r--  1 root root 553122 Jun 18 05:41 moduli
-rw-r--r--  1 root root   1723 Jun 18 05:41 ssh_config
-rw-r--r--  1 root root   3284 Oct 16 01:36 sshd_config
-rw-------  1 root root   1679 Oct 16 01:35 ssh_host_rsa_key
-rw-r--r--  1 root root    396 Oct 16 01:35 ssh_host_rsa_key.pub

Is this expected behaviour?

My plugins.txt is

plugins/Raspbian/copies-and-fills
plugins/Raspbian/kernel
plugins/Raspbian/rpitools
plugins/Foundation/apt-key
plugins/stretch/extrapackages
plugins/stretch/noPersistentNetGen
plugins/stretch/oldNetNaming
plugins/alsa
plugins/common
plugins/dbreconfig
plugins/disableroot
plugins/extrapackages
plugins/fake-hwclock
plugins/fsckboot
plugins/hostname
plugins/kodi
plugins/mythtvlight
plugins/resizefs
plugins/rpi-next
plugins/sshkeys
plugins/swap
plugins/tmpfs

[TheSin-]

they should never be present they are generated on first boot so you can use the image on multiple systems. If it's bailing before the end they might still exist, but when then image is done they should never be there with the plugin/sshkeys being run.

[rajil]

Which script generates the keys on the pi on firstboot?

It seems the key generation is a bit random. I have another instance here where the only key in /etc/ssh is ssh_host_dsa_key and ssh_host_dsa_key.pub. The rsa/ecdsa/ed25519 keys were not generated. I get a 'Connection reset' when i try to ssh, probably due to lack of rsa host key.

I tried systemctl status sshd_check_keys and it reported Active: inactive (dead) Condition: start condition failed.

Only way to recover from this was to do a ssh-keygen -A.

[TheSin-]

it shouldn't be it's using the openssl script to build em

https://github.com/TheSin-/rpi-img-builder/blob/master/plugins/sshkeys/files/etc/systemd/system/sshd_check_keys.service

[rajil]

Maybe the sleep time is not enough, ExecStart=/bin/sleep 15

[TheSin-]

Or too much and the restart happens too early?

[rajil]

On an idle system i get 6 seconds,

# time ssh-keygen -A
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 

real    0m6.646s
user    0m6.630s
sys     0m0.010s

Are the ExeStart scripts meant to run sequential or parallel? If it is the former than the sleep time should be more.

[rajil]

I made two builds,

• ExecStart=/bin/sleep 30 produced rsa and dsa keys
• ExecStart=/bin/sleep 8 correctly produced all the keys

So, shorter time is better.

[TheSin-]

wonder if the timer is required at all, the way systemd works I'm not sure it would be. I took this script from some place maybe I should rework it. I won't be able to run tests till next week but I think I'll just need to use preStart and Start

[TheSin-]

try that commit see if it helps with consistency and let me know

[rajil]

Unfortunately, the new changes made it worse. I dont get a key generated on the RPI.

Environment=TERM=linux
Type=oneshot
ExecStartPre=/usr/bin/ssh-keygen -A
ExecStart=/bin/sleep 10
ExecStartPost=/usr/sbin/service sshd restart
StandardError=syslog
RemainAfterExit=no

[TheSin-]

it wouldn't happen till the last reboot.

as it reboots it'll remove files from /var/run/system-... sshkeys won't get till all those files are gone. So maybe it needs one more reboot to gen, I'll revisit the logic, that last files might be getting removed after the keys try to gen but no reboot happens.

if that isn't it can you boot and run systemctl -l status sshkeys.service for me and paste the results.

[rajil]

There were no files /var/run/system-*. I rebooted but did not get any keys. Here are photos i captured after the reboot. I suspect the disk check is causing them to fail.

[TheSin-]

why is your fs ReadOnly??

it's trying to run but the fs is RO

[rajil]

I have no idea. This is a clean boot after dding the image. The dd should have wiped out the previous filesystem. It is surprising that the disk check has kicked in somehow. Maybe after the first boot, the RPI rebooted and corrupted the filesystem.

[TheSin-]

did you enable the readonlyfs plugin?

Also fsck=yes is in the default codlin.txt, being and SD card I like to check every boot

[rajil]

I rebooted again, and this time it generated the keys.

[rajil]

No, readonlyfs is disabled. I will continue to keep the fsck option.

[TheSin-]

okay just making sure. Cause I'll have to rework that one for the changes in sshkeys ;)

I don't like that you had to manually reboot, I'll look deeper into the order