stuck at generating ssh keys

[builderjer]

I hope you can help me. I am trying to build a custom image for a voice assistant. I want to start with a "lite" image and go from there. So, I thought that I would JUST build the "lite" image and make sure it all works like I want and then expand from there.

My problem is, on first boot, it goes through the boot cycle and show the screen "resizing image..." but when it switched to the "generating ssh keys", it gets stuck. I have a very basic config file as I will expand from this once this works.

My config file

IMG_NAME="ovos-dev"
STAGE_LIST="stage0 stage1 stage2"
ENABLE_SSH=1

and I removed the "EXPORT_NOOBS" file from "stage2" but left the "EXPORT_IMAGE" file

I'm not sure what I am doing or not doing to let the pi reboot on first boot

Any help would be much appreciated

[jimmypw]

I also have this issue. I've done some debugging and the hang is caused by:

/usr/lib/raspberrypi-sys-mods/regenerate_ssh_host_keys:4 dd if=/dev/hwrng of=/dev/urandom count=1 bs=4096 status=none

then again by

/usr/lib/raspberrypi-sys-mods/firstboot:66 DISKID="$(tr -dc 'a-f0-9' < /dev/hwrng | dd bs=1 count=8 2>/dev/null)"

It's almost as if /dev/hwrng is not producing anything however if left for long enough it will eventually gather enough entropy and the script moves on.

Still digging.

[jimmypw]

There's a regression in raspberrypi/linux (discussion https://github.com/raspberrypi/linux/issues/5390) that seems to have broken hwrng in pi's prior to version 4. The offending patch has been reverted however there's no date yet to release the updated package through apt.

Until a fix is released pi-gen based builds will only work reliably on pi4. I'm using a pi3 a+ and I can't wait an indefinite amount of time for a fix so I'm going to look for a work around.

edit: spelling

[builderjer]

Thank you for the response, I will also look into this, as I really need it to get past this point.

[builderjer]

could regenerate_ssh_host_keys be disabled? and generate them with another script after it boots? I have been trying to figure out how to do that also.

[jimmypw]

It wouldn't be wise to disable those functions completely. I'm working on a patch to use /dev/urandom instead. I'll share it once i've tested it.

[jimmypw]

Worked first time (what did I do wrong lol)

3 files in total. Place them under stage2/01-sys-tweaks

1) 00-patches/90-regenerate_ssh_host_keys.diff

--- a/rootfs/usr/lib/raspberrypi-sys-mods/regenerate_ssh_host_keys  2023-03-26 15:00:41.355880751 +0100
+++ b/rootfs/usr/lib/raspberrypi-sys-mods/regenerate_ssh_host_keys  2023-03-26 15:02:49.028446093 +0100
@@ -1,8 +1,5 @@
 #!/bin/sh -e

-if [ -c /dev/hwrng ]; then
-  dd if=/dev/hwrng of=/dev/urandom count=1 bs=4096 status=none
-fi
 rm -f /etc/ssh/ssh_host_*_key*
 ssh-keygen -A > /dev/null
 systemctl -q disable regenerate_ssh_host_keys

2) 00-patches/90-firstboot.diff

--- a/rootfs/usr/lib/raspberrypi-sys-mods/firstboot 2023-03-26 15:00:41.355880751 +0100
+++ b/rootfs/usr/lib/raspberrypi-sys-mods/firstboot 2023-03-26 15:02:49.028446093 +0100
@@ -63,7 +63,7 @@
 fix_partuuid() {
   mount -o remount,rw "$ROOT_PART_DEV"
   mount -o remount,rw "$BOOT_PART_DEV"
-  DISKID="$(tr -dc 'a-f0-9' < /dev/hwrng | dd bs=1 count=8 2>/dev/null)"
+  DISKID="$(tr -dc 'a-f0-9' < /dev/urandom | dd bs=1 count=8 2>/dev/null)"
   fdisk "$ROOT_DEV" > /dev/null <<EOF
 x
 i

3) 00-patches/series

Append the following...

90-regenerate_ssh_host_keys.diff
90-firstboot.diff

[jimmypw]

Probably worth noting to anyone that may need this. The code above marks a downgrade of security. It's not insecure but it shouldn't be used where high grade entropy is required.

[builderjer]

testing now. bout an hour and half. Thanks for this!!

[builderjer]

Nice!!! This fixed it. One other question, is it required to be placed in that specific spot? or can I create a stage that executes after stage2? as it is now, I do not need to fork this repo, I just add to it with external stages. If possible, I would like to keep it that way.

[jimmypw]

Feel free to put the patch in wherever works for you. I edited my answer to keep the patches close to where rasperry-pi-sys-mods are installed to keep in the spirit of this repo but in practice you'd probably want the patches somewhere that's easy to disable. That said one of the final steps of stage2 is probably the correct place. The LITE image is built off the contents of stage2 AND raspberry-pi-sys-mods isn't installed until early in stage2. If we do the fix any later it won't make it in to the lite image. Any earlier and the files won't be there to patch.

Glad the fix worked for you too. Just remember that at some point in the near future this problem will be fixed and these patches won't be necessary. I'll try and remember to put a note here once I see a fix released.

[KwadFan]

Want to mention https://github.com/RPi-Distro/raspberrypi-sys-mods/pull/71 here. So, will be fixed in newer releases.

[builderjer]

@KwadFan does that merge fix the issue? If so, we can close this

[KwadFan]

Yes, it should.

[builderjer]

but still use arm64 branch, correct?

[builderjer]

this does not work for me. using arm64 branch, still sticks on generating ssh keys If I apply the patch above, it works fine

[KwadFan]

Well, I changed the code according to @pelwell 's mentioning. But it doesnt stuck at generating keys. it stucks at the newer screen "Fix PARTUUID" if thats the case. Generating SSH Keys from urandom is not 'secure' enough in my humble opinion, in my test it takes up to 3 Minutes to fully generate an SSH Key.

[pelwell]

You must not be running one of our newer kernels with the fix.

[jimmypw]

There has been a new version of the kernel released to the official repos. I've tested /dev/hwrng on a pi3 and it's back to spewing out entropy. I've not tested with a pi-gen build yet but it looks promising that this issue has been resolved and the patch is no longer necessary.

sudo dd if=/dev/hwrng of=/dev/null
^C794+0 records in
794+0 records out
406528 bytes (407 kB, 397 KiB) copied, 3.65846 s, 111 kB/s

[builderjer]

I will build one with pi-gen and make sure. Last one I tried, it still wasn't working. Thanks for keeping me updated

[builderjer]

I have confirmed that the new kernel fixes this issue. It works with the pi-gen arm64 branch. Closing this, thank you for all your help