search

khuedoan / homelab

Fully automated homelab from empty disk to running services with a single command.
https://homelab.khuedoan.com
GNU General Public License v3.0
7.91k stars 708 forks source link

Raspberry Pi Support #57

Open bmatheny opened 2 years ago

bmatheny commented 2 years ago

Hello. Do you know if anyone has successfully used this project with raspberry pi's? If not, any thoughts on (at a high level) what kinds of changes might be needed to support them?

Thanks!

khuedoan commented 2 years ago

I'm not aware of anyone, and I don't have the hardware needed to test. However everything other than PXE boot should work as long as your apps support ARM images.

You can have some RPis with Rocky Linux preinstalled, then remove the boot target in metal/Makefile like this:

- default: boot cluster
+ default: cluster

I'm not sure how PXE boot works on RPi, but here's a starting point if you want to port/improve the current setup:

https://github.com/khuedoan/homelab/blob/9c243c69f5c1840146067a79ab2f32c77be680e3/metal/roles/pxe_server/defaults/main.yml#L1-L2

https://github.com/khuedoan/homelab/blob/9c243c69f5c1840146067a79ab2f32c77be680e3/metal/roles/pxe_server/templates/dhcpd.conf.j2#L16-L18

bmatheny commented 2 years ago

Awesome. I'll have a go at it over the weekend and put up a PR if it makes sense, worst case I'll just comment back here. Thanks!

minhtt159 commented 2 years ago

@bmatheny How is your experience with Rasp Pi? Since PXE booting on Raspberry Pi is kinda different.

Bourne-ID commented 2 years ago

Hey, I've been looking at this and playing around with it for a few days. I had hoped the new HTTP_BOOT (option 7) might have made it easier, IE flash PI boot order, throw some host configs at it and then install over ethernet. However I'm falling short on this -at least for the time being- as downloads need to be signed and the EEPROM containing the public key - which adds a manual step to this process, something I don't want to add.

I'm relatively new at PXE, TFTP and Pi's unique PXE strategy, but I feel there's a reason there's a majority of blog posts giving Pi a diskless PXE instead of a full initialisation and installation. However I feel the latest bootloader may have added some additional options.

My current plan for Pi is to try a diskless boot, network mount and then cloud-init an installation onto the SD card, followed by a reboot. It would require the boot order to be SDCard, Network, Restart which is a pre-step, therefore first boot on an empty SD card would default to Network, and would then hit SD card on the reboot.

Is there a better way? Probably. Does it gel well with this project? Well, time will tell. I think taking the worst case from above, flashing Rocky Linux manually and bypassing the pxe Metal stage may be an option.

No promises. But more as I have it.

mrsauravsahu commented 2 years ago

I have a Raspberry PI 4 with Ubuntu Server running. I'm running into errors, checking if I can fix them and get it running.

clainor commented 2 years ago

Joining effort, I have a bunch of pi 4 and I already have success on a PXE boot on pi 3 diskless (with iSCSi setup and some prep beforehand).

Bourne-ID commented 1 year ago

Quick update on my side - manual steps of Net Boot (w/ blank SD card and NFS) into Rocky, flash the IMG file to local storage and then reboot seems to be a working solution - #70 will allow for multiarch PXE, and an NFS docker pod will be required as a future change.

I still believe booting Rocky over NFS to then flash Rocky from IMG to SD optimisable...

Is anyone interested in investigating the embedded RPI-imager to see if this or a similarly small (512MB) image can be automated to do a flash from network to SD Card? If so then the first PXE boot can be a minimal image and then let the RPI boot from SD Card afterwards with a seed first-boot.sh file into the RPI.

I'm hoping the RPI option 7 Netboot will be a final solution in the future; however right now any HTTP downloads on option 7 needs to be signed and the public keys flashed to the RPI, which isn't an automatic remote process.

(That and IP Based HTTP downloads are currently broken. something to come back to later.)

mrsauravsahu commented 1 year ago

Unfortunately, I've decided to try other ways of setting up my PI - mainly because I want to use Ubuntu on the PI, and terraform to do the setup. This is a great project though.

Bourne-ID commented 1 year ago

Still tapping away at this. I've managed to get RPI to boot through dnsmasq to pick up rpi essential files from TFTP.

I've then managed to use a NFS4 docker image using TCP over custom ports to connect the RPI root FS over NFS (having to use custom ports over 2049 because rpcbind seems to enjoy stealing the port from Docker, at least on my Ubnutu box).

However I'm currently stuck on extracting the root FS using Ansible inside the tools container -

The image for Rocky RPI is a .img.xz, a MBR file with 3 partitions of which we need two (0.fat, boot, and 2.img, root). 0.fat can be extracted through 7z, however the 2.img must be mounted to ensure symlinks are preserved - something which 7z is unable to do on extraction.

But, as ansible is run through the tools container and therefore doesn't have sudo/root access (unless I broke something?), and because mount -o requires root, my usual methods of mounting the img file is being blocked.

I'm wondering if I should go back to see if I can find a mini-linux image - the objective is to net boot just to then do a run-once dd copy to the SD card on the rpi. If there's a mini linux image that I can git clone instead for example, could solve this issue...

Let me know if anyone has any thoughts.

khuedoan commented 1 year ago

as ansible is run through the tools container and therefore doesn't have sudo/root access

This was introduced in https://github.com/khuedoan/homelab/commit/c5602bdb8b522feeaa75a9b80573433d692139be because the files created by the tools container are owned by root:root, but I think you can revert it without side effects (except file permissions)

remy415 commented 1 year ago

I don't know about RPis, but I'm using Khadas VIM3s ATM and am currently playing with the configuration. I originally purchased these because of the A311D SOC with NPU, but currently it seems like the NPU has very little software support so far so AI/ML workloads on SBC = not quite yet. I did find a guide on their site for NPU set up, I'm executing the setup now and will report back later if I have any success. It's a challenge because it seems like anytime I mess something up, I need to boot to the rescue image and re-install Ubuntu.

The VIM3 uses a small rescue image + cloud image installs to re-image. I haven't explored any network-based install methods yet.

Has anyone had success with any ARM64 SBC-based K8S clusters? Can they be added to a standard K8S cluster controller (ie Rancher/etc)?

khuedoan commented 1 year ago

I just stumble upon this Raspberry Pi 4 UEFI Firmware project, which might be a better way to PXE boot because we don't have to add custom logic for the Pi.

It might be a while until I can test this myself though (they are out of stock everywhere in my country, and I refuse to pay the $220+ scraper price :disappointed_relieved:)