Not an issue, more of a recommendation: try ARMbian as a base for Proxmox on RK3588 based SBCs

[abufrejoval]

I've been using VMware since 1999, openvz containers since 2008, oVirt/RHV since 2016 and Proxmox since 2020...

x86 was the base of my IT career since 1984, but I'm much readier to embrace ARM and RISC-V than clouds, so VMs and containers on ARM sounds great, while Proxmox has not just become my favorite HCI choice, but pretty near the last man standing for a hyperconverged infrastructure you might host yourself.

With the Orange PI 5+ including 32GB of RAM, ARM is as close to the Atom (32GB each) and NUC (64GB each) based RHV and Proxmox clusters (typically 3 node HCI with GlusterFS or Ceph) I run in my home lab so I've started to fiddle a bit with both an RP5 and an OP5+.

For the OP5+ the vendor images actually don't seem to be maintained all that well: I had issues mostly with GPU support and Wayland there.

But then I found ARMbian images, evidently based in Slovakia with very international sponsors, which seem to be quite well maintained (Vulkan is still missing) and are somewhat less under direct Chinese control, which these days is a blank security risk, as open source becomes yet another battleground that no developer can escape. They seem to support a very broad range of SBCs.

So, first of all: thank you ever so much for your effort, I'll try to dig in and test as time and budgets permit!

And second: if you want a good and practical starting base, the Orange PI5+ with 32GB and ARMbian seem a good combination for something that isn't just academic, but powerful enough to have practical use. I can't wait to manage x86 and ARM based SBCs under a single management pane using Proxmox...

[jiangcuo]

I don't have a raspberry pi or rockpi soc device. I only have a generic arm pc motherboard(uefi boot). So I'm a little bit helpless about those。

[crazyquark]

That's what I did as well(Orange Pi 5+). I built a custom Armbian with kernel 6.1 and ran on top of that. I am thinking of trying edge(kernel 6.8). I previously was running on EDK2 + generic ISO but I had stability issues and I had no sensors in the firmware... I might go back to that when it matures more.

[abufrejoval]

I don't have a raspberry pi or rockpi soc device. I only have a generic arm pc motherboard(uefi boot). So I'm a little bit helpless about those。

I'd have wanted one of those, too, but I was unable to find such a thing at a reasonable cost/performance level and that's why I resorted to the RP4/5 and now an Orange PI 5+ as the closest available equivalent.

I've seen Ampere based workstations coming out and stuff like SolidRun before, but those aren't exactly price competitive against x86.

So I'd really be interested in what you have there...

But back to the topic:

On x86 Proxmox I've used both the ISO provided by Proxmox, which has some distinct advantages like ZFS boot support, and a standard Debian install with the Proxmox repo and packges added on top.

I guess the latter approach should also work with what you do, at least once whatever kernel dependencies might be required are worked out and the repo logistics are settled.

[abufrejoval]

That's what I did as well(Orange Pi 5+). I built a custom Armbian with kernel 6.1 and ran on top of that. I am thinking of trying edge(kernel 6.8). I previously was running on EDK2 + generic ISO but I had stability issues and I had no sensors in the firmware... I might go back to that when it matures more.

This looks rather promising: What did you have to add the the kernel? Was it KVM support?

On x86 it's standard on pretty near any recent distribution, but I noticed that the standard KVM packages I see on x86 Debian weren't even listed for ARMbian/Jammy, while there is plenty of Xen stuff listed.

While I remember Xen and KVM being ported to ARM ages ago (and nearly all cloud ARM servers supporting some hypervisor), I can see why it wouldn't be standard on "PI" class SBC kernels, but I'd have assumed the KVM support packages to be there...

[abufrejoval]

I've downloaded the UEFI firmware for the RK3588 and put that on an SD-card.

Then connected two USB sticks, one (slow) with the PVE ISO written via Rufus/DD to it and aother (SSD-quality and big) to install PVE on...

It just worked! Orange PI 5 Plus booted the UEFI firmware, found the PVE ISO and started the Proxmox install.

Now I'm just having issues finding an ISO and settings for runing VMs, containers should be easier I guess.

I could even add it to an existing PVE Cluster with x86 members, but I'm not expecting live-migration to work between them (I'd love to see how it reacts...)

[abufrejoval]

I now did the same with a Raspberry PI 5: works just as well!

[Anexgohan]

I now did the same with a Raspberry PI 5: works just as well!

could you point me to the UEFI firmware your talking about, I would love to not have PiOS lite running in the background kernel.

[abufrejoval]

For the OP5 you'll find it here: https://github.com/edk2-porting/edk2-rk3588/releases and for the RP5 it's this one: https://github.com/worproject/rpi5-uefi

I put them on a µ-SD card that's formatted with fat32 and then put the ISOs on a smaller USB3 stick e.g. via Rufus or 'dd' while I have a bigger "SATA-class USB3" stick ready to host Proxmox.

Using a RealTek USB3 2.5Gbit NIC on the RP5 gets around the issue of non-working Raspberry Ethernet support while the dual 2.5 Gbit ports on the OP5+ work just fine out of the box.

On the RP5 I use "force_turbo=1" in the 'config.txt' to get some semi-useful speed, on the OP5 you're supposed to be able to set that within the UEFI, but unfortunately hitting ESC isn't recognized... still working on a solution there, because at 800MHz the OP5 is very little fun.

[titogrima]

Hi!!

For orange pi 5 I used ARMBIAN OS

https://www.armbian.com/orangepi-5/

Download Orange pi 5 https://dl.armbian.com/orangepi5/archive/Armbian_24.2.1_Orangepi5_bookworm_legacy_5.10.160.img.xz

Download Orange pi 5 plus https://fi.mirror.armbian.de/dl/orangepi5-plus/archive/Armbian_24.2.3_Orangepi5-plus_bookworm_legacy_5.10.160_minimal.img.xz

And install it on high speed sdcard with dd

And install on top Proxmox https://github.com/jiangcuo/Proxmox-Port/wiki/Install-Proxmox-VE-on-Debian-bookworm

The only problem I noticed is the pve2-edk package that is need downgraded https://github.com/jiangcuo/Proxmox-Arm64/issues/62

And CPU affinity that is necesary to set in the same group of CPU (0 to 3 for low performance CPU 4 to 7 for high performance CPU)

I trying 6.8 kernel that fixed both problems but appears others.....

For storage I use nvme with GlusterFS in the same three Orange PI 5 Proxmox cluster for now but maybe I will migrate this to Ceph

And yes for me have practical use with OpenWrt, Suricata IPS, email server, DMZ and more in multiple VM

sorry for mi english XD

[abufrejoval]

I've progressed a little, too.

background note:

• I also operate a Proxmox HCI cluster based on three Intel x86 NUCs, which provide Ceph storage (replica 3).

• And then I run ARM Proxmox also on a Raspberry 5 8GB: I really want to have live-migration work between the ARM nodes as well (just like it works on x86).

Since the UEFI boot on the OP5+ gave me a bit of grief with a slow clock that I couln't change etc. I tried using the 'normal' kernels and installing Proxmox from the repos instead of the ISO.

I went with the original OrangePI Debian BullsEye with the 6.1.43 kernel, mostly because I wanted to test it vs. Armbian.

The OrangePI variant comes with a lot of vendor extras and gives much more control over all hardware. It also matches the very extensive and up-to-date documentation: I've been really impressed!

Then I followed jiancuo's excellent guide for installing Proxmox from his repos.

The only snag there was that I had to set up the vmbr0 bridge manually, because for some reason the Proxmox installation didn't do that.

After that it worked and I had it join the existing cluster with the x86. No, I don't expect live migration to work across CPU different architectures, but I'd like to administrate both types of servers from a single pane and I want to take advantage of the existing Ceph storage, backup facilies etc., especially since I only have 2 ARM machines.

Anyway, joining the cluster was no problem, but when you create VMs you need to use the WebGUI on a matching machine: you can't create an ARM VM on an x86 host and vice-versa.

I installed Ubuntu Jammy as a VM without issues using the x86 HCI Ceph for storage and booting off ISOs hosted on NFS there, making sure I had the downgraded OVFM firmware on the OP5+ (I tested vs the current, which didn't work).

I found I had to disable secure boot in the OVMF "UEFI" setup or the newly installed system could not be booted.

What I can't get to work, is live migration. The RAM gets copied ok, but the machine then fails to start on the target host. I didn't find any obvious error messages in the logs (migrating a stopped machine is a snap, especially with Ceph, where it's little more than copying the metadata).

I first thought it had to do with the CPU type being set to "host", which normally precludes live-migrations. I tried downgrading to A76, but then the VM won't even start. Next I tried setting it to "max", which allowed the VM to start, inside it saw an A76 CPU, but then live migration failed the same.

But before I spend more time investigating, I'd just like to know if anyone has already had luck live-migration VMs on Proxmox for ARM.

Promox isn't (yet?) designed to handle distinct CPU architectures, so mixing AARCH64 and AMD64 can lead to quite a bit of grief that is hard to diagnose. But when you're careful, it works pretty much as well as can be expected: I guess it helps that nearly everyone is little-endian these days.

It's soo much fun!

[abufrejoval]

On the Raspberry 5 things turned out even easier: I switched from doing an UEFI boot of the ISO provided here to simply using the standard Raspi OS and adding the packages from the repo and I only had to do the steps jiangcuo documented really well.

Switching to 4k page size just involves using the non-RP5-optimized kernel by "kernel=kernel8.img" in /boot/firmware/config.txt and then for some reason I again had to add the bridge network manually.

Unfortunately, I am still out of luck live migrating VMs, so if anyone with more than one ARM could give it a try, I'd be glad to hear about it.

And yeah, I wish there was an easy and affordable way to have Raspis with 32GB of RAM, like my OP5+, once you start to virtualize with VMs that extra RAM does come quite handy! Or that live migration worked with LXC containers, I know there is a variant for OpenVZ...

[abufrejoval]

Since following the steps outlined by jiangcuo for a repo installation pretty much solved the problem, I am closing this issue, because I evidently just overlooked the documentation.

Thanks a lot jiangcuo for your excellent work!