search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
5.01k stars 981 forks source link

HAOS for ARM/AArch64 CPU (ESXi Arm Edition) #1050

Closed shirou93 closed 2 years ago

shirou93 commented 3 years ago

Why not release hassos ova for arm? VMware released ESXi Arm Edition and RPI4 8gb make sense.

Is it possible convert rpi.img to working vmdk?

agners commented 3 years ago

@shirou93 patch welcome :smile: Yeah you can convert the disk image to a vmdk and try create a machine which boots. The utility qemu-img can do that.

ipha commented 3 years ago

I tried building a generic arm vm image, but I couldn't get a bootloader working. Best I can tell barebox doesn't support arm EFI.

GRUB could be an option, but I don't know if it supports the A/B partitions.

shirou93 commented 3 years ago

@ipha @agners i try but not booting. Maybe bootloader was broken when convert img to vmdk.

agners commented 3 years ago

Hm, yeah bootloader is a problem: We use U-Boot in all Arm builds. It seems that ESXi Arm is expecting a UEFI bootloader such as Grub2. Gleening at this blog post it seems that booting something else is possible in theory, but it seems that requires some hacking: https://blogs.vmware.com/arm/2020/10/23/uefi-less-vms-with-esxi-arm-fling-1-1/

myericbaba commented 3 years ago

anyone try to follow this guideline for HA? https://www.virtuallyghetto.com/2020/10/how-to-run-raspberry-pi-os-as-a-vm-on-esxi-arm.html

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Grandma-Betty commented 2 years ago

I am interested in this too. Would be so nice to run HA OS on ESXi ARM! Edit: Made the feature request here: https://community.home-assistant.io/t/support-haos-for-vmware-esxi-arm/365534

agners commented 2 years ago

As noted in #1549, there are now images named generic-aarch64 which support UEFI boot on arm64. I haven't tested them on ESXi for ARM, I am keen to here if the image also works on ESXi.

There are development builds available at https://os-builds.home-assistant.io/8.0.dev20220223/.

Grandma-Betty commented 2 years ago

As noted in #1549, there are now images named generic-aarch64 which support UEFI boot on arm64. I haven't tested them on ESXi for ARM, I am keen to here if the image also works on ESXi.

There are development builds available at https://os-builds.home-assistant.io/8.0.dev20220223/.

Just tried it with the same settings as HAOS runs on ESXi-ARM, without any luck. I have used the .vmdk disk image as well as the .img file from source. I also tried to play around with hard disk controller settings too, no luck. It's not booting.

agners commented 2 years ago

@Grandma-Betty you did use the generic-aarch64 images right? Did anything come up at all (Grub boot loader maybe?)

Grandma-Betty commented 2 years ago

Yes, I used the said images. No, it was not even reaching a boot loader. With the .vmdk image I was not even able to start the VM itself from the Hypervisor. With the .img file the VM did of course start but was waiting forever for PXE boot.

agners commented 2 years ago

Can you configure the UEFI BIOS in ESXi somehow? Maybe secure boot or something?

tweetysat commented 2 years ago

Also tired with both (latest) generic vmdk and ova vmdk. After adding the disk, changed sata to ide and verified that the boot is EFI, the vm is not starting. It tells me : Failed - Disconnected from virtual machine. And :

Power On VM Key haTask-2-vim.VirtualMachine.powerOn-4294853845

Description Power On this virtual machine

Virtual machine HomeAssistant OS State Failed - Disconnected from virtual machine.

Errors Disconnected from virtual machine. Failed to establish transport connection (9): There is no VMware process running for config file /vmfs/volumes/5f4e56d3-d3071bfa-bf11-dca632e04cec/HomeAssistant OS/HomeAssistant OS.vmx. Remote disconnected

Grandma-Betty commented 2 years ago

Yes, I had the same issue. Just forgot to report back, apologies @agners.

Grandma-Betty commented 2 years ago

Hm, yeah bootloader is a problem: We use U-Boot in all Arm builds. It seems that ESXi Arm is expecting a UEFI bootloader such as Grub2. Gleening at this blog post it seems that booting something else is possible in theory, but it seems that requires some hacking: https://blogs.vmware.com/arm/2020/10/23/uefi-less-vms-with-esxi-arm-fling-1-1/

Any news on this? Seems pretty interesting!

agners commented 2 years ago

No updates, I don't plan to look into ESXi Arm myself.

Home Assistant OS uses regular GRUB2 built for aarch64. The bootloader is present on the ESP (EFI System Partition) as it is common in UEFI boot flows. ESXi really should be able to pick up that GRUB2 and start booting, it's unclear to me why it can't.

Unfortunately I haven't seen a useful log so far. If that is really all ESXi is logging, then its pretty useless to be honest :cry:

I recommend using KVM, as it works and offers also logging which helps us developer to debug the problem. Plus its open source :muscle:

Grandma-Betty commented 2 years ago

@agners Well, I was able to get the time now to try again. I located more VM specific logfiles which I attached. Maybe this gives you any hints in any way? I was trying to boot with the HAOS .img file btw. (latest version from here) Thank you! vmware.log vmware-1.log vmware-2.log vmware-3.log

agners commented 2 years ago

Are those multiple boots? In log 1-3 it seems to crash similarly every time:

2022-05-12T11:37:22.420Z| vcpu-0| I125: VIDE: Register ide0:0) vscsiID = 0, channel = 56
2022-05-12T11:37:22.420Z| vcpu-0| W115: [0 ] bad0003          [1 ] fffffccf92e0     [2 ] 0                [3 ] 427f00000000    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [4 ] 0                [5 ] 128cfd           [6 ] 451a07ca1900     [7 ] ffffffffff758116
2022-05-12T11:37:22.420Z| vcpu-0| W115: [8 ] 2                [9 ] 5                [10] 2                [11] 451a1a4a19e8    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [12] 451a07ca19e8     [13] 420040800080     [14] 41ffd0dd7c08     [15] 8               
2022-05-12T11:37:22.420Z| vcpu-0| W115: [16] 1                [17] 4                [18] 3                [19] fffffcca12a8    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [20] 0                [21] 0                [22] fffffcca1208     [23] fffffc044500    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [24] 0                [25] fffffc044000     [26] fffffcca1208     [27] 0               
2022-05-12T11:37:22.420Z| vcpu-0| W115: [28] 0                [29] fffffc407f90     [30] fffffc04448c    
2022-05-12T11:37:22.420Z| vcpu-0| W115: pc fffffc0444e4 sp fffffc407f30 cpsr 40000248
2022-05-12T11:37:22.420Z| vcpu-0| W115: (relative to vmm .text fffffc000000: pc 444e4 x30 4448c)
2022-05-12T11:37:22.420Z| vcpu-0| W115: Backtrace 'ASSERT':
2022-05-12T11:37:22.420Z| vcpu-0| W115: [ 0] fffffc009324
2022-05-12T11:37:22.420Z| vcpu-0| W115: MONITOR PANIC: vcpu-0:VERIFY devices/vide/iovmk/videVMK-vmkio.c:1904
2022-05-12T11:37:22.420Z| vcpu-0| I125: Core dump with build build-19076756

From the logs, it seems there are two disks attached?

...
ide0:0.fileName = "HAOS_ARM_test.vmdk"
...
sata0:1.fileName = "/vmfs/volumes/6203b160-68b10596-af83-ac1f6b96bfb0/haos_generic-aarch64-8.0.dev20220510.img"
...

Can you try attaching just the vmdk and just the img once? Ideally, as a S-ATA drive, but it probably doesn't matter.

Grandma-Betty commented 2 years ago

Yes, those are multiple boots. I tried to boot with IDE controller first, but the VM does not even start with IDE. Seems like the "Other 4.x or later Linux (64-bit)" template does not support IDE controllers:

Capture

At last I tried SATA controller and then the VM started but never reached the GRUB2 bootloader. The latest logs should be with the SATA controller setting.

Grandma-Betty commented 2 years ago

Wait, let me create the VM again and only boot once with the correct settings. I'll post the new logs in a minute.

agners commented 2 years ago

The last log (vmware-3.log) does has that vdmk file still, I am not sure what that is/if that interferes maybe:

...
ide0:0.fileName = "HAOS_ARM_test.vmdk"
...

Is HAOS_ARM_test the name of the VM? Is that something automatically generated?

Grandma-Betty commented 2 years ago

HAOS_ARM_test is the name I have given manually to the VM. Here are the new logs after creating the VM and only booting once: vmware.log

agners commented 2 years ago

It seems the system doesn't find anything bootable. However, there is clearly a EFI System Partition, a long with the proper boot files.

2022-05-12T12:33:33.598Z| vcpu-0| I125: [msg.Backdoor.OsNotFound] No operating system was found. If you have an operating system installation disc, you can insert the disc into the system's CD-ROM drive and restart the virtual machine.

There could be two reasons a) The VM is trying to boot some other method (not UEFI) b) It is trying to boot via UEFI, but somehow does not accept the default location /EFI/BOOT/BOOTAA64.EFI. It migth be related to #1760. If that is the case, booting a Live Ubuntu CD-ROM (for Aarch64, not sure if that exists) and try to write to use the work around suggested in https://github.com/home-assistant/operating-system/issues/1760#issuecomment-1107654728 might help (replace bootx64.efi with bootaa64.efi).

Do you see something on the screen, like a BIOS or something?

Grandma-Betty commented 2 years ago

Yes, after a while not booting the system goes automatically into BIOS. But I can force it too to enter the BIOS: Capture

agners commented 2 years ago

Yeah from the logs and that screenshot it looks like its an EFI BIOS. So that leaves us with option b): For some reason it does not pick up /EFI/BOOT/BOOTAA64.EFI.

As reported by #1760, some x86-64 UEFI BIOS require an entry in EFI variables for internal disks. You can try creating such variables using a Live Distribution and efibootmgr as well.

Alternatively, maybe you can make the drive to be considered as "external disk". Can you choose a USB as virtual disk controller? Or maybe attach the vmdk file as a CD-ROM?

Grandma-Betty commented 2 years ago

Alternatively, maybe you can make the drive to be considered as "external disk". Can you choose a USB as virtual disk controller? Or maybe attach the vmdk file as a CD-ROM?

No, both of your questions won't work in ESXi.

As reported by #1760, some x86-64 UEFI BIOS require an entry in EFI variables for internal disks. You can try creating such variables using a Live Distribution and efibootmgr as well.

I have to check if I can get this to accomplish somehow. I'll report back as soon as I have more information but it will take a little. Thanks so far for looking into the logs and giving your thoughts, very appreciated mate!

Grandma-Betty commented 2 years ago

@agners Good news! I worte a comment yesterday on the ESXi ARM website and an ESXi ARM dev will also have a look into it: https://flings.vmware.com/esxi-arm-edition/comments

claplace commented 2 years ago

Hi @agners and @Grandma-Betty, I am the ESXi ARM dev :-) so I created a new VM, set the type to Linux/Other4.x, removed the new hard drive, added the existing haos.vmdk, and started the VM. It worked!

Okay there's one thing I've done, it's importing the vmdk into ESXi. If you want the vmdk to be on the native ESXi VMFS partition, it needs to be imported to be usable. Here, from a NFS storage:

vmkfstools -i /
vmfs/volumes/nfs/haos_generic-aarch64-8.0.dev20220223.vmdk  /vmfs/volumes/datastore1/haos/haos.vmdk

If you're using the vmdk directly from NFS this shouldn't be necessary.

claplace commented 2 years ago

In other tries I noticed that using a NVMe virtual drive doesn't work, linux doesn't find the root partition. But as a SATA drive it's fine.

Grandma-Betty commented 2 years ago

@claplace Nice! So importing the .vmdk via esxi web interface to esxi datastore will not work? What about iSCSI?

@agners I know the aarch64 images are still in development state but for testing purposes it would be nice to have my existing HA setup. So is there a reliable process to migrate from x86 to aarch64 version of home assistant?

claplace commented 2 years ago

@Grandma-Betty I tried by uploading the vmdk file via esxi web interface, and it's only doing that: uploading the file, not importing it as a disk. I get the error "[Failed to power on virtual machine haos. Unsupported or invalid disk type 2 for 'sata0:0'. Ensure that the disk has been imported. Click here for more details.]". You still need that vmkfstool step for it to be usable.

iSCSI works, but the block device will be formatted as VMFS, so the vmkfstool step is also required.

agners commented 2 years ago

I know the aarch64 images are still in development state but for testing purposes it would be nice to have my existing HA setup. So is there a reliable process to migrate from x86 to aarch64 version of home assistant?

The backup only backups data, and a reference to the Core/Add-ons versions. Restoring a backup on a different architecture should work without further doing.

That said, it could be that some add-ons store "architecture specific" data, but that would be up to that add-on.

Grandma-Betty commented 2 years ago

@agners Nice, thanks. I just mentioned the new HA release from today which contains some of your changes about GRUB2. Did the aarch64 image from here leave its development state?

Grandma-Betty commented 2 years ago

Just wanted to report back that I migrated now successfully from HAOS (x86) to HAOS (aarch64) with the native backup feature of HAOS. Some minor manual steps were required to make all add-ons work again but nothing major. Only thing missing is open-vm-tools included in the aarch64 version. As far as I remember open-vm-tools is part of the native HAOS x86 virtual image but not of the aarch64 .vmdk file.

So far I had nothing that I was not able to migrate. In case anyone else is interested, I'm currently running the following add-ons/integrations which I have all migrated successfully from x86 to aarch64 version of HAOS:

Add-ons:

Integrations:

@agners @claplace Many thanks to both of you guys for your investigations, you really made my day! The issue can be closed now I guess.

claplace commented 2 years ago

Excellent! For open-vm-tools, you can

EDIT: uh, I don't even know if HA is based on debian packaging system...

agners commented 2 years ago

Did the aarch64 image from here leave its development state?

Yes, HAOS 8.0 is considered stable, and generic-aarch64 image is part of it. That said, since there aren't that many users, its certainly not the most well tested image :sweat_smile:

Only thing missing is open-vm-tools included in the aarch64 version. As far as I remember open-vm-tools is part of the native HAOS x86 virtual image but not of the aarch64 .vmdk file.

EDIT: uh, I don't even know if HA is based on debian packaging system...

HAOS is based on Buildroot. Buildroot has openvmtools available as a package, so it should be relatively easy to add.

I'll let you know once I'll create a 9.0 dev build with it included tomorrow. If that works, we can backport it to rel-8 branch and include it in OS 8.1.

Grandma-Betty commented 2 years ago

Yes, HAOS 8.0 is considered stable, and generic-aarch64 image is part of it. That said, since there aren't that many users, its certainly not the most well tested image 😅

No worries. Ganny will be happy to bite the bullet for y'all youngsters 💪

I'll let you know once I'll create a 9.0 dev build with it included tomorrow. If that works, we can backport it to rel-8 branch and include it in OS 8.1.

Granny is here to test and report back.

Grandma-Betty commented 2 years ago

@agners Does this image already include open-vm-tools and your latest NVMe driver commits?

https://os-builds.home-assistant.io/9.0.dev20220518/

EDIT: I tested it now. NVMe controller indeed works now in my case but open-vm-tools seems not to be recognized by ESXi yet:

Unbenannt

Also all features that open-vm-tools are providing are not working so it's not implemented correctly yet.

Grandma-Betty commented 2 years ago

Tested the latest dev image from here: https://os-builds.home-assistant.io/9.0.dev20220519/ Still the same as above. NVMe works, open-vm-tools not yet.

agners commented 2 years ago

Hm, open-vm-tools should be part of that build. Can you get OS shell access (using login) and check if the udev rules are present?

ls -la /usr/lib/udev/rules.d/99-vmware-scsi-udev.rules
Grandma-Betty commented 2 years ago

Nope, there's not even the /usr/lib/udev directory available: Unbenannt

It's completely missing.

agners commented 2 years ago

Hm, I see, the build system prevents the package from being selected under AArch64 :see_no_evil: should be a easy fix.

Grandma-Betty commented 2 years ago

Let me know when the next image is ready.

Grandma-Betty commented 2 years ago

I can confirm open-vm-tools and nvme boot works now with the latest 8.1 update. Thanks a lot @agners for your support mate! Bischen geile Siech ;-)

Grandma-Betty commented 2 years ago

Just a minor thing I found: Isn't the system supposed to show "true" for the "Virtual Environment" entry when using a .vmdk as a installation source? I mean, where else would you use a .vmdk file for than within a virtual environment? Or am I misunderstanding the meaning of this? Unbenannt

mondface commented 2 years ago

Now I also run home assistant 8.1 with "vmware tools" on my esxi on arm. Thank you Grandma-Betty!

so far I had my ha running on an intel NUC as vm under esxi 7. since today also runs a ha on a raspi 4 with 8GB RAM as vm under esxi on arm. first download the vmdk "haos_generic-aarch64-8.0.vmdk.zip":

https://github.com/home-assistant/operating-system/releases/tag/8.0

Create a vm as "other 4th linux-system 64-bit", but without disk then copy the unzipped vmdk to the esx and import it there (please adjust paths):

vmkfstools -i /vmfs/volumes/iso/haos_generic-aarch64-8.0.dev20220223.vmdk /vmfs/volumes/datastore1/haos/haos.vmdk

Include vmdk in ha's configuration first go to 8.1 after installation, now you also have "vmware-tools"

image

Grandma-Betty commented 2 years ago

@mondface You're welcome in any way, but don't thank me, thank @agners and @claplace, they're the real heroes here ;-)

claplace commented 2 years ago

oh I need to upgrade to 8.1 then!

Detecting VMware virtualization is possible by looking at the device tree. e.g. I added those lines to systemd: https://github.com/systemd/systemd/blob/df423851fcc05cf02281d11aab6aee7b476c1c3b/src/basic/virt.c#L134

mondface commented 2 years ago

I have a zwave-me stick, but it doesn't run smoothly. USB I have to assign 3.1 for the vm. how did you implement that?

image

image

Grandma-Betty commented 2 years ago

@mondface Well I don't use zwave, but it's kinda common that you have to use a USB 3.1 controller. I was never able for any distribution running on ESXi ARM to make USB 2.0 controller working reliably. But since USB 3.1 controller never gave me any issues it's not really a disadvantage to me. I think the USB backwards compatibility is always given so I don't care. You just have to know it, that's all ;-)

I remember once writing an issue about this according to Debian distro which may also be interesting to you: https://flings.vmware.com/esxi-arm-edition/comments/18497#comment_18505