Migrating to genimage for image generation in hdd-image.sh

[jameshilliard]

I noticed that hdd-image.sh is using a rather complex sfdisk and sgdisk based disk image generation scheme, the standard image generator for buildroot based projects is typically genimage which should be significantly simpler and more maintainable than manually generating images as configuration is done via declarative config files.

Is there a reason the Home Assistant Operating System is not able to use genimage for image generation?

From what I can tell genimage should support everything necessary here.

I maintain a number of buildroot packages myself in addition to downstream projects based on it and am quite familiar with genimage so I can probably help with porting hdd-image.sh to use genimage if it is feasible.

[agners]

This is something which is on my todo list. At this point I am not aware of a show-stopper for using genimage. There is a bit a of MBR magic going on to support hybrid MBR/GPT mode for some platforms, so that certainly needs careful attention. I hope to get this done for the next major release 7.

[jameshilliard]

There is a bit a of MBR magic going on to support hybrid MBR/GPT mode for some platforms, so that certainly needs careful attention.

That should now be fairly trivial, I wrote the hybrid MBR/GPT implementation for genimage a little while back.

[jameshilliard]

FYI you might also want to look into potentially migrating from rauc to swupdate(similar to rauc but somewhat more advanced/flexible), my patch adding hybrid MBR/GPT diskpart support there has been merged.

[agners]

migrating from rauc to swupdate

Such a change would be a bit more involved, and would need a convincing reason to do it. Besides OS, it would need changes in our CI, update distribution setup and in Supervisor.

similar to rauc but somewhat more advanced/flexible

We are quite happy with rauc so far. Any specific features you are missing with rauc?

[pvizeli]

Based on our design, the OTA updater needs to have dbus support. Rauc was the only updater with dbus at this time and it work perfectly fine. It's not simple to move and without real reason, we should not invest time to do that.

For genimage. As we start the project, it was not able to handle our need for flexibility. Maybe it can work today, I only looked partial at the progress there. However, I think makes sense to move to genimage if it works today.

[jameshilliard]

Such a change would be a bit more involved, and would need a convincing reason to do it.

Yeah, not sure it makes sense, both are decent but I recall rauc was missing some flexibility/features that swupdate had.

Besides OS, it would need changes in our CI, update distribution setup and in Supervisor.

Not entirely sure but I skimmed some of the supervisor code and a good amount of what it's doing could probably be offloaded to swupdate, I noticed a lot of somewhat more manual update logic there that could probably be handled by swupdate as a backend for supervisor essentially.

Any specific features you are missing with rauc?

Things may have changed but last I evaluated rauc was missing some rather common features like integrated local web server based update capability.

Based on our design, the OTA updater needs to have dbus support. Rauc was the only updater with dbus at this time and it work perfectly fine. It's not simple to move and without real reason, we should not invest time to do that.

Oh, why does that require dbus? From what I could tell supervisor(I'm assuming that's what does the OTA updates right?) just needs an API/management interface for managing the updates which swupdate does have. Not entirely sure this is correct but from my reading of the code it seems supervisor isn't even dbus capable itself but rather has to call out and parse xml data from an external gdbus binary for interfacing with dbus anyways. I've been using adbus for natively interfacing with dbus from python in some of my buildroot projects. I should probably get around to upstreaming the buildroot package for adbus, it's currently still in one of my BR2_EXTERNAL's I think. There's a lot of architectural similarities with some of my projects and home assistant operating system in general that I've noticed, I even use aiohttp for the local web server as well.

For genimage. As we start the project, it was not able to handle our need for flexibility. Maybe it can work today, I only looked partial at the progress there. However, I think makes sense to move to genimage if it works today.

Yeah, if you run into any issues there let me know.

[pvizeli]

Not entirely sure this is correct but from my reading of the code it seems supervisor isn't even dbus capable itself but rather has to call out and parse xml data from an external gdbus binary for interfacing with dbus anyways.

Right, I don't want to explain that why we do that, just it works perfectly fast and good ;) Feel free to jump into the development and play with the code. However, dbus is a hard requirement for everything and the way how we speak with the host and manage the system.

[jameshilliard]

Right, I don't want to explain that why we do that, just it works perfectly fast and good ;) Feel free to jump into the development and play with the code.

Is there documentation on why supervisor doesn't use native dbus? My assumption is that it is just due to not finding a good async library for dbus interfacing at the time, adbus was fairly new when I started using it as I did end up upstreaming a few bug fixes early on to it, I think it's now fairly stable though. I'm mostly using adbus for controlling networkmanager at the moment and have wrapper async functions around a good bit of its functionality already, I could potentially extract+open source that as a library or something if that might be useful here.

I was thinking at some point I would probably write a native asyncio wrapper library for swupdate as well, I hadn't decided if it made more sense to wrap the unix socket IPC interface from the python side or just try and add a dbus API to swupdate and interface with that instead, is there an advantage to dbus there over say just an asyncio library around the unix socket API?

There's some non-async python client libraries for swupdate such as igupd that could probably be made asynchronous without too much difficulty that just use the unix socket API. I think swupdate has at least some systemd socket activation features but may also make sense to give it full dbus support.

Anyways just throwing some potential ideas out there as I'm doing a lot of similar things with my own projects that may make sense to collaborate on.

However, dbus is a hard requirement for everything and the way how we speak with the host and manage the system.

Yeah, it's def handy for that sort of thing in general, I'm mostly using it with network manager at the moment but have a few other things planned with it for managing/interfacing with services and such.