Multi-arch builds

[cedricve]

We are running a multi arch build in a github workflow using docker buildx build, and are wondering how to manage the balena deploy so all architecture images are merged in a single release.

https://github.com/kerberos-io/agent/blob/master/.github/workflows/docker.yml#L63-L67

[pipex]

Hi @cedricve, thanks for your question. I was also looking into this myself. Unfortunately the limitation here is that the balena CLI, which this action uses under the hood, does not yet support pushing multiple images for the same release to our registry, and it doesn't push the multi-arch manifest either. Multi-arch support is on our roadmap but we don't have an ETA for that feature yet or the implementation details for it.

That being said, I am looking into adding a useDeploy feature, that could use image artifacts from a previous step in order to do a balena deploy instead of a balena push

The action would look in its local image store for one or more <projectName>_<serviceName>:<architecture> or <projectName>_<serviceName>:latest images (loaded by a previous step of the same job), and then deploy the images matching the target fleet architecture.

What do you think about that as an interface?

[cedricve]

Thanks @pipex that would already make more sense. Where I think a lot of confusion starts is the difference between the "direct" deployment on Balena Hub and the forking of a fleet.

1) When browsing to the Balena Hub, opening the Kerberos Agent project, and pressing the Get Started button, it will list all the available supported architectures. However under the hood the fleet was pointed to the arm7 architecture, so only that image is available in the balena registry. If you select an ARM64 device for example Raspberry Pi 4 from the list, it will not boot. Which is not expected and can been seen by the users as malfunctioning of the project, or even worse Balena.

2. On the other hand when you "fork" the fleet to your own fleet. You can select all (even non-supported) architectures from the list. This helps to get the other "available" architectures which are supported, as after creation it will make a new release using the right base image. However, here you can select a non-supported architecture from the list, and then it will not work.

Bottom line is that it's really not straight forward to a users what is working and what is not.

[pipex]

Hi again @cedricve, thanks for clarifiying, I think I was jumping the gun on what you were asking so I apologize.

So as I mentioned before, the balena registry is single arch for now, this means, if you have a docker-compose.yml pointing to a multi-arch image, for instance

version: "2.1"

services:
    hello:
        image: hello-world

when you do a balena push myfleet via the command line (or using this action), the builder gets the architecture from the target fleet (myfleet) and pushes just the image for the correct architecture to the balena registry. So if the target fleet is rapspberrypi3, then the builder will download the arm/v7 image and push just that to the registry.

The arm/v7 image can be used by any compatible device type, including arm/v8 devices (e.g. the pi4), since arm/v8 is backwards compatible, which explains the range of possible device types you see on the dashboard.

A simple test can be done to confirm this, on a raspberry pi 4 device running balenaOS for arm64, I can do the following

balena pull --platform linux/arm/v7 alpine
balena run --rm -ti alpine sh
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# despite the above warning, the command below works
$ apk add --update nodejs 

I tested the same with the kerberos image

balena pull --platform linux/arm/v7 kerberos/agent
balena run --rm -ti kerberos/agent sh
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# The command below works
$ ./main run opensource 80
{"level":"info","msg":"Successfully Opened config.json from opensource","time":"2022-09-27T15:36:06+02:00"}
{"level":"info","msg":"Main: updated unique key for agent to: S0UH034CAAHDXYIWZU2DUFGYXZAHYX","time":"2022-09-27T15:36:06+02:00"}

Which makes me think there might be something else going on when provisioning a rpi4 to the kerberos fleet. I would suggest trying to replicate the issue with your own fleet, that is, creating a raspberrypi3 fleet, pushing a release from the kerberos agent and then trying to provision a raspberry pi 4. Having access to the device via the dashboard will give you more visibility on what is going on.

I would also encourage you to look in the forums for answers, as our community is very active and we do active support there as well.

[cedricve]

Thanks for explaining, I didn't know that v8 was backwards compatible. However that wouldn't solve the amd64 boards right, as the target board on your fleet is determining the specific architecture build.

[pipex]

If your fleet is raspberry pi3 (armv7), you should be able to add raspberry pi 4 (armv8) devices to it, the other way around is not possible: a raspberry pi3 (arvm7) will fail if deployed into an armv8 fleet.

[cedricve]

Understood! So we’ll need to wait for the multi architecture capability. I think it would be nice and intuitive you can just select the right board (and under the hood the correct base image is selected arm or amd).

On 27 Sep 2022, at 15:59, Felipe Lalanne @.***> wrote:

If your fleet is raspberry pi3 (armv7), you should be able to add raspberry pi 4 (armv8) devices to it, the other way around is not possible: a raspberry pi3 (arvm7) will fail if deployed into an armv8 fleet.

— Reply to this email directly, view it on GitHub https://github.com/balena-io/deploy-to-balena-action/issues/200#issuecomment-1259553503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALZUGZJDZBITTH2ETIST5DWAL4VNANCNFSM6AAAAAAQSMJXRE. You are receiving this because you were mentioned.

[klutchell]

I'm hoping to have more to share on multi-arch apps soon!

[cedricve]

exciting !

[avihaibd]

Hi @pipex

Any updates on deploying for multi-architecture? I am looking for implementing the following flow:

1. Build each component in the docker-compose as multiplatform docker image
2. Push to private docker registry which is not accessible to balena servers.
3. Use images in the docker compose, build locally for multi-arch and deploy to balena.

Currently balena deploy enforces images to match platform of the fleet default machine type, (regardless of supported types in balena.yml), while devices of other machine types can still join the fleet.

Also, regarding

The action would look in its local image store for one or more : or :latest images (loaded by a previous step of the same job), and then deploy the images matching the target fleet architecture.

What do you think about that as an interface?

Consider allowing non-latest images, for example: <architecture>/<projectName>_<serviceName>:<tag>.
Also note that docker also allows building for multiple platforms in a single command, in which case the docker manifest will just use the digests of all pushed platforms.

Thanks

[klutchell]

Hey @avihaibd, thanks for your interest in this feature! Here is the related roadmap item that you can follow for updates!

https://roadmap.balena.io/posts/28/mixed-architecture-fleets-and-apps

In short, there are a lot of backend changes required to our API and database to support mixed device fleets, and multi-arch applications, but it is a priority project that we are excited to progress!

[robsonvn]

Meanwhile, this is what I am doing

FROM arm64v8/nginx:1.25 as nginx-aarch64
FROM arm32v7/nginx:1.25 as nginx-armv7hf
FROM nginx:1.25 as nginx-amd64

FROM nginx-%%BALENA_ARCH%%