Only hosts that have arm64 cpu architecture can be added to this cluster.

[texkelly]

I've created an ARM64 SNO cluster on my M1 MacBookPro using the Assisted Installer based instructions and the -multi ISO (running under UTM Apple Virtualisation with Rosetta enabled) but the add hosts dialog reports "Only hosts that have arm64 cpu architecture can be added to this cluster." when I want to add an x86_64 architecture node to the cluster. I should be able to do that, right?

bkelly@Brendans-M1-MBP sno % oc adm release info -o jsonpath="{ .metadata.metadata}" {"release.openshift.io/architecture":"multi","url":"[https://access.redhat.com/errata/RHSA-2024:0290"}%](https://access.redhat.com/errata/RHSA-2024:0290%22%7D%)

The whole point of the exercise is to be able to run both arm64 and x86 containers in the cluster. I am fine to create a small emulated VM in UTM for the x86 node. I know it will be slow.

All this is only necessary because OCP won't schedule x86 containers on an arm64 node, even with Rosetta enabled there.

[texkelly]

I've also tried CRC/Openshift Local - same issue there.

[tnk4on]

Unfortunately, this is currently not possible. The reason is a limitation of OpenShift's mixed-architecture cluster. The OpenShift product documentation describes it.

From OpenShift Container Platform version 4.12.0 and later, a cluster with an x86_64 control plane can support mixed-architecture worker nodes of two different CPU architectures. Mixed-architecture clusters combine the strengths of each architecture and support a variety of workloads.

From version 4.12.0, you can add arm64 worker nodes to an existing OpenShift cluster with an x86_64 control plane. 

https://access.redhat.com/documentation/en-us/assisted_installer_for_openshift_container_platform/2024/html/installing_openshift_container_platform_with_the_assisted_installer/expanding-the-cluster#installing-mixed-architecture-clusters_expanding-the-cluster

[tnk4on]

I also got the worker.ign from the arm64 SNO cluster and started the x86_64 RHCOS node and forced it to be added. As a result I got the following logs.

Feb 08 08:37:32 rhcos.sno.home.lab bash[2622]: WARNING: image platform (linux/arm64) does not match the expected platform (linux/amd64)
Feb 08 08:37:32 rhcos.sno.home.lab systemd[1]: Started libcontainer container 0c859879f4d370fae46afe50f71b1f3f57a0d85271eb470ac4b0ec51895b53f1.
Feb 08 08:37:33 rhcos.sno.home.lab systemd[1]: libpod-0c859879f4d370fae46afe50f71b1f3f57a0d85271eb470ac4b0ec51895b53f1.scope: Deactivated successfully.
Feb 08 08:37:33 rhcos.sno.home.lab bash[2622]: exec /usr/bin/runtimecfg: exec format error

As you can see from this, x86_64 compute nodes cannot be added to the arm64 control plane.

[texkelly]

Ok, thanks. That is a very annoying limitation. How can we get that changed? It makes running OpenShift on ARM64 pretty much useless.

[tnk4on]

This is probably due to the product strategy of Red Hat Inc. x86_64 is still the mainstream in many situations. This can also be imagined from the extremely small number of officially supported Operators for arm64. You will need to clarify your use case and provide an RFE.

IMO, WASM (WasmEdge) will be supported in a near future version of OpenShift, and the situation will become more complex as WASM becomes more popular.