Arm64 resources for testing and building

Itxaka commented 2 months ago

Please fill out the details below to file a request for access to the CNCF Community Infrastructure Lab. Please note that access is targeted to people working on specific open source projects; this is not designed just to get your feet wet. The most important answer is the URL of the project you'll be working with.

If you're looking to learn Kubernetes and related technologies, please try out Killercoda or Play With Kubernetes instead.

First and Last Name

Itxaka Serrano

Email

Itxaka@kairos.io

Company/Organization

Kairos/SpectroCloud

Job Title

Open source developer

Project Title (i.e., a summary of what do you want to do, not what is the name of the open source project you're working with)

Build arm64 artifacts natively

Briefly describe the project (i.e., what are the details of what you're planning to do with these servers?)

We build a lot of arm64 artifacts and having them built on arm64 workers directly would speed up a lot our building processes. From full isos, to oci artifacts to single packages, having to cross build makes the process very slow and prone to failures. We also have a very small test coverage of arm artifacts due to the slowness of cross qemu testing, so it would also help a lot to expand our coverage to be sure that arm and amd arches are on sync.

Is the code that you're going to run 100% open source? If so, what is the URL or URLs where it is located? What is your association with that project?

Yes. kairos.io / github.com/kairos-io Im a maintainer of the project

What kind of machines and how many do you expect to use (see: https://deploy.equinix.com/product/bare-metal/servers/)?

c3.large.arm64 x1as there doesnt seem to be any other smaller arm64 servers?

What operating system and networking are you planning to use?

Ubuntu 24.04, dont care about networking that much.

Any other relevant details we should know about

idvoretskyi commented 1 month ago

@Itxaka thanks you for submitting. Can you please provide more details about the workflow of the artifacts building?

Specifically, I'm curious whether you can benefit from the ARM GitHub Actions Runners?

Itxaka commented 1 month ago

@Itxaka thanks you for submitting. Can you please provide more details about the workflow of the artifacts building?

Specifically, I'm curious whether you can benefit from the ARM GitHub Actions Runners?

Hey @idvoretskyi !

For example, on version bumping of dependant packages on https://github.com/kairos-io/packages/tree/main/packages they are built for arm64 arches. This usually takes a lot of time in some cases like our systemd-boot packages (built from source) due to the speed of cross-compilation (we are talking about 6/8 minutes built time in x86 VS around 90 or more in cross arm64)

Thankfully that has been solved via using the smaller github-arm64-2c-8gb :D

But on the pure Kairos side, we build and test x86 artifacts via VMs as we need to test a variety of scenarios (encrypted partitions, reset, UKI, secureboot, different install options, recovery) and distros (ubuntu, debian,suse,alpine, etc...) like show here: https://github.com/kairos-io/kairos/actions/runs/9998714411

On arm64, unfortunately we dont test anything. We just buildt a variety of arm images (https://github.com/kairos-io/kairos/actions/runs/10038355293 for example) including a generic arm image that runs under Qemu, but cant test those as doing cross testing of arm64 artifacts in x86 runners is way too slow.

Having an arm runner would help us on:

reducing building times of those Kairos arm artifacts (Usually each artifact takes about 1 hour of CI running time, VS around 10/15 minutes of an x86 artifact)
Introduce testing for arm artifacts. Currently we have to manually QA arm artifacts before we do a release, and sometimes they are broken in the most simplest of things, but we dont notice until we start testing those.

Both of those tasks are a bit heavy for the single github-arm64-2c-8gb worker, plus when we build we built several artifacts in parallel as our matrix is a bit big on push merges (4 distros * 2 devices(rpi3 and rpi4) + 1 generic arm64) Our release is even a bit bigger as we release artifacts with the latest 3 supported k8s versions (so 9 core images + 27 k8s images!)

Inf act, one of the reasons we dont build more than 1 generic arm iso is that we have no way of testing it anywhere, so it makes no sense, but the idea would be to build those on the same level as the x86 flavours (so several distros like leap,ubuntu,alpine) and test those with the same battery of tests we do so we can prevent arm64 only bugs escaping our almost-non-existant testing.

I hope this makes it clearer? Let me know if you need more info about this or some suggestion on how to continue.

Thanks!

idvoretskyi commented 1 month ago

@Itxaka understood, thanks.

Access granted!

Itxaka commented 1 month ago

@Itxaka understood, thanks.

Access granted!

Thanks!

Itxaka commented 1 month ago

@idvoretskyi a couple of questions.

Now that Im accepted I can login into the equinix page, Im supposed to order an on-demand server myself? I dont want to do anything until its confirmed :D Also, its possible to give other team members access to the same team? Im going on holidays soon :D

Cheers!

idvoretskyi commented 1 month ago

@Itxaka yes, just spin up the instance.

Feel free to DM me with the emails of the folks whom to add, I’ll take care of this :)

cncf / cluster