Support AWS M6g instance types

[marians]

User Story

• As a cluster admin, I want to make use of AWS m6g instance types, in order to make use of them in general purpose worker nodes with a good cost to performance ratio.

Acceptance Criteria

• [ ] Given I create a node pool, I should be able to select an m6g.* instance type and the worker nodes should in fact make use of them.

Tasks

• [ ] Provide OS AMI for use with ARM

Background

AWS advertises the M6g instance types as 40% more cost-efficient than M5 types. They are using a custom ARM-based CPU named Graviton2. The feature is currently (March 2020) in preview.

The types specified so far:

Note: m6g.medium and m6g.large do not offer enough CPU and/or RAM resources to match our general requirements.

[calvix]

as noted in the internal discussion, we need to build/provide all binaries and all docker images for the arm64 arch as now we assume we run only amd64 arch

[othylmann]

@alex-dabija Is there any change here? Is there a way to do this in Q3? There might be a sustainability discussion at a customer and we would want do do this if possible. But sure, their teams also would need to see how fast they can move.

[alex-dabija]

Unfortunately, nothing has changed on this.

As @calvix mentioned, our product assumes that it's running on x86_64 and in order to support arm64 we would have to build/provide all the binaries for the arm64 architecture.

This would be a big project and is going to require a lot of effort. On the positive side we just need to do it for workload clusters and the management clusters can stay as they are.

Also, Flatcar added support for arm64 last December: https://www.flatcar.org/releases/#release-3033.2.0 .

In the context of CAPI, we also need to check what the state of arm64 is. I just added a quick note to the Phoenix board to have a quick look into it because we would probably want to do this for CAPI. Depending on this we would have to see if it's doable for Q3. It's difficult to provide a better answer until we have more information.

[T-Kukawka]

@AverageMarcus @njuettner @calvix can u share your recent 'attempt' to run arm64 images so we can have a status update?

[AverageMarcus]

• CAPI currently doesn't have any arm64 VM images available (ubuntu or flatcar). Upstream image-builder doesn't currently support building ARM images. I started working on a PR to add support upstream but haven't finished. Unless it's a priority over CAPA/CAPG I'm unlikely to pick it up again anytime soon.
• None of our docker images in Quay added by retagger have the arm64 arch included, even if the upstream image does. We only copy the amd64 version.
• Some upstream images (e.g. etcd) build a completely different image tag for arm64 rather than relying on the multi-arch support of docker image. We'd need to go through all we use and check if any others also do.
• ARM instances are available on AWS, GCP and Azure so we should ensure we handle all in the same way.

[calvix]

@T-Kukawka we did test for vintage only, I posted my notes to the google doc we had on the onsite, in short, the problematic part was that there are still many docker images on the vintage machines that need arm64 build before the kubelet and API could start, little tricky was kubelet and kubectl binary which we kinda hack on the fly with retagger and ship it with custom image

also it's important to not forget about the pause container which is configured in containerd config

other than it was was mostly about building all images for arm64 arch, since the flatcar already has arm64 AMIs we didn't need to do that

[T-Kukawka]

needs validation after CAPA - we will need first all components to be ARM compatible

[dalvarezquiroga]

Hi @T-Kukawka it will be finally compatible with CAPA clusters?