sig-node e2e tests machine hardware requirements

[ffromani]

sig-node owns a set of features related to exposing and using hardware details which require some hardware features to exercise the code. Examples are exclusive CPU allocation (cpumanager), device allocation (device manager), NUMA alignment (topology manager), NUMA alignment considering distances between NUMA zones (topology manager).

Note: some requirement overlap. Easy example: a powerful high end (at time of writing) server CPU can have at the same time multi core count, exposing multiple NUMA nodes, and have split L3, satisfying in one go all cpumanager requirements

Hardware requirements, driven by feature, rationale

• cpumanager (GA): x86_64, arm: machine with at least 4 cores exposed, better like 16 or more. We need more cores to be able to run different set of tests in different scenarios. 4 cores is the minimum to run basic test about the feature. We already got machines with 4 cores
• topology manager (GA): x86_64, arm: machine with 2 or more NUMA nodes. We need to align on NUMA nodes, so we need 2 or more nodes to begin with
• topology manager (KEP https://github.com/kubernetes/enhancements/issues/3545): x86_64, arm: machine with 4 or more NUMA nodes. We need to consider NUMA distances in allocation, so we need 4 or more nodes with different distances between each other
• topology manager (KEP https://github.com/kubernetes/enhancements/issues/4622): arm only?: machine with 9 or more NUMA nodes (!) like grace gpus
• cpumanager (KEP https://github.com/kubernetes/enhancements/pull/4810#): x86_64, arm: a machine with split (non-uniform) L3 cache, like epyc cpus
• cpumanager (KEP https://github.com/kubernetes/enhancements/issues/3327): x86_64, arm: a machine with multiple CPU sockets, in order to exercise alignment by socket.
• devicemanager (GA): x86_64, arm: hardware devices controlled by device plugins. Most common usecases are SRIOV cards and GPUs. One device is the bare minimum, we would like 2 or more to be able to use them also in the topology manager tests as deciding factor (see topology manager in this list)
• memorymanager (Beta, GA graduating): x86_64, arm: totally overlaps with topology manager requirements, listed here for the sake of completeness.

this list will be updated after more review of the ongoing sig-node features

[ffromani]

tagging some relevant sig-node people: @kannon92 @PiotrProkop @klueska

[ffromani]

slack thread for context: https://kubernetes.slack.com/archives/CCK68P2Q2/p1727202732284529

[ameukam]

cc @dims @upodroid @BenTheElder

[BenTheElder]

We have EC2 and GCE pretty well setup in particular at the moment, do any of the machine types available there meet your requirements?

Please make sure any new resources you use on any platform are handled by the kubernetes-sigs/boskos cleanup scripts. If you're using GCP projects / AWS accounts with VMs that should already work.

[ffromani]

@catblade kindly pointed out equinix donated cloud credits and their offering seems also interesting and maybe we can use it. Some CNCF TAGs already make use if it. This is the reference I got: https://github.com/cncf-tags/green-reviews-tooling/

[ameukam]

@ffromani why can't we use AWS EC2 instances to run those tests ?

[ameukam]

CPU architectures available:

GCP: https://cloud.google.com/compute/docs/cpu-platforms AWS:

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-hardware-specs

[ffromani]

@ffromani why can't we use AWS EC2 instances to run those tests ?

I think we totally can, I'm not aware of any blocker. The efforts in this area have been somehow sparse, we're taking the chance of sig-node 1.32 planning to re-evaluate and improve the current state. Will review the GCP/AWS offerings and comment.

[BenTheElder]

@catblade kindly pointed out equinix donated cloud credits and their offering seems also interesting and maybe we can use it. Some CNCF TAGs already make use if it. This is the reference I got: https://github.com/cncf-tags/green-reviews-tooling/

Yes, however we generally are running critical infra on Kubernetes specific resource allocations, and we don't currently have a lot setup to manage this. For equinix SIG K8s Infra doesn't currently have observability into the amount of resources available and the spending trends which has bitten us in the past (see reports like https://kubernetes.slack.com/archives/CCK68P2Q2/p1727127173398879 for some of the others).

(@dims does have cs.k8s.io running on equinix currently, we also have some presence in DO and Azure but not as mature yet, and Fastly for CDN)

It would be easier if we can use one of vendors for which we already have tooling (like https://github.com/kubernetes-sigs/boskos) setup to avoid resource leaks etc.

Otherwise we need help to invest in and onboard new resource types, observability into utilization and remaining credits, etc