Redesign CI

[richiejp]

There are a number of issues to break out here.

• GPU testing
• Speed up pulling containers
• New control plane?

Possibly they all have the same solution or not, it needs to be investigated.

[richiejp]

Using GitHub's GPU beta runners the cost per minute is $0.07 and our e2e tests used to take 40 mins, so that is $2.8 per job. Per hour it is $4.2. Also these machines are based on Tesla T2's which are quite old now, although that is perhaps a good thing. Other CI providers have similar or greater costs. The price of an A16 machine on Vultr is $0.5 per hour for comparison, although we may have to manage creating and destroying instances.

That is CPU only as well, once we bring all the dependencies for GPU into the mix then a full run could take longer.

[richiejp]

V100 on DataCrunch (on demand) is $0.88 per hour

[richiejp]

One option would be to take a QEMU VM snapshot of a running k3s cluster with the NVIDIA operator installed. Then load the VM snapshot at the start of each test and do the install etc.

Problems:

• Nested virt unless we are running on baremetal already
• Host to guest communication could require some networking magic (in the past I have used MACVTAP/MACVLAN)

Pros:

• Excluding virt and network requirements, not provider specific
• Could be done locally
• No boot or installation time for the OS or K3s

[richiejp]

• QEMU supports port forwarding (I never knew)
• K3sup for initial image creation or if we can't snapshot

[richiejp]

Seems there is only limited crossover between GPUs supporting virtualization/passthrough and GPUs supported by the operator. Pretty much limited to data center GPUs with passive cooling.

• https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-550-54-15/index.html#virtualization
• https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html#supported-nvidia-data-center-gpus-and-systems