search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
48 stars 87 forks source link

Flexible pod VM instance sizes #321

Closed bpradipt closed 1 year ago

bpradipt commented 2 years ago

Today the pod VM instance size is fixed. Creating this issue to discuss on how how we can make it variable depending on the Pod cpu and memory requests and limits?

One possible approach is for the cloud admin to configure a list of pod vm instance types, including a default instance size. CAA or mutation webhook takes the closest match based on the cpu and memory resource request/limit.

When no cpu and memory requests/limits are present in the pod spec, the default instance type is used.

The advantage of this approach is that it avoids keeping track of different cloud instance types. However it comes with limited flexibility.

Any other approaches ?

bpradipt commented 2 years ago

cc @yoheiueda @stevenhorsman @huoqifeng @knrt10 @surajssd @mkulke

huoqifeng commented 2 years ago

Yes, a profiles list as a configure for CAA is needed, I think it's acceptable to give a fixed profiles list for each cloud provider as first step. Then, 2nd step maybe leverage Cloud Provider API to get the profiles.

mkulke commented 2 years ago

One possible approach is for the cloud admin to configure a list of pod vm instance types, including a default instance size. CAA or mutation webhook takes the closest match based on the cpu and memory resource request/limit.

I think for the first iteration this is a reasonable approach. Where would such a admin-curated list be stored? I suppose it has to be a tuple (instancetype, vcpu, mem) to match a workload to an instance. Adding a list to the cli invocation would require embedding this information in the argument somehow:

--instance-types Standard_DC2as_v5:2:8;Standard_DC4as_v5:4:16

alternatively vcpus and memory could be retrieved from an api call and cached, not sure whether that works for all providers, though.

surajssd commented 2 years ago

I like the tuple idea!

For now this is good, but such configs on cmd line, begs the question if we need a CRD for CAA?

Implementation wise, we can try to match the closest upper limit (or say ceiling) of a machine we have from the list. Like if a user requests 1cpu and 1GB memory and the only closest machine from the list we have is Standard_DC2as_v5:2:8 we can use that.

Above approaches are good for lift and shift use cases, where a user brings in existing Deployment objects and things work OOTB without any config change. Although I wonder how many of the workloads will really be lift and shift! Because users would have to change things like node selectors, runtimeclass and such to influence scheduling of pods on encrypted hardware machine types.

For advanced users, who wants to provide a machine type themselves, we should allow them to provide annotations for machine types on the lines of io.cc.caa.machinetype=Standard_DC4as_v5.

bpradipt commented 2 years ago

I like the tuple idea!

For now this is good, but such configs on cmd line, begs the question if we need a CRD for CAA?

Increasingly the need for a CRD is felt now. Consider the case of cloud resource pruning in case of any errors. CAA is stateless today and there are issues in the error path - node maintenance, CAA restart etc.

May be we can piggy back on @jensfr work - https://github.com/confidential-containers/cloud-api-adaptor/issues/328 ?

Above approaches are good for lift and shift use cases, where a user brings in existing Deployment objects and things work OOTB without any config change. Although I wonder how many of the workloads will really be lift and shift! Because users would have to change things like node selectors, runtimeclass and such to influence scheduling of pods on encrypted hardware machine types.

Changing the spec can be also done automatically using policy engines like opa or kyverno. So solutions exist which can be customised as needed by the cluster admin.

For advanced users, who wants to provide a machine type themselves, we should allow them to provide annotations for machine types on the lines of io.cc.caa.machinetype=Standard_DC4as_v5.

Makes sense.

surajssd commented 1 year ago

Oh sure, we can use CRDs for all my concerns. It would be a great upgrade to the CAA.

tumberino commented 1 year ago

@jensfr Is there a plan to add support of this into the controller logic?

@bpradipt Should we get T-shirt sizes added for each provider, i.e. S, M, L for each and then the controller can select from one of these based in the CRD? Or do we want the end user to have the option to just specify the instance profile directly in the CRD?

bpradipt commented 1 year ago

@jtumber-ibm I have some poc code that I'm currently working on. Should be able to share an early draft in next few days