AWS automated refresh of EC2 instance types

alex-dabija commented 2 years ago

Story

-As a cluster admin, I want to use any EC2 instance type available in order to have flexibly configure clusters and to use the best instance type for our use case.

Background

The list of EC2 instance types for an installation is static, outdated and updates are manual.

We are also slow in adding additional instance types because not all of them are available in all regions. The maximum number of pods calculation needs to be updated every time a new instance type is added. A new release is required because the calculation is done on every EC2 instance and the script is part of k8s-cloudconfig.

Requirements

We need to strive for a common solution between AWS & Azure.

Possible solutions

The biggest problem we have is the propagation of the EC2 instance data required by the maximum pod calculation to every instance. There are two possible solutions:

Custom AMI with EC2 instance types lookup table baked in;
- China AMI are generated by our pipelines;
- AMIs need to be pushed to all regions where we have clusters;
- CAPI will be easily supported when switching to Flatcar;
Propagate the instance type lookup table through out our system:
- Git -> Installation -> S3 bucket -> EC2 instance;
- the CI pipeline will generate the lookup table and create a configmap per installation with the information;
- a new operator will create a S3 bucket for each cluster and upload the lookup table data to the bucket;
- this operator will also upload the data to every cluster bucket every time the configmap is updated;
- when a new instance starts it download the lookup table data from the S3 bucket;
- the instance is aware of the bucket through k8s-cloudconfig;
- the kublet environment script uses the lookup table data to calculate the maximum number of pods;

Resources

othylmann commented 2 years ago

We just need 2 different machine times for obi.

alex-dabija commented 2 years ago

Unfortunately, our product has many limitations and at the moment in order to have proper support for those additional instance types we have to do a release because of some hard-coded calculations in aws-operator (e.g. number of IPs which can be used by the instance). That's why we want to put in place some automation and extract the calculation from aws-operator in order to stop having to do these kinds of updates.

Yes, I do know that this should be just a simple operation, but it's not.

alex-dabija commented 2 years ago

Blocked until we decide on how we want to make the EC2 instance data available to every EC2 instance we run as part of our Kubernetes clusters.

alex-dabija commented 2 years ago

@alex-dabija will draw a diagram on how this is going to work.

alex-dabija commented 2 years ago

This will not be needed if we switch to Cillium.

T-Kukawka commented 2 years ago

Will have to revisit the tool for the config repo after we have Cillium: https://github.com/giantswarm/instance-types-util

T-Kukawka commented 1 year ago

still paused due to Cilium

giantswarm / roadmap