investigate deploying EKS cluster in MCP using unity-cs-infra

Creating EKS cluster on MCP using eksctl

manually running what is done via Github Actions: https://github.com/unity-sds/unity-cs-infra/blob/main/.github/workflows/deploy_eks.yml
Prerequisites
authenticate to MCP AWS account using tenantOperator role
- for this exercise, using the MAAP-HEC MCP AWS account as it is as close to a vanilla MCP AWS account I have access to and won't contain any Unity-specific AWS resources
get ARN of tenantOperator policy to use for permissions boundary
installation of eksctl and AWS CLI

Procedure
1. Prepare the provisioning machine by retrieving the short-term AWS keys for tenantOperator role on your MCP AWS account via KION API and set AWS credentials file. For reusability do these steps:
2. Create a file that contains your KION API key (1-year expiration date):
```
echo "export KION_API_KEY=app_555_1234567890" > kion_api_key.sh
```
3. Create a bash script, update_creds.sh that will query for the keys and update the AWS credentials file:
```
#!/bin/bash
AWS_ACCOUNT_ID=$1
source soamc_maap_hec/kion_api_key.sh
json=$(curl --no-progress-meter -XPOST "https://login.mcp.nasa.gov/api/v3/temporary-credentials" \
-H "accept: application/json" \
-H "Authorization: Bearer ${KION_API_KEY}" \
-H "Content-Type: application/json" \
-d "{
\"account_number\": \"${AWS_ACCOUNT_ID}\",
\"iam_role_name\": \"mcp-tenantOperator\"
}")
export AWS_ACCESS_KEY_ID=$(echo $json | jq --raw-output '.data.access_key')
export AWS_SECRET_ACCESS_KEY=$(echo $json | jq --raw-output '.data.secret_access_key')
export AWS_SESSION_TOKEN=$(echo $json | jq --raw-output '.data.session_token')
mv ~/.aws/credentials ~/.aws/credentials.lastgood
cat << EOF > ~/.aws/credentials
[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
aws_session_token = $AWS_SESSION_TOKEN
output = json
region = us-west-2
EOF
```
4. Run it passing in the AWS account ID:
```
./update_creds.sh 1234567890
```
5. Download and install the unity-cs-manager binary:
```
wget https://github.com/unity-sds/unity-cs-manager/releases/download/0.1.14-Alpha/unity-cs-manager-0.1.14-Alpha-darwin-amd64.tar.gz
tar xvf unity-cs-manager-0.1.14-Alpha-darwin-amd64.tar.gz
```
6. Run the unity-cs-manager to create an eksctl config YAML. Modify the params accordingly:
```
./unity-cs-manager eks --clustername maap-hec-eks-cluster-hysds --owner MAAP-HEC --managenodegroups dafaultgroup,1,3,1,m6i.xlarge --instancetype m6i.xlarge --projectname MAAP-HEC --servicename MAAP-HEC > eksctl-config.yaml
```
7. The resulting eksctl-config.yaml file will be barebones so extract the following information using the AWS dashboard:
  - AWS region: e.g. us-west-2
  - tenantOperator policy ARN: e.g. arn:aws:iam::1234567890:policy/mcp-tenantOperator
  - subnet IDs for both public and private subnets
  - MCP blessed EKS AMI
  - in AMIs, search for MCP Amazon Linux 2 EKS-Optimized and modify the file like so using AWS resource IDs in your MCP AWS account:
```
$ diff -u eksctl-config.yaml.orig eksctl-config.yaml
--- eksctl-config.yaml.orig     2022-10-19 10:20:35.000000000 -0700
+++ eksctl-config.yaml  2022-10-19 11:30:36.000000000 -0700
@@ -1,29 +1,30 @@
```
  - apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig
iam:
- serviceRoleARN:
- serviceRoleARN:
- serviceRolePermissionsBoundary: arn:aws:iam::1234567890:policy/mcp-tenantOperator withOIDC: false
metadata: name: maap-hec-eks-cluster-hysds
- region:
- region: us-west-2 version: "" tags: service: "MAAP-HEC" project: "MAAP-HEC"
- vpc: subnets: private:
- securityGroup:
- sharedNodeSecurityGroup:
- manageSharedNodeSecurityGroupRules: false
- us-west-2b: { id: subnet-1234567890 }
- us-west-2a: { id: subnet-123456789a }
- public:
- us-west-2b: { id: subnet-123456789b }
- us-west-2a: { id: subnet-123456789c }
- securityGroup:
- sharedNodeSecurityGroup:
- manageSharedNodeSecurityGroupRules: true
managedNodeGroups:
- name: dafaultgroupNodeGroup @@ -31,12 +32,13 @@ maxSize: 3 desiredCapacity: 1 instanceType: m6i.xlarge
  - ami:
  - ami: ami-1234567890 tags: service: "MAAP-HEC" project: "MAAP-HEC" iam:
  - instanceRoleARN:
  - instanceRoleARN:
  - instanceRolePermissionsBoundary: arn:aws:iam::1234567890:policy/mcp-tenantOperator privateNetworking: true overrideBootstrapCommand: |
    !/bin/bash
```
The final file should look something like this:
```
    apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig
iam: serviceRoleARN: serviceRolePermissionsBoundary: arn:aws:iam::1234567890:policy/mcp-tenantOperator withOIDC: false

metadata: name: maap-hec-eks-cluster-hysds region: us-west-2 version: "" tags: service: "MAAP-HEC" project: "MAAP-HEC"

vpc: subnets: private: us-west-2b: { id: subnet-1234567890 } us-west-2a: { id: subnet-123456789a } public: us-west-2b: { id: subnet-123456789b } us-west-2a: { id: subnet-123456789c } securityGroup: sharedNodeSecurityGroup: manageSharedNodeSecurityGroupRules: true

managedNodeGroups:
- name: dafaultgroupNodeGroup minSize: 1 maxSize: 3 desiredCapacity: 1 instanceType: m6i.xlarge ami: ami-1234567890 tags: service: "MAAP-HEC" project: "MAAP-HEC" iam: instanceRoleARN: instanceRolePermissionsBoundary: arn:aws:iam::1234567890:policy/mcp-tenantOperator privateNetworking: true overrideBootstrapCommand: |
  !/bin/bash
  
  /etc/eks/bootstrap.sh maap-hec-eks-cluster-hysds
```
1. Run eksctl to create the cluster:
```
  $ eksctl create cluster -f eksctl-config.yaml 2022-10-19 11:30:46 [ℹ] eksctl version 0.108.0 2022-10-19 11:30:46 [ℹ] using region us-west-2 2022-10-19 11:30:47 [✔] using existing VPC (vpc-) and subnets (private:map[us-west-2a:{subnet-0 379747e5833aa2c5 us-west-2a 10.52.106.0/25 0} us-west-2b:{subnet- us-west-2b 10.52.106.128/25 0} ] public:map[us-west-2a:{subnet- us-west-2a 10.52.104.0/24 0} us-west-2b:{subnet- 96 us-west-2b 10.52.105.0/24 0}]) 2022-10-19 11:30:47 [!] custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets 2022-10-19 11:30:47 [ℹ] nodegroup "dafaultgroupNodeGroup" will use "ami-*****" [AmazonLinux2/1.22] 2022-10-19 11:30:47 [ℹ] using Kubernetes version 1.22 2022-10-19 11:30:47 [ℹ] creating EKS cluster "maap-hec-eks-cluster-hysds" in "us-west-2" region with managed nod es 2022-10-19 11:30:47 [ℹ] 1 nodegroup (dafaultgroupNodeGroup) was included (based on the include/exclude rules) 2022-10-19 11:30:47 [ℹ] will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s) 2022-10-19 11:30:47 [ℹ] will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s) 2022-10-19 11:30:47 [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe- stacks --region=us-west-2 --cluster=maap-hec-eks-cluster-hysds' 2022-10-19 11:30:47 [ℹ] Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=fal se} for cluster "maap-hec-eks-cluster-hysds" in "us-west-2" 2022-10-19 11:30:47 [ℹ] CloudWatch logging will not be enabled for cluster "maap-hec-eks-cluster-hysds" in "us-w est-2" 2022-10-19 11:30:47 [ℹ] you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR -LOG-TYPES-HERE (e.g. all)} --region=us-west-2 --cluster=maap-hec-eks-cluster-hysds' 2022-10-19 11:30:47 [ℹ]
  2 sequential tasks: { create cluster control plane "maap-hec-eks-cluster-hysds", 2 sequential sub-tasks: { wait for control plane to become ready, create managed nodegroup "dafaultgroupNodeGroup", } } 2022-10-19 11:30:47 [ℹ] building cluster stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:30:48 [ℹ] deploying stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:31:18 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:31:48 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:32:48 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:33:48 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:34:49 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:35:49 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:36:49 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:37:50 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:38:50 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:39:50 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:40:50 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:41:51 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-cluster" 2022-10-19 11:43:53 [ℹ] building managed nodegroup stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:43:54 [ℹ] deploying stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:43:54 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:44:24 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:45:08 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:46:35 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:47:30 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:49:16 [ℹ] waiting for CloudFormation stack "eksctl-maap-hec-eks-cluster-hysds-nodegroup-dafaultgroupNodeGroup" 2022-10-19 11:49:16 [ℹ] waiting for the control plane availability... 2022-10-19 11:49:17 [✔] saved kubeconfig as "/Users/gmanipon/.kube/config" 2022-10-19 11:49:17 [ℹ] no tasks 2022-10-19 11:49:17 [✔] all EKS cluster resources for "maap-hec-eks-cluster-hysds" have been created 2022-10-19 11:49:18 [ℹ] nodegroup "dafaultgroupNodeGroup" has 1 node(s) 2022-10-19 11:49:18 [ℹ] node "ip-10-52-106-241.us-west-2.compute.internal" is ready 2022-10-19 11:49:18 [ℹ] waiting for at least 1 node(s) to become ready in "dafaultgroupNodeGroup" 2022-10-19 11:49:18 [ℹ] nodegroup "dafaultgroupNodeGroup" has 1 node(s) 2022-10-19 11:49:18 [ℹ] node "ip-10-52-106-241.us-west-2.compute.internal" is ready 2022-10-19 11:49:20 [ℹ] kubectl command should work with "/Users/gmanipon/.kube/config", try 'kubectl get nodes' 2022-10-19 11:49:20 [✔] EKS cluster "maap-hec-eks-cluster-hysds" in "us-west-2" region is ready
```
The deployment took a total of about 20 minutes.
**NOTE**: If you look at the CloudFormation screen on the AWS dashboard, there will be 2 stacks: 1 for the EKS cluster and the other for the node group.
1. Update kube config:
```
  aws eks update-kubeconfig —region us-west-2 —name maap-hec-eks-cluster-hysds
```
1. MCP CAVEAT (lesson-learned from MAAP-HEC): EKS worker nodes in private subnets need this configuration set to be able to access the Internet:
```
  kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true
```
1. Run a quick smoke test to check that you can run a pod and access the internet from it:
```
  kubectl run my-shell --rm -i --tty --image rockylinux:8 -- bash
```
Once you have the shell into the pod, run:
```
  dnf install git -y git clone https://github.com/unity-sds/unity-cs-infra.git exit
```
1. To destroy the cluster:
```
  eksctl delete cluster -f eksctl-config.yaml

unity-sds / unity-on-demand

investigate deploying EKS cluster in MCP using unity-cs-infra #6

Background

Task

DoD

Creating EKS cluster on MCP using eksctl

Prerequisites

Procedure

!/bin/bash

!/bin/bash