[AWS EKS]: L2 EKS constructs which use L1s instead of custom resources

jbhalla-godaddy commented 1 year ago

Describe the feature

We created an EKS Cluster via CDK V2 (2.63.1) using the L2 Construct (code shown below) and noticed that the creation of the cluster created multiple resources including some custom resource lambdas.

Sample Code to create a test EKS Cluster using L2 Construct

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class TestingStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        const eksCluster = new cdk.aws_eks.Cluster(this, 'simple-eks-cluster', {
            version: cdk.aws_eks.KubernetesVersion.V1_21,
        });
        console.log(eksCluster)
    }
}

From a deeper investigation we understand that CDK's EKS L2 construct deploys some custom resource lambdas that use AWS SDK inside those lambdas to actually create the EKS cluster.

The objective of this ticket is to understand why did you choose to use AWS SDK to create the cluster (via custom resource lambda) and not used CloudFormation (AWS::EKS::Cluster) to create the EKS Cluster?

We are seeking to understand the design decision behind this implementation.

Use Case

We are using CloudFormation to deploy all our AWS Resources with CloudFormation Hooks to govern certain configurations for our customers.

Now, we want to restrict our customers to only create EKS clusters via CDK using CloudFormation, but because the current version of EKS L2 Construct doesn't use CloudFormation to create EKS Cluster, the CloudFormation Hooks doesn't get triggered when we use EKS L2 Construct which makes it harder for us to govern our customers.

Proposed Solution

Is it possible for AWS CDK to update the EKS L2 construct to create the EKS Cluster using CloudFormation (AWS::EKS::Cluster) instead of AWS SDK

Other Information

No response

Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

CDK version used

2.62.0

Environment details (OS name and version, etc.)

MacOS

jbhalla-godaddy commented 1 year ago

CC'ing @shapirov103 @rameshv29 @dsilbergleithcu-godaddy on this one

peterwoodworth commented 1 year ago

Our L2 EKS constructs use custom resources because we are able to achieve more functionality through custom resources than just through using what CloudFormation offers. I don't remember off the top of my head everything that CloudFormation doesn't support though

I think it would be unlikely we implement L2 EKS constructs that make use of the L1s, but we can leave this open to gather more feedback on this. If we were to start working on this, an RFC would have to be approved first, so I recommend creating an issue in the RFCs repo if you're interested in this.

I would overall recommend to drop to L1s if you need to use the resources CloudFormation offers instead of custom resources

pahud commented 1 year ago

@jbhalla-godaddy

As far as I know, if we deploy EKS with cloudformation CfnCluster L1 resource, the admin role of the cluster will be the CFN deploying role. With the admin role, we need a few follow up steps to complete the whole deployment including to update the aws-auth ConfigMap with kubectl CLI to add the nodegroup role into it, which is performed by a Lambda function as the cluster admin role.

With that being said, the lambda function running kubectl command would have to be the CFN deploying role which creates the cluster and this could be a concern if we need to separate the CFN deploying role and the EKS cluster role.

With current design, we create a new IAM admin role for cluster creation and allow this role to run kubectl against the cluster from lambda function so the cluster admin role will not be bound to the CFN deploying role.

I probably missed some other context but back to the early days when EKS was first launched there were a lot of restrictions and current design was one of the best alternatives for CDK to create a fully working EKS cluster. I agree there are some pros and cons with current design and I believe there's no reason for the team not to re-visit a new implementation for EKS cluster provisioning with the native CFN resources. Any comments or thoughts is highly welcome and appreciated here.

gricey432 commented 10 months ago

I believe this is now possible thanks to the new eks feature "Access Entries". According to the docs [1] this is now the recommended option for granting access.

[1] https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html

I've gone ahead and made a proof of concept for this which I've got deployed and successfully using kubectl to get a value out of a new cluster. https://github.com/gricey432/cdk-eks-poc

Extra Links:

gricey432 commented 10 months ago

Looks like there's an RFC for this now https://github.com/aws/aws-cdk-rfcs/issues/605

pahud commented 7 months ago

@gricey432

Yes we have the plan for the rewrite per the RFC https://github.com/aws/aws-cdk-rfcs/issues/605

adamjkeller commented 6 months ago

Hey y’all,

As part of our goal to improve service coverage with CDK constructs, we are actively working on this feature! While we can’t commit to specific dates, we’re making progress and will keep you updated along the way. Your feedback and support are helping us shape the roadmap, so thank you!

xazhao commented 1 month ago

Hi all,

We have started the RFC process for the EKS L2 Re-write. Will post updates here and share the RFC in https://github.com/aws/aws-cdk-rfcs/issues/605 once it's ready.

xazhao commented 1 week ago

This is the RFC for EKS Re-write. Feel free to take a read and leave comments there.

aws / aws-cdk