Higher level Constructs

Thomas-X commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Currently there's only 1:1 mappings for most TF providers, and this is an awesome thing to have. But the big selling point for CDK(in my eyes) is that there's higher level constructs which reduce all the bits and pieces you have to think about when using terraform or cloudformation, let's call the 1:1 mappings L1 constructs and anything above that L2.

This would involve having a higher level construct for highly used resources. Think of adding a L2 construct for AWS Lambda, a loadbalanced AWS ECS cluster etc.

Basically supercharged terraform modules 😃

Cool, how would this look like?

Consider the following Lambda function setup:

// rest of the stuff is 
const lambdaRolePolicy = { // this could be typed!
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
};
const role = new aws.iam.IamRole(this, "lambda-exec", {
  name: `learn-cdktf-${name}-${pet.id}`,
  assumeRolePolicy: JSON.stringify(lambdaRolePolicy)
});

const lambdaFunc = new aws.lambdafunction.LambdaFunction(this, "learn-cdktf-lambda", {
  functionName: `learn-cdktf-${name}-${pet.id}`,
  s3Bucket: bucket.bucket,
  s3Key: lambdaArchive.key,
  handler: config.handler,
  runtime: config.runtime, // this could be a nice enum
  role: role.arn
});

This requires a bunch of stuff I don't want to worry about when developing a simple lambda function (CDKTF could lighten this!)

const lambdaFunc = new LambdaFunction(this, "l2-lambda-func", {
  functionName: 'foobar',
  // role can be entirely omitted, just generate one behind the scenes that's assumed by lambda principal
  runtime: aws.Lambda.Runtimes.DOTNETCORE3.1,
  code: new AssetCode("./code.zip"),
  // for the rest provide sensible defaults, use nullable features where possible to indicate a required property etc
});

I'm more than happy to spend some time on this if this is something we actually want

References

There's many, but the start would be removing all these hardcoded strings which could be in enums across most resources and eliminating the size of code needed for highly used resources by adding sensible defaults and 1st class 'modules'. https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.BucketAccessControl.html https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.RuntimeFamily.html https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.RedirectProtocol.html

props like autoDeleteObjects on S3 bucketswould be a big boon https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html#construct-props. There's countless ways we could make the library nicer to work with, which results in cleaner code for everyone.

ansgarm commented 2 years ago

Hi @Thomas-X 👋 I really like your ideas! They sound a bit like #474. Would you say this is a duplicate of that issue?

As you're mostly proposing L2 constructs for AWS: Have you seen the AWS adapter?

Thomas-X commented 2 years ago

Hi @ansgarm, thanks for the quick reply 👋 . Looking at it this is a duplicate of that issue, you can close this if that's preferred.

I checked out the AWS adapter (pretty neat!) but I think we should strive for 1st class citizen L2/L3 abstractions that use cdktf. This would enable us to make L2/L3 abstractions for other cloud providers as well.

I'd like to have a crack at a basic implementation, anything I should know that's not in CONTRIBUTING.MD? I'll just setup a quick repo for one of the AWS services to begin with, basing it off the L2 implementation of the AWS CDK.

I can update the other ticket or this one when I've done that, let me know

ansgarm commented 2 years ago

Ah, I see! Sounds like you have some slightly different thing in mind. Please let me know if I understood you correctly, but instead of only having L2/L3 abstractions for each single cloud provider, you'd rather build a shared L3 first (e.g. an L3 construct for a "hosted static website") and then have different implementations based on the cloud provider (i.e. implementations for that abstraction). Is that right? If yes, we could rephrase this issue's title instead of closing it 👍

We also had some discussions about this in the team in the past. I personally am quite torn about it, as I fear that different cloud providers have too different ways of approaching granting access to resources. So an abstraction like bucket.grantRead() might work with AWS IAM but could become leaky for other cloud providers that have e.g. some group based access model (I think Google does, but I'm not sure). But there might of course still be solutions to this problem and it could be possible to find a model that is not too leaky while supporting multiple patterns of "granting access". I also remember that @DanielMSchmidt also had some ideas around this and is less skeptic about such shared layers than I am 😁

The CONTRIBUTING.md mostly covers PRs to this repo, so feel free to just go ahead with a quick repo 👍 Such a first draft would be much appreciated as it would serve as a good base for further discussions. However, of course we can't promise you that we'll adopt it.

Have you seen the cdktf-plus package? @skorfmann started this some time ago to help with common boilerplate when deploying e.g. AWS lambda functions using CDKTF. It might serve as a good starting point about how to build a construct library.

DanielMSchmidt commented 2 years ago

Yeah, I'd love to build this one day, if you happen to start with it let me know and I'll take a look 👍

github-actions[bot] commented 1 year ago

Hi there! 👋 We haven't heard from you in 30 days and would like to know if the problem has been resolved or if you still need help. If we don't hear from you before then, I'll auto-close this issue in 30 days.

strongishllama commented 1 year ago

At the risk of putting words in @Thomas-X's mouth, I think he is talking about the same thing in issue #474. However, if one of these two issues is going to be closed, I would vote for it to be #474 as the description in this issue is a lot more comprehensive.

I'd also love for something like this to exist, and might have some free time to dedicate to it. @ansgarm & @DanielMSchmidt it's been a while since there was any activity on these two open issues. Any updates on your end you'd like to share? Or a place you've made a start?

Personally, L2 & L3 constructs for GCP would be the most useful for me, so that's likely where I'd put my effort. Likely starting exclusively L2 and leaving L3 once the L2 API feels good and stable.

Would love to hear your thoughts, thanks!

RichiCoder1 commented 1 year ago

I was just thinking about this myself!

The description of this issue alludes to it, but I think an important place to start (and good limited scope) would be asking what are the provider primitives that are missing for L2s that don't just provide higher level opinions.

So, for the AWS NodeJS Lambda example, the missing glue pieces for that are:

Asset management, aka automatically packaging and uploading assets to S3 (ideally in a more forward maintainable way that the CDK today). Ala S3 Assets/ECR Assets in the AWS CDK.
IAM Abstraction ala CDK AWS aws_iam to allow for sharing and creating intelligent iam policies

In addition, other abstractions are for example IConnectable and all the VPC/SG related abstractions.

Building those would enable building L2s (as well as potentially unblocking the AWS adapter for asset cases).

Ditto for GCP and Azure, as I imagine they have the same IAM and Asset management concerns.

ansgarm commented 1 year ago

Hi @strongishllama @RichiCoder1

thank you for bringing up this topic! While there are no concrete plans or prototypes, we're very interested in exploring this!

If I were to start writing some L2 constructs for GCP, I'd probably start with a small NPM package that has a peer dependency on the pre-built provider for GCP and then create a few constructs to get a feel for it. At a later stage this could use the same setup as e.g. the cdktf-tf-module-stack repository (which uses JSII to support all CDK languages). If you have any questions, I'm happy to help and if you start something, let us know!

In regards to shared abstractions (like e.g. IConnectable), I personally think that this might be pretty hard to get right as it may be easy to accidentally create leaky abstractions if the underlying permission models are subtly different. For example some providers – I think it was Azure, but I may be wrong – commonly use groups of resources to define access, while others like AWS use more granular controls. So an abstraction like serverA.allowAccess(serverB); serverB.allowAccess(serverC) might work for AWS but e.g. could put all servers into the same resource group for Azure, which might accidentally also allow access for serverA to serverC if implemented naively by putting all in the same group. But there might still be a way to find a good abstraction that doesn't require the user to know about such pitfalls. That said, this might get even more interesting when access is given between resources from two different providers which might ultimately not be possible without additional supporting infrastructure and would require us to implement a matrix of connections to actually support such cross provider access controls 😄

So in the end, a layer 2 construct library for GCS might have its own abstractions for managing access that is not based on the same primitives that e.g. an AWS L2 construct library might be based on.

However, there will probably be a common ground that is compatible across cloud providers, with e.g. asset management (as mentioned) being one thing that we're already looking at because we want to support that in the AWS Adapter. A Docker container that is built and uploaded to some registry for example would be very similar for lots of providers. Besides assets, there are probably other things that could rely on shared abstractions, and I'd say in general we'd be open to add such general abstractions to the cdktf core package (like e.g. an DockerImageAsset).

RichiCoder1 commented 1 year ago

Yup. To clarify, my intent was to not share anything between providers because of that exact issue. It's more work and possibly more redundancy, but I believe it would only do harm in the end to try and do any sort of multi-provider abstractions from the get-go for L2/L3s.

I'd endorse building the basics specific to the major cloud platform providers, and only then possibly try and build multi-cloud abstractions after that. Even Asset I fear will have platform-specific quirks that won't necessarily translate well when you get into actually giving those assets to cloud services (uploaded from S3, must be in ECR for AWS examples).

strongishllama commented 1 year ago

Great, sounds like we're all on the same page to me. I'm going to start tinkering around here. Feel free to reach out with any thoughts, feeling or feedback, it's all very welcome. It might be a little while before I have something I'm happy with/worth showing, so I'll post back here when there's a bit more there 😁

xiehan commented 1 year ago

I'll add that we here at HashiCorp have nothing against L2/L3 constructs and would love it if there were more native constructs for CDKTF as well, we just don't have the staffing to make that a reality ourselves; there's so much to do on the constructs side that we'd likely need a separate team to just focus on that, and that's not where we are today. This is why the only work we're committing to in the near-term is more work on the AWS adapter; I recognize that won't meet everyone's needs, but giving CDKTF users access to all AWS constructs is a super powerful thing in a world where we don't have enough constructs of our own yet. In the meantime, we're fully supportive of community contributions and recently added a construct authoring guide to our documentation to encourage these. If there are any other meta-level things like that that we can do to make that easier, please let us know! 🙂

RichiCoder1 commented 1 year ago

This is why the only work we're committing to in the near-term is more work on the AWS adapter;

Ironically one of the things missing from that called out here is the lack of asset support 😅. I def could see that bridge being great for AWS consumers otherwise though!

If there are any other meta-level things like that that we can do to make that easier, please let us know! 🙂

One thing that isn't absolutely necessary but might make @strongishllama and I's life easier is maybe expanding the scope of the TerraformAsset and maybe adding a TerraformImage resource. Being able to have the framework handle "building" assets, and then us taking over staging those assets in the appropriate cloud provider would probably help.

hashicorp / terraform-cdk