[Notice] Breaking Change in CDKTF 0.13

xiehan commented 2 years ago

Back in August, we were excited to announce the first generally available (GA) release of Cloud Development Kit for Terraform, with version 0.12. With CDKTF, you can write Terraform configurations in your choice of TypeScript, Python, C#, Java, or Go, and still benefit from the full ecosystem of HashiCorp Terraform providers and modules. Among the changes in 0.12, we announced that Go is now fully supported, after having been classified as experimental in the past.

After that first GA release, we eagerly took in any and all feedback from the community, and one common theme predominated: performance. Go users in particular reported long build times and frequent out-of-memory errors, Java and C# synth times were significantly slower than our TypeScript benchmarks, and Python users reported a related issue that the generated classes for some providers were so big that they were crashing IDEs. While our metrics indicate that CDKTF is working well for the majority of users, the team agreed to prioritize addressing performance issues right away because we recognize that speed and stability are a core part of the user experience, and one of our north star goals is a first-class developer experience across all 5 languages that we support.

The team put our heads together and experimented with different ways to increase performance in a language-agnostic way, culminating in our 0.13 release. With this update, we're excited to announce significant improvements across the board, including (based on benchmark tests):

a 96.8% reduction in cdktf synth time when using Go with the Azure provider
an 83% reduction in cdktf synth time when using Java with the Google provider
a 36.8% reduction in cdktf synth time when using C# with the AWS provider
a 61.5% reduction in cdktf synth time when using TypeScript with the Kubernetes provider

… and more. In addition, the same changes that led to the above enhancements fixed the issue that was causing IDEs to crash for Python users, so we truly did see an improved developer experience across the board.

The downside is that in order to achieve these incredible performance gains, we must introduce a breaking change in CDKTF that will affect all existing users.

What Changed

With the 0.13 release, we are introducing namespaces in every class in our generated provider bindings, with namespaces automatically derived from the Terraform resource or data source it originates from.

Currently, each provider’s exports are a flat list of constructs (resources, data sources, etc.). While that’s really convenient at times, it also results in large packages that compilers and LSP find taxing to handle. Not only that, but generating providers also takes longer for CDKTF. With the move to namespacing, the providers are divided into logical groups based on the kinds of constructs being exported. These numerous small packages are faster for the compiler to process than one large package, resulting in the significant speedups highlighted above.

What You Need To Do

The introduction of namespaces means that the import paths for all of your provider-related classes will need to be updated. As an example, instead of:

import { AwsProvider, ec2 } from "@cdktf/provider-aws";

class MyStack extends TerraformStack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new AwsProvider(this, "AWS", {
      region: "us-west-1",
    });

    const instance = new ec2.Instance(this, "compute", {
      ami: "ami-01456a894f71116f2",
      instanceType: "t2.micro",
    });
  }
}

Starting from 0.13, you would do:

import { AwsProvider } from "@cdktf/provider-aws/lib/provider";
import { Instance } from "@cdktf/provider-aws/lib/instance";

class MyStack extends TerraformStack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new AwsProvider(this, "AWS", {
      region: "us-west-1",
    });

    const instance = new Instance(this, "compute", {
      ami: "ami-01456a894f71116f2",
      instanceType: "t2.micro",
    });
  }
}

For more upgrade information, check out our upgrade guide for version 0.13.

When Do You Need To Update

If you're currently using CDKTF 0.12 and it's working well for you, you don't have to update to 0.13 right away. The only change introduced in 0.13 was the performance enhancement, so you're not missing out on any critical features.

If you'd like to start to take advantage of the performance improvements in 0.13 but aren't ready to change the import paths for all of your existing providers, we have good news for you, too: we've made the 0.13 release backwards-compatible with 0.12 provider bindings. This allows you to have two versions of the same provider in your code (one locally-generated and one pre-built) and to mix namespaced and non-namespaced providers as you like. Our goal here is to reduce the disruption of this breaking change and give users more time to gradually migrate their code to the new syntax.

Once you're ready to fully embrace our newly namespaced world introduced in 0.13, then you would go ahead and update your code as described in the "What You Need To Do" section above.

No matter what, however, you will need to update your existing code to the new namespaced syntax in order to upgrade to 0.14. We are only maintaining the backwards-compatible provider bindings for this one release, meaning the 0.12 (non-namespaced) provider bindings will no longer work with CDKTF 0.14 and above. We are currently targeting November 15, 2022 for the 0.14 release date.

What's Next

The 0.14 release will focus primarily on quality-of-life improvements that make it easier to use pre-built providers. One key discovery we made as we researched all the possibilities for improving performance was that pre-built providers give a vastly better developer experience than locally generating them using cdktf get, and so we want to encourage users to take advantage of these as much as possible. In the coming months, we'll be looking to scale up the number of pre-built providers we make available as installable packages, and until then, we'll be making improvements to the cdktf provider add command introduced in CDKTF 0.11 and the cdktf init workflow to make it as easy as possible to find and add pre-built providers to your project.

Special Thanks

We would like to extend our deepest gratitude to the AWS CDK team, particularly Romain Marcadier (@RomainMuller), for working closely with us to explore all the options for improving the performance of code generation through jsii and helping us validate that introducing namespaces was our best path forward.

Try CDK for Terraform

If you’re new to the project, the tutorials for CDK for Terraform on HashiCorp Learn are the best way to get started. You can dive deeper into our documentation beginning with this overview of CDKTF.

Whether you’re experimenting or actively using CDK for Terraform, we’d love to hear from you. Please file any bugs you encounter, let us know about your feature requests, and share other questions, thoughts, and experiences in the CDK for Terraform discussion forum.

mattste commented 2 years ago

Just a heads up that the tutorial here has outdated code unless it is changed to specify the cdktf version.

skorfmann commented 1 year ago

While these performance improvements are pretty impressive, the changed import structure is quite a huge hit on the DX from my point of view (at least in Typescript). I'll have to import each and every resource explicitly. Are there plans to make this a bit more user friendly in future releases?

ansgarm commented 1 year ago

Hi @skorfmann 👋 You can still import * as aws and use e.g. aws.s3Bucket.S3Bucket in TypeScript – but I agree that it's still a bit more repetitive than it used to be (but at least only a single import required 😇😄)

I don't think we currently have an issue tracking an improvement that would allow to use aws.S3Bucket (in TypeScript, and some other languages where that's possible) but that's a good idea. This might need to be language specific though which makes it more complicated to introduce without breaking performance for languages like e.g. Go again – but I hope to be wrong 😁

yuriy-yarosh commented 1 year ago

I'm forced to use

node --max-old-space-size=1024 --optimize-for-size --gc-interval=500 ../../../node_modules/.bin/cdktf get -l go -o imports --parallelism 4

or

node --max-semi-space-size=512 --max-old-space-size=1536 --optimize-for-size --gc-interval=500 ../../../node_modules/.bin/cdktf get -l go -o imports --parallelism 4

to be able to actually download and codegen golang modules without OOM'ing, on 32Gb RAM Arch Linux box with Node 18 LTS.

--optimize-for-size --gc-interval=500 flags are not a part of NODE_OPTIONS env var, for some weird reason.

Is it possible to cache the existing codegen during get, so it could be incremental ? I'm importing a lot of modules and providers 40+ - it takes too much time to codegen everything.

xiehan commented 1 year ago

@yuriy-yarosh Is there a reason you can't use the prebuilt providers? That is still by far the best solution, especially for huge providers like AWS.

yuriy-yarosh commented 1 year ago

@xiehan I wan't to use it for air-gap's and AWS is still too big that it breaks golang dep management (#2829). I'm not looking for a workaround and 5gb of JSII bundles is simply unacceptable - it's easier to drop CDKTF entirely, for me. If there was a way to remove duplicates between multiple JSII bundles - thet's might've help, to some extent.

I couldn't do much hcl to golang conversion either... tfcdk was unable do process lookup try and other common TF functions, it's really weird when all the related conversion issues were closed. And it does look more like misorganization than regression, for me. Again, if someone had been using cdk converion to golang - they should've point out those issues already... which leads to a conclusion that no one does.

xiehan commented 1 year ago

@yuriy-yarosh Thank you for the feedback! While we believe we've optimized for the golden path that works for the majority of cdktf users, we're aware that performance is still suboptimal for some use cases, and air-gapped is indeed an ongoing challenge for us. What's tough is that when our team last did this performance optimization work with our 0.13 and 0.14 releases, we approached the limit of what we could improve in just cdktf itself; any more meaningful optimizations beyond this would have to happen at the JSII level. While we're not opposed to making PRs against JSII ourselves, we're a very small team so we've mostly been reliant on the AWS CDK team's stewardship of that project. We do sync with them regularly and I'll bring up these concerns with them the next time we meet to see if there's anything more they can help with here.

cdktf convert is still largely an experimental feature and Golang support for convert was only added with our 0.16 release which came out last month, so it's true that not many people have used it yet; we also haven't actively promoted it much because we know it won't work for large production projects. We are still actively working on improving convert; our 0.17 release will be focused on it as well, with a stronger focus on getting it to work better for non-TypeScript languages. If you encounter specific issues, feel free to open new tickets; that's helpful to our team as well, to get more feedback on what's working and what's not.

jsparrow7777 commented 1 year ago

@xiehan I am using version 0.11.2 of cdktf with typescript. Stack is not too big but still cdktf diff or cdktf synth crashes the machine with 8GB memory. It's getting complex to manage its deployment looking at its resource usage. If we upgrade it to version 0.13, is it gonna help ? Also, cdktf convert can convert HCL configuration to cdktf but is there any preferred method to do the opposite ? From what I read, we will have to do it manually using cdk.tf.json file. Open for suggestions !

hashicorp / terraform-cdk