skorfmann
commented
3 years ago Well, first and foremost cdktf is a generator for HCL compatible JSON and relies on Terraform for applying the configuration. As you correctly stated, making it really dynamic would involve customizing the Terraform runtime.
Besides from this, a few bugs and other little issues, I think there are three major limitations which we have:
- jsii limits Typescript capabilities to a commonly supported feature set across all target languages
- Synth is a synchronous operation and makes it difficult to use most SDKs for data fetching (in TS / JS at least)
- Complex operations (loops, functions, etc) aren't abstracted
I believe all points can be addressed, but need thorough thinking before implementation.
cmclaughlin
Community Note
Description
I'm starting to see two main groups of users:
For the 1st group, things mostly work as expected. There are a few bugs, not all hcl features are available, and there aren't yet higher level abstractions. This makes sense since cdktf quite literally generates the terraform configuration language json variant.
The 2nd group experiences quite a few issues since other than provided information, nothing is actually available at runtime. I think this is what someone without a CDK background or deeper knowledge of cdktf would expect. I also think that it is a completely reasonable expectation. That said, I would be very surprised if the terraform team actually had plans to control the terraform runtime via external languages (think replacing the jsii runtime with terraform).
Assuming an actual terraform runtime is out of the question (seems like it), then I think we need to look for ways of emulating it:
One possible avenue is analyzing the syntax tree of the user code and converting to terraform json. This would basically be a very smart generator (still static configuration). A for loop could get translated into a count or a for each, etc. I have no doubt we could get a few simple cases working, but further support could be a very large and brittle project, not to mention that it would quite likely break down in more complex scenarios.
Having access to the multitude of functions built into hcl would go a long ways. These are commonly what are used in hcl to go beyond simple references and make it feel more dynamic. This is also sort of tangential to my main points, but since I brought it up, we should figure out mimicking intrinsic functions from AWS CDK is the way to go, or if having an order of magnitude more functions (+ arithmetic operations) means that a different approach is needed.
One of the most common areas that have common up is actually how to access a list (or set, or map, or complex object). We have
Tokenswhich partially make that work. They are a bit buggy, so that doesn't help anything, but they are also incomplete (and in some ways never can fully handle possible goals). There's also a special case method for accessing a particular index from a list. Generating full wrapper types that track access and convert to for each loops and count index loops in terraform might get us far enough here. Admittedly, something as simple as printing a value or accessing the length in an integer for loop would still not work. Side note, there's also a bug where computed types don't even generate all of their properties.Another possible route is building custom code to simply call various sdks during run time (synth time) to make information actually available. This really feels like building out functionality that terraform already has though. I can imagine building an intermediate layer that is made up of just data sources and ran through terraform. However, even data sources aren't always that simple. They can have dependencies on resources being built and such so that weren't starting to venture into build terraform outside of terraform in order to call terraform.
I'd like to look towards AWS CDK for some guidance; however, terraform is by nature more dynamic than cloud formation, so we may need to forge our own path on a lot of this. A lot of thought and effort needs to go into solving these problems, but I'm confident the end result can be something very powerful and extremely useful.
This isn't meant to be a spec or a rant; just a brain dump that can hopefully serve as a starting point for some design conversations. I truly want this project to succeed. We just need to ask ourselves these hard questions to push the design forward.
References