API to interact with terraform cli

[JayDoubleu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

It would be very useful to have an API to interact with terraform cli (not cloud) from cdk.

Example below:

app = App()
MyStack(app, "automate-everything")
synth = app.synth()
plan = app.plan(synth)

if plan.changes is not None:
    if len(plan.changes.module.custom.fruits) > 4:
       app.apply(synth, target='module.custom.fruits')

References

https://github.com/hashicorp/terraform-cdk/issues/225

[skorfmann]

The general idea of this is quite appealing to me.

However, At the moment, the cdktf cli is almost independent from a synthesized Stack. Means, it's really just interfacing with the Terraform CLI and interacting with the generated plan and the stdout / stderr output from it.

In the cdktf cli, we do have the plan at hand , synth isn't available as it's just doing a terraform apply in the cdktf.out directory. Means, as long as it's just about analyzing a plan and dynamically changing cli arguments, this could theoretically work within the current cli concept. However, I don't think this would be part of the app, more like some sort of hook in the CLI. Since the cli is just available in Typescript, it'd likely be Typescript code.

I think a similar concept could work for https://github.com/hashicorp/terraform-cdk/issues/219

[1mamute]

Maybe we can use terraform-exec to interact with synthesized outputs from the stack?

[skorfmann]

Maybe we can use terraform-exec to interact with synthesized outputs from the stack?

Since terraform-exec is a go module, it's not the best fit for this project since it's mostly written in Typescript. We sort of built something similar in Typescript.

[1mamute]

Maybe we can use terraform-exec to interact with synthesized outputs from the stack?

Since terraform-exec is a go module, it's not the best fit for this project since it's mostly written in Typescript. We sort of built something similar in Typescript.

Mind sharing the link or is it not publicly avaiable?

[skorfmann]

Was playing around with this a bit. While it requires the usage of JS / Typescript, it's sort of possible to drive synth and deploy / destroy bypassing the full cdktf-cli workflow. It's a bit clunky, but works. Please be aware, that this is using internal APIs of the cdktf-cli which might break without prior warning.

deploy

import { SynthStack } from 'cdktf-cli/bin/cmds/helper/synth-stack';
import { TerraformCli } from 'cdktf-cli/bin/cmds/ui/models/terraform-cli';

const setup = async () => {
  const stacks = await SynthStack.synth('node stack.js', 'cdktf.test.out');
  const cli = new TerraformCli(stacks[0]);
  await cli.init();
  await cli.deploy('', (chunk: Buffer) => (console.log(chunk.toString('utf-8'))));
}

(async function() { await setup() } ());

destroy

import { SynthStack } from 'cdktf-cli/bin/cmds/helper/synth-stack';
import { TerraformCli } from 'cdktf-cli/bin/cmds/ui/models/terraform-cli';

const teardown = async () => {
  const stacks = await SynthStack.synth('node stack.js', 'cdktf.test.out');
  const cli = new TerraformCli(stacks[0]);
  await cli.init();
  await cli.destroy((chunk: Buffer) => (console.log(chunk.toString('utf-8'))));
}

(async function() { await teardown() } ());

I'm using this right now for driving automated test setups / teardowns. Since this skips generating the plan and is running headless, it's also a bit faster compared to using the cdktf-cli directly.

[drandell]

@skorfmann I think this would be a great feature, it would mirror some of the functionality available in pulumi with its API. Is there a potential python approach to your sample?

[skorfmann]

Is there a potential python approach to your sample?

Since the CLI is written in Typescript only, I don't think there's a similar way to do this right now - except for shelling out und using the cli directly. However, cross language support for programmatic interaction is something we'd like to address - but I can't give you an ETA on that.

[drandell]

Is there a potential python approach to your sample?

Since the CLI is written in Typescript only, I don't think there's a similar way to do this right now - except for shelling out und using the cli directly. However, cross language support for programmatic interaction is something we'd like to address - but I can't give you an ETA on that.

Took a bit of time, but I managed to play around with JSII and add the necessary files into a local cdktf package, its not clear and I've probably brutalised it somewhat but I've got the following working in python;

def main():
    app = App()
    tfStack = MyStack(app, "python-cdk", GCP_PROJECT_NAME)
    app.synth()
    stack = SynthesizedStack(**tfStack.get_stack_args())
    cli = TerraformCli(stack=stack)
    print(cli.version())
    # print(cli.init())
    # plan = cli.plan()
    # print(plan)
    # print(cli.deploy())
    # print(cli.destroy())

And running that produces;

[kc-dot-io]

Thanks for these examples @skorfmann! This is my new favorite way to use CDKTF. I'd love to see a more formal API...

A few things from my experience that would help this... 1) Better error handling with remote runners using try/catch or promise 2) Better API for handing / templating response to stdin/stdout 3) Ideally JSON events with lifecycle hooks would be great 4) API support for multi-stack delpoyments / cross stack references

Looking forward to seeing this evolve. This is the most excited I've been about IaC in a while.

[danielloader]

Just to add to this issue, though it's not anything other than more justification: I'm looking to design a SaaS deployment project to deploy a simple Lambda Serverless stack with some auxiliary services, and for simplicity sakes was hoping I could utilise CDKTF in a Lambda function to deploy a new customer, patch/update it or delete a customer (fed off a DynamoDB table with streams for some audit log of infrastructure changes).

Pulumi works with the automation API but hesitant to commit to using it when both AWS CDK, and Terraform CDK are maturing and both offer advantages (already a Terraform Cloud customer at work and cloudformation state in AWS appeals in some scenarios). AWS CDK still doesn't have a programmatic interface, and the demo example from @drandell looks promising - though as cited, you'd want a formalised API before taking this into production.

Ideally the interface would allow you to run a stack synth, stack deploy and then take the outputs from the resolved deployed stack to do something with (insertion into dynamo/webhooks for notification of deployment and triggering an email etc).

[JayDoubleu]

@danielloader I've since fully moved on to pulumi automation API and I couldn't be happier. Not only I can do what you described (and more) but no longer have to deal with burdens of terraform itself ( fighting with provider bugs and new *features that completely change development workflow )

I took me a while to convince myself to it but now I wouldn't go back even if those features would be implemented and the reason is because I finally realised that I was completely locked in to terraform, same way as many are trying to avoid vendor lock-in's but not realising terraform discretely managed to create another term ... Tooling lock in.

One think I did tho in pulumi is making myself completely independant of their cloud APIs ( for state files) but luckily automaton API allows you to do that.

[danielloader]

@JayDoubleu Sounds good, using Pulumi cloud for state or S3 etc? I do wonder if it ultimately if it matters what to use, to do the work - The only selling point for CDKTF for me is we're already using Terraform Cloud for state management - and I'd be happy to continue doing so with this functionality in AWS Lambda functions - it's just not formalised into a public API so can't risk it - though I guess the whole CDKTF API isn't formalised yet.

[JayDoubleu]

using Pulumi cloud for state or S3 etc? I don't run any IaC code with third party solutions, I just don't like the idea of third parties having visibility into a state given it's sensitivity regardless of how "strong" they security or encryption would be. So yes, the answer is S3/Azure Blob storage.

My main selling point for pulumi is "native" providers. Meaning if it's out there available as API on Azure or AWS it's automatically available for you to use within automation. While terraform has a layer of providers and I find myself and other teams spending too much time "maintaining" and fixing bugs with providers/terraform rather than focusing on actual infrastructure. Also it's often that people are now recruiting the "terraform" experts with knowledge on how to fix bugs and work around terraform rather than general expert knowledge about programming/scripting language + infrastructure. With pulumi these problems do go away. In fact you can even build your own "terraform" like product with pulumi while you can't do it the other way around.

[danielloader]

As a follow up, Pulumi is agonisingly slow to resolve changes compared to CDKTF so guess I'll be hoping this "automation API" can be formalised soonish. Good job to Hashicorp.

[irfanjs]

Hi All, when can we see this feature release ? Any ETA please ? we are seriously looking for API integration , how we can call this synth process (plan, apply and destroy) using sprintBoot ? we are developing some JAVA API which will call terraform to deploy the EKS cluster in AWS . something like POST http://localhost/api/eks/create will hit the cdktf synth process and deploy the EKS cluster .

please suggest

[anamitra-saikia]

Hi all, It will be fantastic to have this API feature in Terraform cdk. Any estimated time for the development of this feature? Also @skorfmann the bypass for cdktf-cli you mentioned is exactly what I was looking for but it is applicable for old versions of the cli. Could you show how to achieve the same in the current version of cdktf-cli?

[irfanjs]

@anamitra-saikia : No workaround yet and havent implemented . Still waiting for this feature from Terraform-cdk