search

aws / aws-cdk-rfcs

RFCs for the AWS CDK
Apache License 2.0
519 stars 81 forks source link

Programmatic access of AWS CDK CLI #300

Open hassankhan opened 5 years ago

hassankhan commented 5 years ago

Description

Hi there, thanks for creating this package! I was just wondering if there's any way of triggering CDK CLI commands programatically? This would really help with respect to integrating with existing tools and workflows.

RFC 300: Programmatic access of AWS CDK CLI

Roles

Role User
Proposed by @hassankhan, @mrgrain
Author(s) @mrgrain
API Bar Raiser @rix0rrr
Stakeholders

See RFC Process for details

Workflow

Author is responsible to progress the RFC according to this checklist, and apply the relevant labels to this issue so that the RFC table in README gets updated.

rix0rrr commented 5 years ago

Hi @hassankhan,

The aws-cdk NodeJS package does expose some library functions that can be used from JavaScript/TypeScript projects directly. However, it's not a polished experience at the moment, because our attention has been focused more towards the construct library so far. You're very welcome to try it and tell us if you run into any issues, though. As a last resort, you can always shell out to the command-line tool.

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

hassankhan commented 5 years ago

Hi @rix0rrr,

Thanks for the info! In an ideal world, I'd love to see something like:

import CDK from '@aws-cdk/cdk';

import MyApp from './MyApp';

const cdk = new CDK({ app: '', versionReporting: false });

cdk.deploy(new MyApp())
  .then(() => {
    return remove(new App());
  })

With regards to integration, essentially, our aim was to define resources such as Lambdas and DynamoDB tables once in our application code, and generate valid snippets from them and insert them into a template for deployment. For example, a theoretical DynamoDB model would look something like this (using dynamodb-data-mapper):

@table('pets')
class Pet {
    @hashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @attribute()
    isCat?: boolean;
}

The table schema would then be read from this class as part of the deployment process, converted into its constituent Construct and added to the App/Stack. We could use the same approach for other resource types such as SNS Topics and Lambda functions.

rix0rrr commented 5 years ago

Okay, thanks. I definitely see the value of the first one, especially in defining one-off tasks like EMR jobs or integ tests.

For the second one: you can already do this today, even without tool integration.

If you define you data model in a library, you can use this library both in your runtime code as well as introspect it in your infrastructure code. Your build step would produce both a CloudFormation template as well as a JAR, and during deployment you deploy both together.

rui-ktei commented 5 years ago

+1 I mentioned this in a duplicate ticket as well.

BDQ commented 5 years ago

I'd like this too, in my case I'd like to wrap a deploy of an ECS cluster so I can scale it manually after the deploy is done (I create it with desiredCount: 0 so the CloudFormation creating is quicker).

dsilvasc commented 5 years ago

Additional use cases from https://github.com/awslabs/aws-cdk/issues/1133 - synthesizing stacks from an IDE debugger and embedding into JVM build systems (like Gradle) without having to shell out to a CLI that depends on node.js.

mmoulton commented 5 years ago

My team is currently programmatically invoking CDK to support dynamic composition of constructs to create stacks based on the answers to question we ask in a custom CLI tool we wrote. We are currently primarily using it to spin up either SAM apps, or static sites using codepipeline, codebuild, rds, vpc, ssm params, s3 buckets, cloudfront, certs, zones, etc.

I'd very much like an officially supported way of accomplishing this. Currently we are doing some very dirty things to mimic the nature of the CDK CLI and the fact it executes the cdk app multiple times to meet dependencies.

If the CDK team does not have plans to officially support this, I'd very much appreciate not making it harder to achieve this as the notes in https://github.com/awslabs/aws-cdk/issues/2044 and https://github.com/awslabs/aws-cdk/issues/2016 suggest.

KarthickEmis commented 5 years ago

Is there anyway of triggering AWS CDK ClI (CDK synth , CDK deploy) commands pro-grammatically in typescript ?

auser commented 5 years ago

I found a way to handle this. I'll create an example repo (hopefully) to demonstrate how I accomplished this process, but the short version looks like this:

Create an AppStack instance and set the synthesizer to the output of app.run():

import { AppStacks } from 'aws-cdk/lib/api/cxapp/stacks'
import { Configuration } from 'aws-cdk/lib/settings';
// ...
const out = this._app.run();        
const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

const appStacks = new AppStacks({
    aws: this.sdk,
    configuration: configuration,
    synthesizer: async () => out
})

Using the AppStack instance, create an instance of the CdkToolkit:

const appStacks = await this.getAppStacks();

const provisioner = new CloudFormationDeploymentTarget({
    aws: this.sdk,
})
const cli = new CdkToolkit({ appStacks, provisioner });

In order to get the credentials, we'll need to muck around at a low-level with the AWS-SDK:

import { debug } from '@aws-rocket/core';
import AWS = require('aws-sdk')
import { SDK } from 'aws-cdk/lib/api/util/sdk';
import { Mode } from 'aws-cdk/lib/api/aws-auth/credentials'

async function getCredentialsConfig(sdk: SDK): Promise<any> {
    const region = await sdk.defaultRegion()
    const defaultAccount = await sdk.defaultAccount()
    const credentials = await (sdk as any).credentialsCache.get(defaultAccount, Mode.ForReading)

    return {
      region,
      credentials
    }
  }

export const awsCredentialsMiddleware = async (argv: any) => {
    debug(`Setting profile to: ${argv.profile}`)
    if (argv.profile) {
        const sdk = new SDK({
            profile: argv.profile
        })
        const credentials = await getCredentialsConfig(sdk)
        AWS.config.update(credentials)
        AWS.config.setPromisesDependency(Promise);
        argv.AWS = AWS
        argv.SDK = sdk
    }
    return argv
}

Finally, this all comes together using the actual deploy like so:

const appStacks = await this.getAppStacks();
const allStacks = await appStacks.listStacks()
const allStackNames = allStacks
    .map((s: cxapi.CloudFormationStackArtifact) => s.name)

const cli = await this.getCdkToolkit()
try {
    const res = await cli.deploy({
        stackNames: allStackNames,
        // `requireApproval` is so that the user never needs to 
        // accept anything for the deploy to actually occur
        requireApproval: RequireApproval.Never,
    })
    console.log('deploy result ->', res);
} catch (e) {
    console.error(`Error deploying`, e);
    process.exit(-1)
}

Hope this helps and hopfully this becomes a part of the actual library. I'll try to submit a PR if I can, but no promises. If anyone else has any better way of handling this, I'd love suggestions.

jd-carroll commented 4 years ago

This is all very interesting and hopeful that it will get implemented 🤞

@auser - Were you ever able to put together a repo / gist with all of this information? I follow the logic but I'm not sure how I follow how to assemble all of the pieces. Is each code snippet supposed to be in the same file and if so does each code section follow one an other? I can guess, but I also don't know where the this._app and this._argv come from.

@fulghum / @shivlaks - It looks like there is some movement towards an implementation. Do feel programmatic access might be limited in some areas or is the goal to have a full API access the same you would have on command line?

More or less, do you think something like the following would be supported:

import AWS from 'aws-sdk';
import { deploy } from 'aws-cdk';
import MyStack from './my-stack';

const cfnOutput = await deploy(new MyStack());
console.log(`Successfully deployed stack:\n${cfnOutput}`);

const s3FileAndDetails = { /* ... */ };
const s3 = new AWS.S3();
await s3.putObject(s3FileAndDetails).promise();

Also, instead of needing to run the configuration like @auser has above:

const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

It would be great if we could skip the command line altogether and just pass in a json object with relevant configuration. (Thereby eliminating the need for a cdk.json, but one could exist with overrides)

// Use factory function to allow reading config overrides in async
const configuration = await Configuration.create({ /* json from cdk.json */ })
ryan-mars commented 4 years ago

If you want to synth programmatically to the directory of your choice... 

// Let's say you want it synthed to ./build
const outdir = path.join(process.cwd(), 'build') 
// App's constructor let's you specify the outdir 
const app = new cdk.App({outdir})
new ApiLambdaCrudDynamoDBStack(app, 'ApiLambdaCrudDynamoDBExample')
app.synth()

If you want to inspect what was produced app.synth() returns a CloudAssembly which can be interrogated with getStack('your stack name')

I haven't tried deploying programmatically yet.

jd-carroll commented 4 years ago

When implementing this feature, would it make sense to do it in the context of aws-sdk-js-v3? https://github.com/aws/aws-sdk-js-v3

ralovely commented 4 years ago

(One of) the premise of CDK being "programmatic CloudFormation", it opens the way for fully programmable infrastructure, potentially even starting with the account creation (and that's awesome, we (building an "Ops as a Service" product) have been hoping for this for quite some time, even looking into developing our own solution). Creating a full, real-life infrastructure requires more features than CloudFormation can offers, even considering CustomResources. Some of those will require the SDK ; some others need to talk to 3rd party services ; fetch data before ; output/notify after.

Being able to use CDK not so much as a toolkit/cli, not as a complete solution, but as a lib, as a way to integrate the infrastructure-control layer into a programm would be great. As great as CDK promise to be, it's not a complete solution, probably can't, and shouldn't even try to be one. Of course, one can always shell out, wrap around CDK CLI, but that's not ideal.

It ties in with a few other GitHub issues:

jfaixo commented 4 years ago

Another use case : when your CDK stack creates Cognito UserPool Client Apps, it generates client id and secrets that need to be injected in your clients. If your client is a webapp in S3, that means you have to do steps in this order:

If it was possible to deploy programmatically, it could be possible to have the whole process written in the same tool (for example nodejs). Otherwise (as of now I do this way), you have to rely to some other external tool to call the CLI multiple times and schedule these actions (like a bash script, or other nodejs script through process exec..).

abierbaum commented 4 years ago

I just found a nice little helper that wraps some of this here: https://github.com/dmitrii-t/cdk-util May be worth a look for people that want to do this type of thing. Definitely can't wait to see it in the core API with first class support.

ryan-mars commented 4 years ago

@jfaixo

If it was possible to deploy programmatically

It is. 

import * as s3 from "@aws-cdk/aws-s3";
import * as s3deploy from "@aws-cdk/aws-s3-deployment"; // <<--- this right here
import * as cdk from "@aws-cdk/core";

export class WebStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const websiteBucket = new s3.Bucket(this, "WebsiteBucket", {
      websiteIndexDocument: "index.html",
      publicReadAccess: true,
      versioned: true
    });

    new s3deploy.BucketDeployment(this, "DeployWebsite", {
      sources: [s3deploy.Source.asset("./build")], // <<--- folder to upload
      destinationBucket: websiteBucket
    });
  }
}
abierbaum commented 4 years ago

Unfortunately I have found that the code and helpers I can find online are all out of date with the latest version of the CDK. How are people using the CDK in the case where you have a command line tool that needs to dynamically allocate and manage resources on AWS?

sukhwant commented 4 years ago

I am looking to use it from JAVA.

zyrorl commented 4 years ago

I've been able to sort of get this working programatically but have discovered that the cli.deploy etc functions return void, undefined, boolean and 0/1 values instead of useful values like cloudformation stack arns, outputs etc.

shivlaks commented 4 years ago

from @benwainwright

Currently, I can get a list of stacks that will be generated by my project in the order they should be deployed (this is important; see below) by running the command cdk list It would be useful if the output from this command could be exposed as an api in a way that doesn't require full synthesis of the templates under the hood.

full use case in aws/aws-cdk#8436

Guitarkalle commented 4 years ago

This would be wonderful. Now it feels like CI integration is a pain.. You have to parse output for changeset names and stack ids after deploying to do any more serious CI flow. Using the synthesizing & deploy parts of the tool programmatically and also having the CLI able to output in a JSON format instead of just console logs with necessary information would be perfect

moshir commented 3 years ago

Hi, any update on this one ? I'm trying to automate the creation of AWS resources from a lambda function, and would be great to have a programmatic API in addition to the cdk CLI.
Alternatively, is there any way to simply get the cloudformation template string generated by synth(), and then pass it to boto3 cloudformation API ?

rossng commented 3 years ago

This would be very useful for my use case: dynamically deploying MediaLive broadcast pipelines. I don't really see shelling out to the CLI as a viable option.

ignaloidas commented 3 years ago

This would also be very useful for us for running integration tests. We currently shell out to CDK CLI for this, but that means we have to do some annoying things like having a separate app in test folder that adds more outputs, etc. Ability to deploy stacks with programmatic usage would help greatly for our use case.

thenikso commented 3 years ago

I managed to do this following some @ryan-mars suggestions.

yarn add aws-cdk
yarn add @aws-cdk/core
# plus all the @aws-cdk/... you need
import { App, Stack } from '@aws-cdk/core';
import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';
import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';

class ExampleStack extends Stack { /* your stack here */ }

const app = new App();
const stack = new ExampleStack(app, 'StackName', {});

const stackArtifact = app.synth().getStackByName(stack.stackName);

const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({
  profile: 'your ~/.aws/config profile name here',
});
const cloudFormation = new CloudFormationDeployments({ sdkProvider });
const deployResultPromise = cloudFormation.deployStack({
  stack: stackArtifact,
});
iDVB commented 3 years ago

I managed to do this following some @ryan-mars suggestions.


yarn add aws-cdk

yarn add @aws-cdk/core

# plus all the @aws-cdk/... you need


import { App, Stack } from '@aws-cdk/core';

import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';

import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';

class ExampleStack extends Stack { /* your stack here */ }

const app = new App();

const stack = new ExampleStack(app, 'StackName', {});

const stackArtifact = app.synth().getStackByName(stack.stackName);

const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({

  profile: 'your ~/.aws/config profile name here',

});

const cloudFormation = new CloudFormationDeployments({ sdkProvider });

const deployResultPromise = cloudFormation.deployStack({

  stack: stackArtifact,

});

What does the deployResultPromise response look like? Is there a way at that point to get the stack resolved outputs?

thenikso commented 3 years ago

What does the deployResultPromise response look like? Is there a way at that point to get the stack resolved outputs?

It's an (experimental) DeployStackResult when resolved.

I haven't tried to parse it yet, not sure how to use it. Yet I'm guessing the outputs map contains everything. Worst case the stackArn could be used with the regular SDK to get the stack status

iDVB commented 3 years ago

@thenikso noice! Thanks! I've gonna give something a try now.

rossng commented 3 years ago

This workaround has been really useful for me. We are now using it to dynamically deploy our MediaLive+MediaPackage+CloudFront stacks at runtime.

To keep things relatively reliable and avoid request timeouts, I don't await the result of deployment. Instead I:

I'd love to see this become an officially supported API, since it's much better than the alternatives of using the SDK manually or using traditional CloudFormation configuration files. Hopefully this example code helps anyone else who's trying to do something similar 🙂

suryatejabandlamudi commented 3 years ago 
import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';
import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';
...........................
.............................
const cloudFormation = new CloudFormationDeployments({ sdkProvider });
const deployResultPromise = cloudFormation.deployStack({
  stack: stackArtifact,
});

@thenikso I understand that CDK Toolkit is written in TypeScript. But is there some way I could use python to do a similar deployment? As most of our codebase is in python. It would be helpful to us if you could point me in the right direction, I could take it from there.

Something like :

from aws_cdk import ( aws_cdk.lib as cloud_formation_deployment )
 .....
cloud_formation_deployment.deploy_stack(stack)
codeedog commented 3 years ago

See @thenikso comment of 12 March 2021 for context.

I had to formulate the deploy call as an async call at the module level. This compiled and ran. My particular application requires that I pass in a variable stack name. Although I'm happy to run from a shell, I need to parameterize the name. A programmatic API would be one way to solve this problem.

(async () => {
  const synth = app.synth().getStackByName(stack.stackName)
  const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({ profile: '<aws profile>' });
  const cloudFormation = new CloudFormationDeployments({ sdkProvider: sdkProvider});
  const deployResultPromise = cloudFormation.deployStack({ stack: synth });
})();

I wound up going a different way. Inside my ./bin/platform.ts invocation code, I look for an environment variable and use that instead. Even so, the above code worked well for me at the time.

iDVB commented 3 years ago

@thenikso, how did you get top-level await working in cdk?

Also, I'm reading through this other issue and wondering the same thing about how to get programmtic use of CDK to use the same settings and defaults as the CLI. For starters how to pass in context or get it to use the context file?

thenikso commented 2 years ago

as @codeedog noted, you might want to wrap it all in an async function

pharindoko commented 2 years ago

are there any plans to officially support using only the api functions without the cli ? This issue is open for 3 years now ...

phitoduck commented 2 years ago

+1 for this. My team has been using troposphere for a while, and I'm seriously envious of the CDK community. Here are some use cases our team has been using the programmatic API of troposphere (a Python IaC library) to solve:

  1. A custom CLI tool to abstract the hard parts of creating REST APIs. Our data scientists are able to place a app-config.yml file in a repository. They then write a FastAPI app compatible with Lambda to serve their models. Then they run our-custom-cli-tool deploy serverless-rest-api --config app-config.yml. This is great, because the data scientists don't have to know anything about troposphere, aws-cdk, or any other IaC tool, but they can easily deploy things.

  2. Integration testing: with troposphere, we can create a stack--for example, a serverless REST API--and then run tests against it. That could mean sending several requests to a REST API and watching for expected responses. If the tests fail, we tear down the stack. Sure, you can do this with a sequence of CLI calls, or maybe by calling CDK as a subprocess, but it feels clunky.

  3. Using stack outputs as inputs to other stacks at runtime. Suppose I want to create an IAM user and put those user credentials into the user-data.sh script of an AWS Lightsail instance. First, I would create an IAM user stack and save the credentials in AWS Secrets Manager. Then, I would wait for that creation to complete and fetch the credentials. Finally, I would create a stack with an AWS Lightsail instance, rendering the credentials into the user-data.sh body.

Conclusion:

I saw that pulumi has an "Automation API" in all the same languages as CDK (they're probably using the JSII inspired by AWS CDK, haha). That said, I've really enjoyed the abstraction of CDK and the community support. The other day I created an S3 bucket with the CDK construct. I set the "expire" policy so objects would auto-delete. Then I saw that to achieve that, an entire lambda function had been created to react to "expiry" events and delete the associated objects. That abstraction is so much nicer than anything we've had with troposphere!

I would dearly love to re-write these apps using AWS CDK. I have no idea the complexity it would take to implement this, since it goes against a fundamental assumption of CDK as of now, but I'd sure be grateful.

pharindoko commented 2 years ago

Yeah pulumi is on my try-out list.

I made my experience with CDK and it had a benefit for simple small serverless workloads. For all other things it was just a pain to work with. In the end it`s just a wrapper for cloudformation and that is the root cause of all nasty IaC problems.

Will use my old good terraform or try out cdktf for real usecases.

mnapoli commented 2 years ago

Watch out, programmatic usage of the CDK was strongly impacted in the latest releases: https://github.com/aws/aws-cdk/pull/18667

If you used that approach this is now broken.

rossng commented 2 years ago

Now that the CDK is bundled into a single package this all seems like a bit of a lost cause - at least for us it's impractically big to include in our production bundles. I think it's probably better to use CDK to generate CloudFormation templates ahead of time.

AntonioAngelino commented 2 years ago

We widely use AWS CDK programmatically, so we will be forced to stick with aws-cdk 2.13.0. I really hope we won't be forced to fork CDK in future 🤦‍♂️

Tanemahuta commented 2 years ago

Please see: https://github.com/aws/aws-cdk/pull/18667#issuecomment-1075348390

fab-mindflow commented 2 years ago

+1. We also use AWS CDK intensively and programmatically.

3p3r commented 2 years ago

programmatic access to the CLI (or at least its functionality) has been a major pain point at two enterprises I worked for and personally I find it unacceptable to just shell out to the CDK CLI to do anything. shelling out is not a secure practice in any environment outside of dev, specifically when people want to integrate with CDK in existing pipelines.

Two doable workarounds exist as of today (that I know of):

  1. using the stack artifact directly, like I do in cdk-web: https://github.com/3p3r/cdk-web/blob/main/cdk-web-cli.js
  2. using cxapi and reloading the artifacts from disk/memory as mentioned here

Unfortunately both approaches rely on CloudFormationDeployments class which is an export out of the CLI package. it seems like offering this API directly as an official API is a great place to start working on resolving this RFC.

It is worth noting that using the AWS SDK to "just deploy" the artifacts does not work at all. Complicated stacks require assets, context lookups, and other metadata related tasks to be done normally via the CLI. so that route is almost like duplication of the CLI's source.

Fortunately, my work has been in Node, JS, and TS and I am lucky to have require and deep imports in Node. other languages and environments are not that lucky.

mnapoli commented 2 years ago

Unfortunately both approaches rely on CloudFormationDeployments class which is an export out of the CLI package.

That's the problem, this class is no longer usable since https://github.com/aws/aws-cdk/pull/18667

3p3r commented 2 years ago

@mnapoli it is usable in Node still. You need to direct require it outside its deep import. That’s a node core feature. It can’t be taken away. Only import syntax won’t work directly. It will always be possible to require anything until node core stops supporting that.

phitoduck commented 2 years ago

@mnapoli Sorry if this is a naive question. What would it take for this feature to move forward?

I feel like this is potentially such an amazing feature that would close a large gap between what CDK offers and what Pulumi offers (thinking of their "Automation API"). As the comments in this thread point out, it would open up a world of totally relevant use cases that many of us IaC users have to meet.

mnapoli commented 2 years ago

For clarity I don't work on the CDK, I'm just reporting a discussion that happened in an issue.

But from what I discussed with some of the CDK maintainers it seems that it's mostly about the effort and they are not ready to invest into it yet, and also that they think they would need to provide a cross-language API for all languages CDK supports (instead of just Node).

My personal wish is that they'd focus on Node only first, and start by not breaking the public API that's already available (i.e. it works already, just don't make it worse).

AntonioAngelino commented 2 years ago

My personal wish is that they'd focus on Node only first, and start by not breaking the public API that's already available (i.e. it works already, just don't make it worse).

Totally agree!

lestephane commented 2 years ago

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

I'm writing integration tests in Kotlin (because the project's most important parts are lambda functions written in Kotlin, in the same monorepo), while keeping the cdk definitions in typescript, and would like to deploy the infrastructure in --hotswap mode without spawning a system command, to give a better error report in the IDE's test results panel when something goes wrong.

coding-bunny commented 2 years ago

Three years and still no suitable solution, or even feedback about this... We are in the same boat as everyone else, with having a custom CLI tool for deploying to AWS and so far the only option is to redo this all through the AWS CLI, which doesn't even allow clean parameter passing and what not unless you use --context.

The this thread started off promising with the ability to invoke the synth and deploy through code, but with that being broken now as well this option is out....

sladg commented 1 year ago

Is there any progress or other tickets we can track for this feature? :)