Programmatic access of AWS CDK CLI

hassankhan commented 5 years ago

Description

Hi there, thanks for creating this package! I was just wondering if there's any way of triggering CDK CLI commands programatically? This would really help with respect to integrating with existing tools and workflows.

~~RFC 300: Programmatic access of AWS CDK CLI~~

Roles

Role	User
Proposed by	@hassankhan, @mrgrain
Author(s)	@mrgrain
API Bar Raiser	@rix0rrr
Stakeholders

See RFC Process for details

Workflow

[x] Tracking issue created (label: status/proposed)
[x] API bar raiser assigned (ping us at #aws-cdk-rfcs if needed)
[x] Kick off meeting
[ ] RFC pull request submitted (label: status/review)
[ ] Community reach out (via Slack and/or Twitter)
[ ] API signed-off (label api-approved applied to pull request)
[ ] Final comments period (label: status/final-comments-period)
[ ] Approved and merged (label: status/approved)
[ ] Execution plan submitted (label: status/planning)
[ ] Plan approved and merged (label: status/implementing)
[ ] Implementation complete (label: status/done)

Author is responsible to progress the RFC according to this checklist, and apply the relevant labels to this issue so that the RFC table in README gets updated.

thdxr commented 1 year ago

We actually are accessing CDK programmatically for SST.

It definitely is not what CDK intends and could probably break at any point but you can look at how we deploy stacks here:

https://github.com/serverless-stack/sst/blob/sst2/packages/sst/src/stacks/deploy.ts#L76

mrgrain commented 1 year ago

Thanks all for the feedback so far. We have now released a first iteration of this: @aws-cdk/cli-lib-alpha

Please let me know what you're thinking and keep posting feature ideas. We are well aware this isn't a complete and ready solution yet (hence the experimental state), but it's an important stepping stone to gather interest and feedback.

louisdussarps commented 1 year ago

We may be a little late following previous AWS Annoucement. Still, please allow me to present how we programmatically manage the CDK via Orbits (an openSource flow for DevOps). It seems to me that Orbits' approch is still relevant if you have complex interactions to manage within your stacks or through a multi - steps deployment.

https://github.com/LaWebcapsule/orbits https://github.com/LaWebcapsule/orbits/tree/main/src/examples/git-cdk-s3

bastosvinicius commented 1 year ago

Hello @mrgrain, I'm trying to implement this using Python 3.11, but actually facing issue related below when running synth:

cli.synth()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/aws_cdk/cli_lib_alpha/__init__.py", line 2276, in synth
    return typing.cast(None, jsii.ainvoke(self, "synth", [options]))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 457, in ainvoke
    promise = self.provider.begin(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 387, in begin
    return self._process.send(request, BeginResponse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 327, in send
    self._process.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

My code is:

Running my click CLI command:

auk cloud aws instance create

Click method:

@aws.command('instance')
@click.pass_context
@click.argument('create', nargs=-1)
def instance(ctx, create: bool = False):
    print('create instances')
    if create:
        producer = Producer()
        cli = AwsCdkCli.from_cloud_assembly_directory_producer(producer)
        cli.synth(
            stacks=["EC2Stack"]
        )

aws.py file:

import os.path

from aws_cdk import Stack, App
from aws_cdk import aws_ec2 as ec2
from aws_cdk import aws_iam as iam
from aws_cdk.aws_s3_assets import Asset
from constructs import Construct
from aws_cdk.cli_lib_alpha import ICloudAssemblyDirectoryProducer
import jsii

dirname = os.path.dirname(__file__)

class EC2InstanceStack(Stack):

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, "VPC",
                      nat_gateways=0,
                      subnet_configuration=[ec2.SubnetConfiguration(
                          name="public", subnet_type=ec2.SubnetType.PUBLIC)]
                      )

        amzn_linux = ec2.MachineImage.latest_amazon_linux(
            generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
            edition=ec2.AmazonLinuxEdition.STANDARD,
            virtualization=ec2.AmazonLinuxVirt.HVM,
            storage=ec2.AmazonLinuxStorage.GENERAL_PURPOSE
        )

        role = iam.Role(self, "InstanceSSM",
                        assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))

        role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name(
            "AmazonSSMManagedInstanceCore"))

        instance = ec2.Instance(self, "Instance",
                                instance_type=ec2.InstanceType("t3.nano"),
                                machine_image=amzn_linux,
                                vpc=vpc,
                                role=role
                                )

        # # Script in S3 as Asset
        # asset = Asset(self, "Asset", path=os.path.join(
        #     dirname, "configure.sh"))
        # local_path = instance.user_data.add_s3_download_command(
        #     bucket=asset.bucket,
        #     bucket_key=asset.s3_object_key
        # )

        # # Userdata executes script from S3
        # instance.user_data.add_execute_file_command(
        #     file_path=local_path
        # )
        # asset.grant_read(instance.role)

@jsii.implements(ICloudAssemblyDirectoryProducer)
class Producer:
    def produce(self, context):
        app = App(context=context)
        # stack = Stack(app)
        stack = EC2InstanceStack(app, "EC2Stack")
        return app.synth().directory

Could you explain better how can I use this?

bastosvinicius commented 1 year ago

Following doc: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.cli_lib_alpha/README.html

Versions:

[tool.poetry.dependencies]
python = "^3.11"
google-cloud-compute = "^1.11.0"
click = "^8.1.3"
aws-cdk-lib = "^2.82.0"
aws-cdk-cli-lib-alpha = "^2.82.0a0"
jsii = "^1.82.0"

mrgrain commented 1 year ago

Hey @bastosvinicius Apologies for this, but unfortunately Python is currently not supported by this alpha module. While there are some scenarios where it seems to work, most don't. This is due to an issue with jsii and how async code is handled. =/

bastosvinicius commented 1 year ago

Thanks for reply @mrgrain. Is there a forecast for definitive python support?

mrgrain commented 1 year ago

It's pretty much this PR: https://github.com/aws/jsii/pull/3919 I was busy with other project work, hoping to get back to it in July.

bastosvinicius commented 1 year ago

I'll follow the PR and wait for the new version of jsii to be released to test it. Thanks for the feedback.

evgenyka commented 5 months ago

Should include other CLI commands, like cdk diff in the scope - https://github.com/aws/aws-cdk/issues/679

aws / aws-cdk-rfcs