[aws-ec2] lookup functions ignore --profile

ps-spark commented 2 months ago

Describe the bug

Hi Guys,

I'm trying to utilize the lookup functions in the ec2 package but it seems that they ignore the env settings in the AWS config file.

ec2.Vpc.from_lookup(self, id="something", tags={"Name": "some_name"})
ec2.SecurityGroup.from_lookup_by_name(self, "SecurityGroupLookup", "ElasticMapReduce-master", emr_vpc)

If I execute cdk diff --profile my-profile the above lines fail with the following error message:

raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Error: Cannot retrieve value from context provider vpc-provider since account/region are not specified at the stack level.
Configure "env" with an account and region when you define your stack.
See https://docs.aws.amazon.com/cdk/latest/guide/environments.html for more details.

If I explicitly define the env in the code then the above code is working. myenv = cdk.Environment(account="xxxx", region="eu-central-1")

Based on the docs --profile should be equivalent with env in the code. I don't want to hard-code the env!

Expected Behavior

Lookup functions work with cdk diff --profile my-profile call.

Current Behavior

RuntimeError(resp.error) from JavaScriptError(resp.stack)

Reproduction Steps

Use any lookup function from ec2 and try to run cdk diff --profile my-profile

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.150.0

Framework Version

No response

Node.js Version

20.16.0

OS

Amazon Linux 2023

Language

Python

Language Version

3.10.12

Other information

No response

ashishdhingra commented 2 months ago

@ps-spark Good afternoon. As noted in comment in your stack entry point where CDK app is initialized: TypeScript (bin\somename.ts)

...
/* Uncomment the next line to specialize this stack for the AWS Account
   * and Region that are implied by the current CLI configuration. */
env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
...

Python (app.py)

..
    # Uncomment the next line to specialize this stack for the AWS Account
    # and Region that are implied by the current CLI configuration.

    env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')),
...

Please uncomment the relevant lines which initializes env to populate account and region.

Thereafter, if you run cdk diff --profile someprofile -v (notice the -v verbose flag), you will notice the values for CDK_DEFAULT_ACCOUNT and CDK_DEFAULT_REGION being populated from your profile (replace someprofile with your AWS CLI configured profile).

For example, for the following CDK app and stack: app.py

#!/usr/bin/env python3
import os

import aws_cdk as cdk

from cdktest_python.cdktest_python_stack import CdktestPythonStack

app = cdk.App()
CdktestPythonStack(app, "CdktestPythonStack",
    # If you don't specify 'env', this stack will be environment-agnostic.
    # Account/Region-dependent features and context lookups will not work,
    # but a single synthesized template can be deployed anywhere.

    # Uncomment the next line to specialize this stack for the AWS Account
    # and Region that are implied by the current CLI configuration.

    env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')),

    # Uncomment the next line if you know exactly what Account and Region you
    # want to deploy the stack to. */

    #env=cdk.Environment(account='123456789012', region='us-east-1'),

    # For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html
    )

app.synth()

cdktest_python\cdktest_python_stack.py

from aws_cdk import (
    # Duration,
    Stack,
    aws_ec2 as ec2,
)
from constructs import Construct

class CdktestPythonStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        ec2.Vpc.from_lookup(
            self, "MyVpc",
            is_default=True
        )

Running cdk diff --profile testprofile -v (assuming that we have a profile named testprofile configured via AWS CLI) returns the following verbose output:

[15:15:48] CDK toolkit version: 2.151.0 (build b8289e2)
...
[15:15:50] outdir: cdk.out
[15:15:50] env: {
  CDK_DEFAULT_REGION: 'us-east-2',
  CDK_DEFAULT_ACCOUNT: '<<ACCOUNT-ID-REDACTED>>',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '36.0.0',
  CDK_CLI_VERSION: '2.151.0'
}
[15:15:54] Some context information is missing. Fetching...
[15:15:54] Retrieved account ID <<ACCOUNT-ID-REDACTED>> from disk cache
[15:15:54] Assuming role 'arn:aws:iam::<<ACCOUNT-ID-REDACTED>>:role/cdk-hnb659fds-lookup-role-<<ACCOUNT-ID-REDACTED>>-us-east-2'.
[15:15:54] Listing VPCs in 139480602983:us-east-2
[15:15:54] Describing VPC vpc-0bcd157c9276f648b
[15:15:55] Setting "vpc-provider:account=<<ACCOUNT-ID-REDACTED>>:filter.isDefault=true:region=us-east-2:returnAsymmetricSubnets=true" context to {"vpcId":"vpc-<<REDACTED>>","vpcCidrBlock":"<<REDACTED>>","ownerAccountId":"<<ACCOUNT-ID-REDACTED>>","availabilityZones":[],"subnetGroups":[{"name":"Public","type":"Public","subnets":[{"subnetId":"subnet-<<REDACTED>>","cidr":"<<REDACTED>>","availabilityZone":"us-east-2a","routeTableId":"rtb-<<REDACTED>>"},{"subnetId":"subnet-<<REDACTED>>","cidr":"172.31.16.0/20","availabilityZone":"us-east-2b","routeTableId":"rtb-<<REDACTED>>"},{"subnetId":"subnet-<<REDACTED>>","cidr":"<<REDACTED>>","availabilityZone":"us-east-2c","routeTableId":"rtb-<<REDACTED>>"}]}]}
[15:15:55] Setting "CDK_DEFAULT_REGION" environment variable to us-east-2
[15:15:55] Setting "CDK_DEFAULT_ACCOUNT" environment variable to <<ACCOUNT-ID-REDACTED>>
[15:15:55] context: {
  'vpc-provider:account=<<ACCOUNT-ID-REDACTED>>:filter.isDefault=true:region=us-east-2:returnAsymmetricSubnets=true': {
    vpcId: 'vpc-<<REDACTED>>',
    vpcCidrBlock: '172.31.0.0/16',
    ownerAccountId: '<<ACCOUNT-ID-REDACTED>>',
    availabilityZones: [],
    subnetGroups: [ [Object] ]
  },
  '@aws-cdk/aws-lambda:recognizeLayerVersion': true,
  '@aws-cdk/core:checkSecretUsage': true,
...
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '**' ]
}
[15:15:55] outdir: cdk.out
[15:15:55] env: {
  CDK_DEFAULT_REGION: 'us-east-2',
  CDK_DEFAULT_ACCOUNT: '<<ACCOUNT-ID-REDACTED>>',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '36.0.0',
  CDK_CLI_VERSION: '2.151.0'
}
Stack CdktestPythonStack
[15:15:58] Retrieved account ID <<ACCOUNT-ID-REDACTED>> from disk cache
[15:15:58] Assuming role 'arn:aws:iam::<<ACCOUNT-ID-REDACTED>>:role/cdk-hnb659fds-lookup-role-<<ACCOUNT-ID-REDACTED>>-us-east-2'.
[15:15:59] Call failed: describeStacks({"StackName":"CdktestPythonStack"}) => Stack with id CdktestPythonStack does not exist (code=ValidationError)
[15:15:59] Retrieved account ID 139480602983 from disk cache
[15:15:59] Call failed: describeStacks({"StackName":"CdktestPythonStack"}) => Stack with id CdktestPythonStack does not exist (code=ValidationError)
[15:15:59] the stack 'CdktestPythonStack' has not been deployed to CloudFormation or describeStacks call failed, skipping changeset creation.
Parameters
[+] Parameter BootstrapVersion BootstrapVersion: {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/cdk-bootstrap/hnb659fds/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"}

Other Changes
[+] Unknown Rules: {"CheckBootstrapVersion":{"Assertions":[{"Assert":{"Fn::Not":[{"Fn::Contains":[["1","2","3","4","5"],{"Ref":"BootstrapVersion"}]}]},"AssertDescription":"CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."}]}}

✨  Number of stacks with differences: 1

Notice how CDK_DEFAULT_REGION and CDK_DEFAULT_ACCOUNT environment variables are automatically populated from the AWS CLI profile.

Hope this helps.

Thanks, Ashish

ps-spark commented 2 months ago

Hi @ashishdhingra,

Thank you, I read that but I already forgot it :( OK, it's not a bug then. I don't say that it's a beautiful solution but it works. Drawback: you have to define the env parameter in each stack you create.

Best wishes, Peter

github-actions[bot] commented 2 months ago

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

aws / aws-cdk