aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.65k stars 3.91k forks source link

(aws-codepipeline): Pipeline stack which uses cross-environment actions must have an explicitly set region #20011

Closed rantoniuk closed 2 years ago

rantoniuk commented 2 years ago

Describe the bug

cdk synth throws the below error after upgrading aws-cdk@1.x to the current version. The returned error message is non-specific, i.e.

Offending code:

    at new Stage (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/private/stage.ts:40:12)
    at Pipeline.addStage (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:461:19)
    at new UiPipelineConstruct (cdk/node_modules/@imi/cdk/dist/src/pipelines/ui-pipeline-construct.js:58:23)

is this:

    this.pipeline.addStage({
      stageName: 'Deploy-TRAINING',
      actions: [
        new CloudFormationCreateUpdateStackAction({
          runOrder: 2,
          actionName: 'DeployCF',
          stackName: props.deployedStackName,
          adminPermissions: false,
          role: this.trainingPipelineRole,
          deploymentRole: this.trainingPipelineChangesetRole,
          templatePath: this.cdk.artifact.atPath(`${templatePath}`),
          cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
          region: Stack.of(this).region,
        }),
        new S3DeployAction({
          runOrder: 3,
          actionName: 'DeployS3',
          role: this.trainingPipelineRole,
          input: trainingBuild.artifact,
          bucket: trainingUiBucket,
        }),
        new LambdaInvokeAction({
          runOrder: 4,
          actionName: 'InvalidateCloudFront',
          role: this.devOpsPipelineRole,
          lambda: cfInvalidationLambda,
          userParameters: {
            bucketName: trainingUiBucket.bucketRegionalDomainName,
            targetPipelineRole: this.trainingPipelineRole.roleArn,
          },
        }),
      ],
    });

Expected Behavior

Previously working fine.

Current Behavior

cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:1045
      throw new Error('Pipeline stack which uses cross-environment actions must have an explicitly set region');
            ^
Error: Pipeline stack which uses cross-environment actions must have an explicitly set region
    at Pipeline.requireRegion (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:1045:13)
    at Pipeline.ensureReplicationResourcesExistFor (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:580:10)
    at Pipeline._attachActionToPipeline (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:528:34)
    at Stage.attachActionToPipeline (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/private/stage.ts:155:27)
    at Stage.addAction (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/private/stage.ts:93:29)
    at new Stage (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/private/stage.ts:40:12)
    at Pipeline.addStage (cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:461:19)
    at new UiPipelineConstruct (cdk/node_modules/@imi/cdk/dist/src/pipelines/ui-pipeline-construct.js:58:23)
    at new DynamicPipelineStack (cdk/node_modules/@imi/cdk/dist/src/pipelines/dynamic-pipeline-stack.js:23:17)
    at Object.<anonymous> (cdk/bin/economic-operator-fo.ts:20:3)

Reproduction Steps

not working on: 1.152.0 (build 9487b39) working on: 1.127.0

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

1.152.0 (build 9487b39)

Framework Version

No response

Node.js Version

v14.19.1

OS

MacOS

Language

Typescript

Language Version

"typescript": "~4.3.5"

Other information

No response

rantoniuk commented 2 years ago

The problem seems to be in the LambdaInvokeAction, because after removing that in the generated .js file the problem disappears. However, LambdaInvokeAction() does not provide a way to specify region, nor does the parent Pipeline(). Moreover, this pipeline is not deployed in different regions, it's a single-region deployment that was previously working fine.

skinny85 commented 2 years ago

Thanks for opening the issue @rantoniuk.

Can you show what is cfInvalidationLambda in your code?

rantoniuk commented 2 years ago
const cfInvalidationLambda = Function.fromFunctionArn(
      this,
      'cfInvalidationLambda',
      'arn:aws:lambda:eu-west-1::function:CloudfrontInvalidationLambda',
    );
skinny85 commented 2 years ago

Yeah, so here's the issue.

I assume you skipped the account ID for anonymity reasons, but there is some account ID there, right? And is it the same account that the pipeline itself is in?

rantoniuk commented 2 years ago

No, I haven't skipped anything - this is exactly the ARN that is used, using the default ARN notation that assumes the current account id if not provided by default from CloudFormation].

I believe this is a regression because it all worked fine in earlier versions.

skinny85 commented 2 years ago

Hmm, where did you get this concept of "default ARN"? The word "default" does not appear anywhere in the page you linked to. I've never seen ARNs used that way, and CDK doesn't interpret missing account the way you described.

I think if you switch from fromFunctionArn() to the new fromFunctionName() method, this should start working correctly again.

rantoniuk commented 2 years ago

The ID of the AWS account that owns the resource. When you use the account number in an ARN or an API operation, you omit the hyphens (for example, 123456789012). The ARNs for some resources don't require an account number, so this component might be omitted. Amazon QuickSight ARNs require an AWS account number. However, the account number and the AWS Region are omitted from S3 bucket ARNs, as shown following.

Trying to switch to:

const cfInvalidationLambda = Function.fromFunctionName(
      this,
      'cfInvalidationLambda',
      'CloudfrontInvalidationLambda',
    );

ends with this:

cdk/node_modules/@aws-cdk/aws-lambda/lib/function-base.ts:421
              throw new Error('Cannot modify permission to lambda function. Function is either imported or $LATEST version.\n'
                    ^
Error: Cannot modify permission to lambda function. Function is either imported or $LATEST version.
If the function is imported from the same account use `fromFunctionAttributes()` API with the `sameEnvironment` flag.
If the function is imported from a different account and already has the correct permissions use `fromFunctionAttributes()` API with the `skipPermissions` flag.

Trying to use:

 const cfInvalidationLambda = Function.fromFunctionAttributes(this, 'cfInvalidationLambda', {
      functionArn: 'arn:aws:lambda:eu-west-1::function:CloudfrontInvalidationLambda',
      sameEnvironment: true,
    });

ends with the same:

cdk/node_modules/@aws-cdk/aws-codepipeline/lib/pipeline.ts:1045
      throw new Error('Pipeline stack which uses cross-environment actions must have an explicitly set region');

Even if I hardcode the AWS account number with:

const cfInvalidationLambda = Function.fromFunctionArn(
      this,
      'cfInvalidationLambda',
      `arn:aws:lambda:eu-west-1:${Accounts.DEVOPS}:function:CloudfrontInvalidationLambda`,
    );

which is coming from an enum, then it also does not work. And I still think this is a regression, because it was working before.

skinny85 commented 2 years ago

The ID of the AWS account that owns the resource. When you use the account number in an ARN or an API operation, you omit the hyphens (for example, 123456789012). The ARNs for some resources don't require an account number, so this component might be omitted. Amazon QuickSight ARNs require an AWS account number. However, the account number and the AWS Region are omitted from S3 bucket ARNs, as shown following.

Right. There's no mention of Lambda Function ARNs having an optional account.

Even if I hardcode the AWS account number with:

const cfInvalidationLambda = Function.fromFunctionArn(
      this,
      'cfInvalidationLambda',
      `arn:aws:lambda:eu-west-1:${Accounts.DEVOPS}:function:CloudfrontInvalidationLambda`,
    );

which is coming from an enum, then it also does not work. And I still think this is a regression, because it was working before.

"Does not work" - what is the error?

rantoniuk commented 2 years ago

There's no mention of Lambda Function ARNs having an optional account: I would debate that, the above sentence says: "when you use account number" == when means it's optional.

Same error:

      throw new Error('Pipeline stack which uses cross-environment actions must have an explicitly set region');
skinny85 commented 2 years ago

OK. I think you have 2 ways of unblocking yourself:

  1. Put the account of the Pipeline in the ARN of the Function you're importing, and then pass that same account (Accounts.DEVOPS) to the Stack that contains the Pipeline (the account property of the env property of the Stack).
  2. Try putting Aws.ACCOUNT_ID as the account in the ARN of the Function you're importing, and see if that helps.
rantoniuk commented 2 years ago

So this workaround works:

      `arn:aws:lambda:eu-west-1:${Accounts.DEVOPS}:function:CloudfrontInvalidationLambda`,

and also on the stack creation side:

 new DynamicPipelineStack(app, 'dynamic-pipeline-stack', {
    env: {
      region: process.env.CDK_DEFAULT_REGION,
    },
  });

Note that I'm passing only the region and not the account (and both should be picked up from default env, so still a bug).

skinny85 commented 2 years ago

Glad you got yourself unblocked @rantoniuk!

I'm keeping this open to investigate why using Function.fromFunctionName() does not work in this case.

rantoniuk commented 2 years ago

@skinny85 I just stumbled on it again with a very simple pipeline:


export class PipelineConstruct extends Construct {
  pipeline: Pipeline;

  constructor(scope: Construct, id: string) {
    super(scope, id);

    this.pipeline = new Pipeline(this, "TestPipeline", {});

    const build = new PipelineProject(this, "TestPipelineCodeBuild", {
      projectName: "MyTestPipelineCodeBuild",
    });

    const artifact = new Artifact();
    const action = new CodeBuildAction({
      actionName: "action",
      project: build,
      input: artifact,
    });

    this.pipeline.addStage({
      stageName: "Build",
      actions: [action],
    });

    this.pipeline.addStage({
      stageName: "Deployment-DEV",
      actions: [
        new CloudFormationCreateUpdateStackAction({
          runOrder: 1,
          actionName: "DeployCF",
          stackName: "TestStack",
          adminPermissions: false,
          // account: Accounts.TEST,
          templatePath: artifact.atPath("TestStack.template.json"),
          cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
        }),
      ],
    });
  }
}

// bin/app.ts:

new DynamicPipelineStack(app, "PipelineStack");

With account: Accounts.TEST commented out, cdk synth works fine. If I uncomment it, I get:

Error: Pipeline stack which uses cross-environment actions must have an explicitly set account
    at Pipeline.getOtherStackIfActionIsCrossAccount (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/pipeline.js:1:13697)
    at Pipeline.getRoleFromActionPropsOrGenerateIfCrossAccount (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/pipeline.js:1:12839)
    at Pipeline.getRoleForAction (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/pipeline.js:1:11476)
    at Pipeline._attachActionToPipeline (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/pipeline.js:1:7771)
    at Stage.attachActionToPipeline (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/private/stage.js:1:3087)
    at Stage.addAction (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/private/stage.js:1:1716)
    at new Stage (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/private/stage.js:1:678)
    at Pipeline.addStage (cdk/node_modules/aws-cdk-lib/aws-codepipeline/lib/pipeline.js:1:6662)
    at new PipelineConstruct (cdk/stacks/pipeline-stack.ts:185:19)
    at new DynamicPipelineStack (cdk/stacks/pipeline-stack.ts:32:5)

It gets funnier, when I specify both:

          account: Accounts.TEST,
          region: "eu-west-1",

the error complains about region missing :-)

Error: Pipeline stack which uses cross-environment actions must have an explicitly set region

cdk version 2.38.1 (build a5ced21)

skinny85 commented 2 years ago

Right. So, you're saying, in the CloudFormationCreateUpdateStackAction, that you want to deploy to the account Accounts.TEST. But, you're not specifying which account you want the pipeline itself to be in. Because of that, the Pipeline construct can't tell if this will be a "regular" CodePipeline, or a cross-account one, and hence why it raises this error.

The same thing goes for region.

rantoniuk commented 2 years ago

Hi Adam,

My point is that is should probably assume to use the current-region and current-account, shouldn't it?

If I don't set region and account and provide a cross-account role instead - it works fine. Moreover, I have a project working where:

This also works fine - but if I try to switch from setting roles explicitly to using account and roles, then I get the above error.

Following your response, I tried to set the env explicitly, but that did not help:

 new DynamicPipelineStack(app, "PipelineStack", {
    env: {
      account: process.env.CDK_DEFAULT_ACCOUNT,
      region: process.env.CDK_DEFAULT_REGION,
    },
  });

Setting it to explicit values also doesn't fix it and I don't see any other place where it can be set (e.g. on the Pipeline() construct level), can you advise where it should be set?

skinny85 commented 2 years ago

You can't leave account/region unset, or to "dynamic" values with CDK_DEFAULT_ACCOUNT/REGION, because this decision, whether a pipeline is cross-account/region, must be made at synth time, and synthesis doesn't know what AWS account you are running it for (and it shouldn't produce different results anyway depending on it - it should be deterministic, and always produce the same result from the same source).

So, it should be:

  new DynamicPipelineStack(app, 'PipelineStack', {
    env: {
      account: Accounts.TEST,
      region: 'eu-west-1',
    },
  });

When you provide the role yourself, we actually ignore the region / account properties (it should be stated in their docs) - we basically trust you at that point that you wired everything correctly.

rantoniuk commented 2 years ago

I was missing passing the props in the super() statement, so that's why it didn't work when I tried it. Thanks for detailed explanation @skinny85!

github-actions[bot] commented 2 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

BwL1289 commented 1 year ago

@skinny85 I am also now experiencing this error trying to run a very simple pipeline, and can't get around it without explicitly removing the env. If I remove env, it bootstraps fine. When env is defined (explicitly, as you stated above), I get: RuntimeError: Error: Pipeline stack which uses cross-environment actions must have an explicitly set region.

To be clear, I'm literally hardcoding the account # and region right now trying to debug this.

Here's my code:

      pipeline = pipelines.CodePipeline(
          self,
          id=id,
          cross_account_keys=True,
          synth=synth,
          self_mutation=False,
          docker_enabled_for_synth=True,
      )

      application = ApplicationStage(
          self,
          env=Environment(
              account="<omitted>",
              region="<omitted>",
          ),
      )

I have also tried turning self_mutation=True and playing with various parameters with no luck.

I've also double checked @rantoniuk's issue to see if i've left out a **kwargs anywhere, and I don't believe I have.

Would appreciate any help.

skinny85 commented 1 year ago

@BwL1289 you need to set the env in the Stack containing the pipeline - not in the Stage that the pipeline is deploying.

BwL1289 commented 1 year ago

@skinny85 sincerely appreciate the response.

In the docs they show you can add an env to a stage:

import aws_cdk as cdk
from constructs import Construct
from aws_cdk.pipelines import CodePipeline, CodePipelineSource, ShellStep
from my_pipeline.my_pipeline_app_stage import MyPipelineAppStage

class MyPipelineStack(cdk.Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        pipeline =  CodePipeline(self, "Pipeline", 
                        pipeline_name="MyPipeline",
                        synth=ShellStep("Synth", 
                            input=CodePipelineSource.git_hub("OWNER/REPO", "main"),
                            commands=["npm install -g aws-cdk",
                                "python -m pip install -r requirements.txt",
                                "cdk synth"]))

        pipeline.add_stage(MyPipelineAppStage(self, "test",
            env=cdk.Environment(account="111111111111", region="eu-west-1")))

Stages also support envs:

:param env: Default AWS environment (account/region) for ``Stack``s in this ``Stage``. Stacks defined inside this ``Stage`` with either ``region`` or ``account`` missing from its env will use the corresponding field given here. If either ``region`` or ``account``is is not configured for ``Stack`` (either on the ``Stack`` itself or on the containing ``Stage``), the Stack will be *environment-agnostic*. Environment-agnostic stacks can be deployed to any environment, may not be able to take advantage of all features of the CDK. For example, they will not be able to use environmental context lookups, will not automatically translate Service Principals to the right format based on the environment's AWS partition, and other such enhancements. Default: - The environments should be configured on the ``Stack``s.

Here's more of the boilerplate code:

class Pipeline(Stack):
    def __init__(
        self,
        scope: Construct,
        *,
        id: str,
        **kwargs,
    ):
        super().__init__(scope, id, **kwargs)

        source = codestar_connection_builder()
        synth = synth_step(id, source)

        pipeline = pipelines.CodePipeline(
            self,
            id=id,
            cross_account_keys=True,
            synth=synth,
            self_mutation=False,  # TODO: turn off for prod
            docker_enabled_for_synth=True,
        )

        application = ApplicationStage(
            self,
            env=Environment(
                 account="<omitted>", 
                 region="<omitted>",
            ),
        )

        app_stage = pipeline.add_stage(application)

What am I missing?

skinny85 commented 1 year ago

What am I missing?

Like I wrote above, the error is most likely because the Stack containing your CodePipeline construct needs the env parameter passed to it as well.

BwL1289 commented 1 year ago

@skinny85 still no luck. Same code as above, but with the env set on the Application, and I removed the env on the ApplicationStage:

class Application(Stack):
    @property
    def function_name(self) -> CfnOutput:
        return self._function_name

    def __init__(self, scope: Construct, id: str = "Application", **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        func = LambdaTestConstruct(self)
        self._function_name = CfnOutput(func, "FuncARN", value=func.function_name)

ApplicationStage(Stage):
    @property
    def umbrella_function_name(self):
        return self._umbrella.function_name

    def __init__(
        self,
        scope: Construct,
        *,
        env: Optional[Union[Environment, dict[str, Any]]] = None,
        outdir: Optional[str] = None,
        permissions_boundary: Optional[PermissionsBoundary] = None,
        stage_name: Optional[str] = None,
        id: str = "ApplicationStage",
    ) -> None:
        super().__init__(
            scope,
            id=id,
            env=env,
            outdir=outdir,
            permissions_boundary=permissions_boundary,
            stage_name=stage_name,
        )

        self._umbrella = Application(
            self, 
            env=Environment(
                account="<omitted>",
                region="<omitted>",
            ),
        )
BwL1289 commented 1 year ago

I finally figured this out. The issue was that I had a stack that created my Codepipeline and I was passing that to a separate stack: that's what was causing the error. The attempt was to modularize pipelines so multiple could be deployed together.

In other words, I had another Stack that I was passing into a different Stack, and that "other" Stack gets passed to app.py.

Instead, if you want to use this approach, use a Construct, not a Stack, to create your CodePipeline, and pass that into a different Stack.

I hope this makes sense and helps someone down the road.

JulianM33 commented 1 year ago

Using an aws_cdk.Environment instead of a dictionairy worked for me.

jacido commented 4 months ago

The issue was that I had a stack that created my Codepipeline and I was passing that to a separate stack: that's what was causing the error. The attempt was to modularize pipelines so multiple could be deployed together.

In other words, I had another Stack that I was passing into a different Stack, and that "other" Stack gets passed to app.py.

Instead, if you want to use this approach, use a Construct, not a Stack, to create your CodePipeline, and pass that into a different Stack.

Hi, would you happen to have a working example of this?