Cross-region/account references

[rix0rrr]

We were in the progress of defining how to transparently make this work.

Tracking it here because it's one of the most important things to pick up post-release.

[rix0rrr]

Use case: cross-region replication in combination with encryption.

• Stack A -- defines the target bucket + a KMS key
• Stack B -- defines the source bucket + an IAM role to use for replication.

For replication, the IAM role must must reference both buckets (local reference + one-way cross-stack reference)

To add encryption, the role must also reference the KMS Key, and the KMS key must also reference the role (bidirectional reference between stacks!)

[rix0rrr]

Note to self: if we build an ARN from a well-known name, we must not forget to manually add a dependency between the resources involved.

Probably, we should just link references between resources, which may then automatically translate into references between stacks.

[skinny85]

Use case: cross-account CodePipeline.

Below is an example of a hacky way of defining a cross-account CodePipeline using CDK code today. This follows the official blog post on the subject, but is a little simpler: in the blog post, there were a total of 4 accounts, 2 used for deployment (test and prod). I'm modelling a 2-stage Pipeline only, so I have 2 accounts: one with the CodeCommit repository in it, and another where the CodeBuild project and the Pipeline itself live.

import { AccountPrincipal, App, ArnPrincipal, PolicyStatement, Stack } from 'aws-cdk';
import * as cp from 'aws-cdk-codepipeline';
import * as cb from 'aws-cdk-codebuild';
import * as cc from 'aws-cdk-codecommit';
import * as s3 from 'aws-cdk-s3';
import * as iam from 'aws-cdk-iam';
import * as kms from 'aws-cdk-kms';
import { codecommit } from 'aws-cdk-resources';
import { HackyIdentity } from "./hacky-identity";

const app = new App(process.argv);

const pipelineAcc = '123456789012';

/*************** the CodeCommit Repository Stack *************************/

const repoAcc = '012345678901';
const repoAccStack = new Stack(app, 'OtherAccStack', {
    env: {
        account: repoAcc,
    }
});

const repoName = 'CrossAccountNewRepo';
new cc.Repository(repoAccStack, 'OtherAccRepo', {
    repositoryName: repoName,
});

const repoRole = new iam.Role(repoAccStack, 'OtherAccRole', {
    roleName: 'CrossAccountRole',
    assumedBy: new AccountPrincipal(pipelineAcc),
});
repoRole.addToPolicy(new PolicyStatement()
    .addAllResources()
    .addActions('s3:*', 'codecommit:*', 'kms:*'));
const repoRoleArn = `arn:aws:iam::${repoAcc}:role/CrossAccountRole`;

/*************** the Pipeline Stack *************************/

const pipelineStack = new Stack(app, 'CrossAccountMainStack');

const key = new kms.EncryptionKey(pipelineStack, 'CrossAccountKmsKey');
const artifactBucket = new s3.Bucket(pipelineStack, 'CrossAccountPipelineBucket', {
    bucketName: 'codepipeline-us-west-2-cross-account-code-pipeline-bucket',
    encryptionKey: key,
    encryption: s3.BucketEncryption.Kms,
});
artifactBucket.grantReadWrite(
    new HackyIdentity(new ArnPrincipal(repoRoleArn)));

const pipeline = new cp.Pipeline(pipelineStack, 'CrossAccountPipeline', {
    pipelineName: 'CrossAccountPipeline',
    artifactBucket: artifactBucket,
});
pipeline.addToRolePolicy(new PolicyStatement()
    .addResource(repoRoleArn)
    .addActions('sts:AssumeRole'));

const sourceStage = new cp.Stage(pipeline, 'Source');
const sourceAction = new cp.CodeCommitSource(sourceStage, 'Source', {
    artifactName: 'SourceOutput',
    repository: cc.RepositoryRef.import(pipelineStack, 'ImportedRepo', {
        repositoryName: new codecommit.RepositoryName(repoName)
    }),
    roleArn: repoRoleArn,
});

const buildStage = new cp.Stage(pipeline, 'Build');
const codeBuildProject = new cb.BuildProject(pipelineStack, 'CrossAccountBuildProject', {
    source: new cb.CodePipelineSource(),
    artifacts: new cb.CodePipelineBuildArtifacts(),
    environment: {
        image: 'aws/codebuild/java:openjdk-8',
    },
    encryptionKey: key,
});
new cp.CodeBuildAction(buildStage, 'Build', {
    source: sourceAction,
    project: codeBuildProject,
});

process.stdout.write(app.run());

(this required adding the field roleArn to the CodePipeline Actions, and HackyIdentity is a very simple class:

import * as iam from 'aws-cdk-iam';
import { PolicyPrincipal, PolicyStatement } from 'aws-cdk';

class HackyIdentity implements iam.IIdentityResource {
    public readonly principal: PolicyPrincipal;

    constructor(principal: PolicyPrincipal) {
        this.principal = principal;
    }

    attachManagedPolicy(_: any): void {
    }

    attachInlinePolicy(_: iam.Policy): void {
    }

    addToPolicy(_: PolicyStatement): void {
    }
}

)

As can bee seen, I had to:

• Give physical names to the IAM resources, like the Role in the CodeCommit Account that will be assumed by CodePipeline. This is in order to construct the ARNs needed to then be granted permissions (that Role needs access to the Pipeline's S3 bucket, and the CodePipeline will assume it, so needs sts:AssumeRole for it).
• I've actually given the CodeCommit Role * access to S3 - that's because I don't always know what's the ARN of the Bucket, as it can be managed by the L2 CodePipeline Construct (in this example, that's not the case, but in general, that's a problem - that means there's a circular dependency between resources in the 2 accounts). Same remark about the KMS key (that's actually required to make a cross-account CodePipeline work, otherwise the artifacts are encrypted with the default account key, and later stages in the Pipeline won't have access to the source retrieved from CodeCommit).

The dream experience would be something like:

// in the CodeCommit account / stack

const repository = new cc.Repository(repoAccStack, 'OtherAccRepo', {
    repositoryName: 'SomeName',
    // ...
});

// in the CodePipeline account / stack
new cp.CodeCommitSource(sourceStage, 'Source', {
    artifactName: 'SourceOutput',
    repository: repository,
});
// At this moment, the L2 CodePipeline construct notices that the Repository
// passed as the argument when constructing the `CodeCommitSource` is not from the same account as the Pipeline.
// In this case, it add the appropriate Role in the CodeCommit stack,
// adds the permissions for the CodePipeline Role to assume it,
// and makes sure that Role has access to the Pipeline's S3 Bucket,
// as well as the CodeCommit repository itself,
// and then sets it as the Role used in this Action.

I'm not sure how realistic it is to get this experience, but I believe it's the best one we can provide for this case.

[rix0rrr]

Obligatory mention of https://github.com/awslabs/aws-cdk/issues/233, which defines a nice and general solution which also encompasses x-stack references.

[rix0rrr]

Yes, you're completely right. I have some cross-stack action going on right now and I'm having to do the same things. Just know that we're thinking about it & working on it! :)

As a general comment: in general we recommend defining a class for every Stack that you want to model. We don't always follow this recommendation ourselves, but only for small demoes and integ tests. Your code will automatically end up more organized with clearer interoperability points between the stacks if you divide them into classes.

[eladb]

@rix0rrr wrote:

Right now, if we're referencing objects between stacks, we can't use Ref and Fn::GetAtt anymore. Consumers need to be constantly cognisant of this, and drop down to physically naming their resources and constructing ARN strings to make these cross-references.

This breaks abstraction because you now need to know (when consuming a resource) whether it's a same-stack or a cross-stack resource.

I propose we build this concept into the framework (potentially at the L2 level): we'll introduce a new type of Token, called StackAwareToken, which will know what object it has been created for, and hence in what Stack that object lives. We'll also propagate the "current stack" while resolving Tokens.

Now, if the current stack != owning stack, the StackToken has the ability to

• Throw a helpful error, because this will never work at CloudFormation deployment time.
• Check whether the owning resource is physically named. If not, throw a helpful error. If it is, return the name or construct the ARN on the spot, whatever the case may be.

Doing this (as opposed to some Output/Export system, potentially involving Toolkit Context Providers) has the advantage that we can avoid ordering between stack deployments, which has two benefits:

• Easier on the user; as long as you eventually deploy everything, the order doesn't matter.
• We can do bidirectional references between stacks (which could never work if we needed to establish a total ordering between stack deploys), which is beneficial for things like IAM permissions.

There is the downside that potential changes in naming aren't automatically propagated/signaled to the user. For Exports, you would be PREVENTED from changing anything that a downstream stack depends on, such as a resource name. (For Outputs or SSM parameters there is no such constraint I think).

[rix0rrr]

The previous comment was written at a time where Stacks were front and center, but everything gets so much better if we automatically slice stacks.

When just build a more generic object graph (https://github.com/awslabs/aws-cdk/issues/233) and slice it down to stacks, we can do the work to transparently make this work at that point.

[eladb]

Agreed (just captured for posterity), however, even if we automatically slice stacks, there still needs to be a mechanism that resolves these cross-stack references, be them explicit or implicit

[kennu]

Use case: Route53 hostname records in a shared hosted zone

• Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)
• Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)
• Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

[aripalo]

Use case:

I just hit this limitation by doing:

const primaryApi = new ApiStack(app, 'ApiStackIreland', { env: { region: 'eu-west-1' } }); // which has public readonly apiUrl: string;

new EdgeStack(app, 'EdgeStack', {
  primaryApiUrl: primaryApi.apiUrl,
  env: { region: 'us-east-1' }
});

And received this error:

node_modules/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.js:65
            throw new Error('Can only reference cross stacks in the same region and account.');

Having "EdgeStack" in us-east-1 makes sense as it is CloudFront's "main region" and is a must as one has to define ACM certificates and Lambda@Edge there.

This currently is somewhat blocking.

Is there any workaround currently?

[rix0rrr]

There is not, unfortunately. You're going to have to thread the API URL across regions manually. That means:

• Deploy the ApiStackIreland first. Note the API URL (maybe make it an Output so that it's easy to get to).
• Deploy the EdgeStack and plug in the value of API URL from the first step.

There are a number of ways you could do this, and honestly I'm not sure which one to recommend.

I guess a pragmatic one would be to use this.node.getContext() and pass the value on the command-line when deploying the second stack (using the -c KEY=VALUE fflag), or put it into cdk.json under the context: {} section.

[rix0rrr]

You could do it automatically using a Lambda-backed Custom Resource in EdgeStack that would make a cross-region call to look up the API URL as well, but that's a little more work.

[briannab12]

@skinny85 Changes have been made since July. cp.CodeCommitSource no longer exists. Is there still a way to set the role arn for a PipelineSourceAction? I am accessing a repo in a different AWS account.

[skinny85]

Hey @briannab12 ,

currently, there is no way to do that. If you're using TypeScript, the following hack should work:

(sourceAction as any)['role'] = yourRole;

We're working on this experience as we speak, and we hope to make it awesome - see the #1924 PR for details.

Thanks, Adam

[briannab12]

(sourceAction as any)['role'] = yourRole; This worked, thank you.

Here are the specifics of my circumstances:

        const RepoBranch = new cdk.Parameter(this, 'RepoBranch', {
        type: 'String',
        description: 'The branch name for the CodeCommit API repository.'
    });

        const sourceStage = pipeline.addStage('Source');

    const repo = cc.Repository.import(this, envNameParam+'RepoRef', {
        repositoryName: sourceRepoNameParam
    });

    const sourceActionApiRepo = new cc.PipelineSourceAction(this, 'SourceActionApiRepo', {
        stage: sourceStage,
        repository: repo,
        branch: RepoBranch.ref
    });

        const crossAccountRole = iam.Role.import(this, envNameParam+'CrossAccountRoleImport',{
        roleArn: crossAccountRoleArnCodeCommitParam
    });

    //Hacky was of setting source action ARN. Setting it is not currently supported by cdk. 
    (sourceActionApiRepo as any)['role'] = crossAccountRole;

[windlessuser]

Use case: Route53 hostname records in a shared hosted zone

• Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)
• Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)
• Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

[saltman424]

Use case: Route53 hostname records in a shared hosted zone

• Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)
• Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)
• Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

@eladb Since this issue is now closed, do we have a solution for the above use case?

I am trying something similar: having a public hosted zone in one account delegate to a public hosted zone in another account.

[cshenrik]

@eladb Can you offer some insights into why this issue was closed? A good generic solution would have taken CDK to the next level (IMHO).

[OperationalFallacy]

It looks like the project is trying to get around Cloudformation's native limitation: it can't read outputs from another account/region. Having this functionality in Cloudformation would be a perfect world.

Meanwhile, cdk can deal with the env variable, contexts, config files - why not make stacks outputs automatically available for subsequent stages in a known artifact?

[skinny85]

@OperationalFallacy I agree, and we have a feature request already for that functionality: https://github.com/aws/aws-cdk/issues/8566 .

Thanks, Adam

[alehatsman]

Use case: Route53 hostname records in a shared hosted zone

• Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)
• Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)
• Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

@eladb Since this issue is now closed, do we have a solution for the above use case?

I am trying something similar: having a public hosted zone in one account delegate to a public hosted zone in another account.

I am having a similar problem currently, has anyone tried to solve it using custom resources and cdk? A custom resource that calls lambda which is able to access the parent host zone and add records from stack A and B?

[OperationalFallacy]

Cdk has now a construct to add records with lambda, and it works well with pipelines

https://github.com/OperationalFallacy/CdkRouter53/pull/1/files#diff-28e171489876432b7aa1c09d7d1a446d777028d46e763081600b19ead74c9111R25

[alehatsman]

Cdk has now a construct to add records with lambda, and it works well with pipelines

https://github.com/OperationalFallacy/CdkRouter53/pull/1/files#diff-28e171489876432b7aa1c09d7d1a446d777028d46e763081600b19ead74c9111R25

Thank you for your reply! It works well!