app-staging-synthesizer: get S3 access denied error on stack deployment

[tmokmss]

Describe the bug

Also see the comment: https://github.com/aws/aws-cdk/issues/27434#issuecomment-1754314951

Hi, in the article about AppStagingSynthesizer recently published, we can see the following description about IAM roles to deploy stacks.

Reduced Bootstrapping Complexity: As the only shared resources required are global Roles, the company now only needs to bootstrap every account in one Region instead of bootstrapping every Region. This simplifies the bootstrapping process, making it easier to manage with CloudFormation StackSets. https://aws.amazon.com/blogs/devops/enhancing-resource-isolation-in-aws-cdk-with-the-app-staging-synthesizer/

So I expect that a cdk app using AppStagingSynthesizer can be deployed to any region if we've done cdk bootstrapping in one region. However, currently it seems that we need to bootstrap in each region we're deploying to.

I'd like to make sure if it is expected or not. Maybe a documentation error? Thanks.

Expected Behavior

A cdk app using AppStagingSynthesizer can be deployed to any region if we've done cdk bootstrapping in one region.

Current Behavior

We need to bootstrap in each region we're deploying to.

Reproduction Steps

Bootstrap a region e.g. us-east-1.

Then, when I try to deploy a cdk app using AppStagingSynthesizer to another region without bootstrapping, I get the following error:

 ❌ Deployment failed: Error [ValidationError]: Role arn:aws:iam::123456789012:role/cdk-hnb659fds-cfn-exec-role-123456789012-eu-west-2 is invalid or cannot be assumed

The role name contains region eu-west-2, so it seems we have to bootstrap eu-west-2 first.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.98.0

Framework Version

2.98.0

Node.js Version

v18.13.0

OS

macOS

Language

Typescript

Language Version

No response

Other information

No response

[indrora]

Can you please provide how you're setting the region for deployment? A minimal working example would be best.

[tmokmss]

Hi @indrora, I noticed that I forget to set deploymentIdentities prop, and if I set this, the error is gone. However, I'm getting another error about permission:

 ❌ Deployment failed: Error [ValidationError]: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

The reproduction code is here:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { AppStagingSynthesizer, DeploymentIdentities } from '@aws-cdk/app-staging-synthesizer-alpha';

const app = new cdk.App({
  defaultStackSynthesizer: AppStagingSynthesizer.defaultResources({
    appId: 'my-app-id',
    deploymentIdentities: DeploymentIdentities.defaultBootstrapRoles({ bootstrapRegion: 'us-east-1' }),
  }),
});

new cdk.Stack(app, 'StagingSynthesizerStack');

And run the following command:

AWS_REGION=us-east-1 npx cdk bootstrap
AWS_REGION=us-west-2 npx cdk deploy StagingSynthesizerStack

Can you reproduce this? Not quite sure if I'm using this feature correctly though.

[sakurai-ryo]

In v2.147.3, the same code seems to cause an error about the original cfn-exec-role error instead of the S3 validation error. ❌ Deployment failed: Error [ValidationError]: Role arn:aws:iam::111111111111:role/cdk-hnb659fds-cfn-exec-role-111111111111-ap-southeast-2 is invalid or cannot be assumed.

I am not sure about the S3 validation error, but as for the error about cfn-exec-role error, the BootstraplessSynthesizer used to synthesize StagingStack is probably the problem. https://github.com/aws/aws-cdk/blob/358ceadd3352b4c692438b9d9061354556fc5bac/packages/%40aws-cdk/app-staging-synthesizer-alpha/lib/default-staging-stack.ts#L262

Since the arn of the CFn Execution Role is not passed as an argument when initializing the BootstraplessSynthesizer, the Synthesizer will use the CFn Execution Role of the region where the StagingStack is deployed. However, even though the IAM Role does not exist in that region, it is used as a service role in the CFn deployment process, resulting in an error. https://github.com/aws/aws-cdk/blob/358ceadd3352b4c692438b9d9061354556fc5bac/packages/aws-cdk-lib/core/lib/stack-synthesizers/default-synthesizer.ts#L366 https://github.com/aws/aws-cdk/blob/358ceadd3352b4c692438b9d9061354556fc5bac/packages/aws-cdk-lib/core/lib/stack-synthesizers/default-synthesizer.ts#L240

So, I think it is necessary to correctly initialize the BootstraplessSynthesizer used to deploy the StagingStack.

Maybe relates https://github.com/aws/aws-cdk/issues/28195.