(aws-chatbot): SlackChannelConfiguration deployed to us-east-1 tries to chatbot:CreateSlackChannelConfiguration in us-east-2

[blimmer]

Describe the bug

I use a custom bootstrapping script to reduce what the CfnExec role can do. Here's the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAccessToServicesViaCloudFormation",
      "Effect": "Allow",
      "Action": ["cur:*", "logs:*", "s3:*", "lambda:*", "sns:*", "ce:*", "chatbot:*"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    },
    {
      "Sid": "AllowAccessToCdkSsmParameters",
      "Effect": "Allow",
      "Action": ["ssm:GetParameters"],
      "Resource": ["arn:aws:ssm:*:*:parameter/cdk-bootstrap/*"]
    },
    {
      "Sid": "AllowAccessToRolesExceptCdk",
      "Effect": "Allow",
      "Action": ["iam:*Role*", "iam:GetPolicy", "iam:CreatePolicy", "iam:DeletePolicy", "iam:*PolicyVersion*"],
      "NotResource": ["arn:aws:iam::*:role/cdk-*", "arn:aws:iam::*:policy/cdkCFExecutionPolicy"]
    }
  ]
}

As you can see chatbot:* is allowed, but only in our bootstrapped regions (us-east-1 and us-west-2).

However, when I try to create a SlackChannelConfiguration, it tries to write a resource in us-east-2. This breaks because the role does not allow access to us-east-2 (see error in section below).

Expected Behavior

I expected the resources to be created in the target region, us-east-1.

Current Behavior

I receive this error:

 ❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named CdkTestStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "User: arn:aws:sts::MYACCOUNT:assumed-role/cdk-hnb659fds-cfn-exec-role-MYACCOUNT-us-east-1/AWSCloudFormation is not authorized to perform: chatbot:CreateSlackChannelConfiguration on resource: arn:aws:chatbot:us-east-2:MYACCOUNT:chat-configuration/slack-channel/Test (Service: AWSChatbot; Status Code: 403; Error Code: AccessDeniedException; Request ID: 1d41775b-e8da-4f69-926a-5138255f1251; Proxy: null)" (RequestToken: e5f6350e-3982-6a40-97a1-ba8f37b320c7, HandlerErrorCode: GeneralServiceException)
    at deployStacks (/private/tmp/cdk-test/node_modules/aws-cdk/lib/index.js:374:129094)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CdkToolkit.deploy (/private/tmp/cdk-test/node_modules/aws-cdk/lib/index.js:374:147518)
    at async exec4 (/private/tmp/cdk-test/node_modules/aws-cdk/lib/index.js:429:51795)

Reproduction Steps

To reproduce this error, you'll need the use a custom IAM policy for the CfnExec role. The one I'm using is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAccessToServicesViaCloudFormation",
      "Effect": "Allow",
      "Action": ["cur:*", "logs:*", "s3:*", "lambda:*", "sns:*", "ce:*", "chatbot:*"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    },
    {
      "Sid": "AllowAccessToCdkSsmParameters",
      "Effect": "Allow",
      "Action": ["ssm:GetParameters"],
      "Resource": ["arn:aws:ssm:*:*:parameter/cdk-bootstrap/*"]
    },
    {
      "Sid": "AllowAccessToRolesExceptCdk",
      "Effect": "Allow",
      "Action": ["iam:*Role*", "iam:GetPolicy", "iam:CreatePolicy", "iam:DeletePolicy", "iam:*PolicyVersion*"],
      "NotResource": ["arn:aws:iam::*:role/cdk-*", "arn:aws:iam::*:policy/cdkCFExecutionPolicy"]
    }
  ]
}

The important part is that access to us-east-2 is not configured in the AllowAccessToServicesViaCloudFormation statement.

Then, try to deploy this stack:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { SlackChannelConfiguration } from 'aws-cdk-lib/aws-chatbot';
import { Topic } from 'aws-cdk-lib/aws-sns';
import { Construct } from 'constructs';

class CdkTestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const snsTopic = new Topic(this, "Topic" );

    new SlackChannelConfiguration(this, "ChatbotChannel", {
      slackChannelConfigurationName: "Test",
      slackWorkspaceId: "<redacted>",
      slackChannelId: "<redacted>",
      notificationTopics: [snsTopic],
    });
  }
}

const app = new cdk.App();
new CdkTestStack(app, 'CdkTestStack', {
  env: { account: 'MYACCOUNT', region: 'us-east-1' },
});

When you try to deploy, you'll encounter the error listed above.

Possible Solution

Looking at the generated template, it doesn't seem like CDK is explicitly doing anything wrong. However, I wasn't sure how to report this directly to CloudFormation (since I don't have a support plan on this account).

I figured, worst case, the CDK team could direct me to the proper place.

Additional Information/Context

Here's the CDK template, it doesn't specify us-east-2 anywhere...

{
 "Resources": {
  "TopicBFC7AF6E": {
   "Type": "AWS::SNS::Topic",
   "Metadata": {
    "aws:cdk:path": "CdkTestStack/Topic/Resource"
   }
  },
  "ChatbotChannelConfigurationRoleFD7AD2E3": {
   "Type": "AWS::IAM::Role",
   "Properties": {
    "AssumeRolePolicyDocument": {
     "Statement": [
      {
       "Action": "sts:AssumeRole",
       "Effect": "Allow",
       "Principal": {
        "Service": "chatbot.amazonaws.com"
       }
      }
     ],
     "Version": "2012-10-17"
    }
   },
   "Metadata": {
    "aws:cdk:path": "CdkTestStack/ChatbotChannel/ConfigurationRole/Resource"
   }
  },
  "ChatbotChannel0C037C2E": {
   "Type": "AWS::Chatbot::SlackChannelConfiguration",
   "Properties": {
    "ConfigurationName": "Test",
    "IamRoleArn": {
     "Fn::GetAtt": [
      "ChatbotChannelConfigurationRoleFD7AD2E3",
      "Arn"
     ]
    },
    "SlackChannelId": "<redacted>",
    "SlackWorkspaceId": "<redacted>",
    "SnsTopicArns": [
     {
      "Ref": "TopicBFC7AF6E"
     }
    ]
   },
   "Metadata": {
    "aws:cdk:path": "CdkTestStack/ChatbotChannel/Resource"
   }
  },
  "CDKMetadata": {
   "Type": "AWS::CDK::Metadata",
   "Properties": {
    "Analytics": "v2:deflate64:H4sIAAAAAAAA/3WKMRKCMBBFz0KfrIbC0ZoboL0TQpCFsOuQoEUmdzcRW6v35v9Xw+kCx0q/vTT9LB12EK9Bm1nk6R49eYg3fqIRzUBfScKMOnQccuhy2IyayLqGacDHtuqATCX+eyaBeoHYsrOlK0xJtNbztpp9+nkSxL2FyR9e6gyqBlVNHlGuGwVcLLQ7PxXGC83EAAAA"
   },
   "Metadata": {
    "aws:cdk:path": "CdkTestStack/CDKMetadata/Default"
   }
  }
 },
 "Parameters": {
  "BootstrapVersion": {
   "Type": "AWS::SSM::Parameter::Value<String>",
   "Default": "/cdk-bootstrap/hnb659fds/version",
   "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
  }
 },
 "Rules": {
  "CheckBootstrapVersion": {
   "Assertions": [
    {
     "Assert": {
      "Fn::Not": [
       {
        "Fn::Contains": [
         [
          "1",
          "2",
          "3",
          "4",
          "5"
         ],
         {
          "Ref": "BootstrapVersion"
         }
        ]
       }
      ]
     },
     "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
    }
   ]
  }
 }
}

CDK CLI Version

2.69.0 (build 60a5b2a)

Framework Version

No response

Node.js Version

v18.12.1

OS

MacOS

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

Yes I can reproduce this.

Looks like the CFN execution role is trying to run chatbot:CreateSlackChannelConfiguration on resource chat-configuration on us-east-2 but I can't find any public document about it.

I will try reach out internally but I can't guarantee any response here. If this is an urgent case I will recommend subscribe a support plan and submit a support case for this.

[blimmer]

I can work around this for now, so it's not urgent. Thanks for bringing this up internally.

[pahud]

HI @blimmer

I got the response from the relevant team.

According to this:

AWS Chatbot is a global service that requires access to all AWS Regions. If there is a policy in place that prevents access to services in certain Regions, you must change the policy to allow global AWS Chatbot access https://docs.aws.amazon.com/chatbot/latest/adminguide/getting-started.html

You will need to grant your CFN execution role with the affected permission on us-east-2 in this case.

[blimmer]

Huh, interesting. Usually, global services mean "it's really in us-east-1" 😄 (e.g., CloudFront, ACM, etc.).

The problem here is that, as Control Tower users, we have explicit region deny controls set up for us-east-2.

I think that means we just can't use ChatBot, which is kinda a bummer. But, it seems like this is a limitation of ChatBot, not CDK.

@pahud , is there a good place (other than the support plan, which we don't have) to open an issue about this with the ChatBot team? For instance, a public GitHub repo, etc.?

In any case, it's probably safe to close this issue, as there's nothing CDK is doing wrong!

EDIT: Also thank you for reaching out to the team for me, I really appreciate your time 😄 💯 🏆

[pahud]

Hey @blimmer no worry. I will share this issue with the ChatBot team and make sure they receive your feedback. I doubt they have public issue report on GitHub but I'll let you know if any.

[pahud]

Hi @blimmer

This might be helpful for regarding the control tower region denylist: https://aws.amazon.com/about-aws/whats-new/2022/07/aws-control-tower-region-deny-guardrail-expands-aws-chatbot-s3-storage-lens-s3-multi-region-access-points-apis/. There are some exemptions for global services which can be put in place by updating landing zone to version 3.0.

Hope it helps!

I am closing this now. Feel free to let us know if you have any further issues around CDK.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[blimmer]

Thank you for all of these details and working with the chatbot team on my behalf!