(custom-resource): Creation Fails Due to Lack of Permissions on Some Services

Conklin-Spencer-bah commented 1 week ago

Describe the bug

When using the custom-resource (AwsCustomResource) module there are arbitrary failures on AWS services. This seems to be because the Custom Resource that is created has a policy associated to the lambda function and immediately executes it. The execution is not timed to wait until permissions are attached or validated. Even when using a CloudFormation Wait Condition the resources are created simultaneously.

Regression Issue

[ ] Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

When using AwsCustomResource and defining the policy the custom resource should wait until policy creation before execution.

Current Behavior

When calling AwsCustomResource and defining the policy for the resource the policy is immediately created along with execution of the lambda function. If the policy creation is not completed or delayed due to some sort of inconsistent latency within AWS the custom resource lambda fails with permission denied.

Reproduction Steps

Create a CDK deployment that creates a Custom Resource using the below policy and Custom Resource.

Policy

    const putAccountContactInformationPolicy = cr.AwsCustomResourcePolicy.fromStatements([
      new iam.PolicyStatement({
        actions: [
          'account:GetAccountInformation',
          'account:PutContactInformation',
        ],
        resources: [`arn:aws:account::${Aws.ACCOUNT_ID}:account`],
      }),
    ]);

Custom Resource

    new cr.AwsCustomResource(this, 'PutContactInformation', {
      onUpdate: {
        service: '@aws-sdk/client-account',
        action: 'PutContactInformationCommand',
        parameters: {
          ContactInformation: {
            FullName: 'redacted',
            AddressLine1: 'redacted',
            City: 'redacted',
            StateOrRegion: 'redacted',
            PostalCode: 'redacted',
            CountryCode: 'redacted,
            PhoneNumber: 'redacted',
            CompanyName: 'redacted,
            WebsiteUrl: 'redacted,
          },
          AccountId: Aws.ACCOUNT_ID,
        },
        physicalResourceId: cr.PhysicalResourceId.of(Date.now().toString()),
      },
      policy: putAccountContactInformationPolicy,
    });

Possible Solution

Decouple creation of the policy for the custom resource from the same call. Allow creation of a depends on for the policy for the custom resource.

Additional Information/Context

See related issue that was closed: 21332

CDK CLI Version

2.147.3 (build 32f0fdb)

Framework Version

No response

Node.js Version

v18.20.4

OS

MacOSX

Language

TypeScript

Language Version

No response

Other information

No response

pahud commented 1 week ago

Yes, custom resource has a lambda function as its provider which has an IAM role and the role has the default policy.

The default policy depends on the role
The Lambda function depends on the role but it won't depend on the default policy So there would be chance when the custom resource is being created, executing the lambda function, the default policy is not yet fully created, hence permission deny.

Possible Solutions

when we CDK create the custom resource as well as the lambda role for the provider, CDK should ensure the dependency that the lambda function depends on the default policy of the role.
alternatively, the custom resource itself should depend on the default policy of the role

Now, let's look into your provided code. When I cdk diff using your code I got this:

Resources
[+] Custom::AWS PutContactInformation/Resource PutContactInformation78FED2C5 
[+] AWS::IAM::Policy PutContactInformation/CustomResourcePolicy PutContactInformationCustomResourcePolicyB3C7E805 
[+] AWS::IAM::Role AWS679f53fac002430cb0da5b7982bd2287/ServiceRole AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2 
[+] AWS::Lambda::Function AWS679f53fac002430cb0da5b7982bd2287 AWS679f53fac002430cb0da5b7982bd22872D164C4C

Obviously, the iam policy with logicalId PutContactInformationCustomResourcePolicyB3C7E805 is the key which should be depended by either the custom resource itself or the lambda function. Now if I check the synthesized template, I noticed this:

"Resources": {
  "PutContactInformation78FED2C5": {
   "Type": "Custom::AWS",
   "Properties": {
...
   },
   "DependsOn": [
    "PutContactInformationCustomResourcePolicyB3C7E805"
   ],

which means custom resource is actually depending on PutContactInformationCustomResourcePolicyB3C7E805 so it technically should be fine.

CR depends on the iam policy
CR depends on the Lambda function
Lambda function depends on the role
iam policy depends on the role

With that being said, when CR is being created, the role and policy should all set and ready. So I don't think this would cause the issue.

Are you able to try to reproduce that and show me the full error message if you see that happening again?

Conklin-Spencer-bah commented 6 days ago

Thank you for the quick response. I can happily reproduce this. As you outlined this is exactly how my code and synthesized template looks. So technically it looks correct but still fails. I'm curious to know if that the execution and the policy attachment are happening too quickly which results in this failure? I do this exact same type of operation for other API endpoints in AWS and do not get the error.

One thing to note is that I am doing this via a stackset. I don't know why that would make a difference. As it is simply just running the CloudFormation template. The other succeeding API calls are also done as a stackset.

Error message requested: Received response status [FAILED] from custom resource. Message returned: User: arn:aws:sts::<redacted>:assumed-role/StackSet-AccountContactIn-<redacted>/StackSet-AccountContactIn-<redacted> is not authorized to perform: account:PutContactInformation (RequestId: <redacted>)

savagete2860 commented 6 days ago

Remove AccountId: Aws.ACCOUNT_ID, from the SDK arguments. I men's warehouse guarantee you're gonna like the way it deploys

Conklin-Spencer-bah commented 3 days ago

Thanks Tim! That worked 10/10. I have zero idea why that would be the case. Anyone else facing similar issues consider removing that flag.

aws / aws-cdk