(CDK Pipelines) Deploy stage, vpc.prepare failed due to S3 Access Denied

[uxth]

I have a working code pipeline using CDK python. Recently someone else in the team did some change just move some of the code to different folder. then the code pipeline stopped working, failed at the first step in deploying stage, vpc.Prepare. the reason was S3 Access Denied.

My first thinking is to revert the change and try again, and I did it. it did not help.

Then I removed all the stacks in cloud formation, started over. it did not help.

I found it was due to S3 Access Denied. so I change the S3 bucket pipelinestack-mappipelineartifactsbucket822fb6ba-1cjz3xh6o8l9j to public access, and allow public read to the artifacts within this bucket. Then the Prepare succeeded.

To verify, I turned off the public access of the s3, then it stopped working, stuck at the Prepare step. Turning it on again, then it is in progress again.

in the cloud formation, I can see that the deployment role policy is administrator access, which should not have this kind access issue, correct? But why after granting public access, it worked?

Keep the bucket publicly accessible is not ideal.

Please help to solve the issue.

Thanks

Reproduction Steps

What did you expect to happen?

Prepare step should not be stuck

What actually happened?

Prepare step got stuck when code pipeline is deploy stage

Environment

• **CDK CLI Version : 1.111.0 (build 556ca93)

• Framework Version:

• Node.js Version:v16.2.0

• **OS : big sure

• Language (Version): Python 3.8.2

Other

This is the error message:

Error message
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: K4T79835ANWTDQY3; S3 Extended Request ID: Hw8uwS1+GrSvsbkem0Xh+XdEDzwdmr2u4yv7Szun9SSbgrtmbw8uwNLq+z1rinQFdqf0xpUiLic=; Proxy: null)

Could you track this request ID to figure out what is happening under the hood?

---

This is :bug: Bug Report

[uxth]

Since this prepare step is totally generated by cdk / code pipeline, I have no idea what it did. I don't have any tools to debug this issue.

[skinny85]

Hey @uxth,

did you use the CDK Pipelines module to create this CodePipeline?

Thanks, Adam

[uxth]

Hello @skinny85 ,

Yes, that is the one I am using.

Best

[skinny85]

Thanks. Just wanted to make sure this gets routed to the correct folks.

[uxth]

thank you

[njlynch]

Hi @uxth ,

I believe you're encountering this issue due to an out-of-date bootstrap stack. We've recently made some changes (#15192, #15266, #15348) to the bootstrap stack to limit permissions to what's absolutely necessary; the unfortunate net result of this though is that if your app and bootstrap stack are out of sync (e.g., updating the CDK, but not re-bootstrapping), you may receive these errors.

What bootstrap stack version are you currently using? You can find this by looking at the "BootstrapVersion" output of the "CDKToolkit" stack in the account/region that's showing this error. If it's not the latest (7), then re-bootstrapping (env CDK_NEW_BOOTSTRAP=1 cdk bootstrap...) should solve the issue.

If you're already up to version 7 in each environment where you're deploying, another potential cause of you getting these errors is if the bootstrap stack for that environment (account + region) was deleted (and recreated) after the initial pipeline creation. I've actually just written up some work-around steps for how to fix this here: https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/pipelines#action-execution-denied .

Please let me know if that helps!

[devendhirababu]

Hi @njlynch,

Currently I am also facing the same issue "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: K4T79835ANWTDQY3; S3 Extended Request ID: Hw8uwS1+GrSvsbkem0Xh+XdEDzwdmr2u4yv7Szun9SSbgrtmbw8uwNLq+z1rinQFdqf0xpUiLic=; Proxy: null)" post newer version of CDK-boostrap-version7.

Earlier my environment was bootstrapped with cdk-bootstrap-version4, and there were many resources created through CDK-Pipeline, but when I try to deploy one new stack got an error message as "This application requires CDK-bootstrap version 6 and found 4" so i deleted the bootstrap-version4 Cloudformation stack and created new bootstrap-stack version7 the bootstrap resource were created successfully with default qualifier name (hnb659fds). Then made changes in my stack code(typescript) and pushed into repo the pipeline source and build, updatepipeline was succeeded, but the application stage was failed and the error message was same as mentioned above.

Latter created another bootstrap stack with custom qualifier name and added the following entry "@aws-cdk/core:bootstrapQualifier: "mycustomqualifername" in cdk.json file then pushed the code to repo. Again it was failed at the application deployment stage.

I have raised a case with AWS support and worked with them regarding the same, but they told that they will raise a ticket with their internal team and get back with an update.

Please help me to fix this issue.

Environment: CDK CLI Version: 1.106.1 Typescript: 3.9.9 NodeJs: v15.8.0

[peterwoodworth]

@devendhirababu is everything under the same account? And are you creating the Bucket in your stack?

Unfortunately we aren't able to track the request id to figure out what's happening under the hood. Hopefully support has been able to help you with this issue

[czubocha]

Hi, this is duplicate of https://github.com/aws/aws-cdk/issues/15307 which is unfortunately closed but not solved (the fix seems unsuccessful).

[peterwoodworth]

Thanks for bringing this up @czubocha, it seems other people are still having issues with that bug after the fix. I've reopened the issue #15307, please take your concerns to that thread 😄

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rix0rrr]

There are multiple causes for "S3 Access Denied".

I saw someone mention in the other thread they deleted and recreated their bootstrap stack.

Doing that will permanently break Pipelines.

See the pipelines README on how to unbreak yourself in that situation

[peterwoodworth]

Reopening this as customers are still having issues with S3 Access Denied errors with the new bootstrap stack

[devendhirababu]

Hi @peterwoodworth,

Thanks for re-opening this issue, I would like to inform you that the issue was resolved yesterday with the help of an AWS support team(Hammad R) we had multiple calls, after spending one week of time.

He analyzed the error message deeply and found that issues with an Bootstrap bucket policy. We added the permission in the bootstrap bucket policy allowing cfn-exe-role, deployment-role, pipeline-role (Principle) and "Action":s3* for the resources (Artifact bucket).

He saved me from the embarrassing situation, I really thankful to Hammad R (AWS Support) and aws-cdk-GitHub Team.

Regards, Deva

[peterwoodworth]

Thanks for sharing @devendhirababu, much appreciated.

Editing the bootstrap policies is the suggested way to deal with these issues if you're on the latest bootstrap version. If people are still encountering this issue let me know, I'll be closing it again for now.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[jy-metservice]

I was affected by this issue. The S3 bucket that codepipeline was using had no ACL on it. After bootstrapping the cdk toolkit stack to the latest template version 13 I had to put an explicit ALLOW policy on the S3 bucket for the cfn-exec-role and deploy-role that the toolkit strap creates. This overcame the S3 permission denied error on CFN prepare stage of codepipeline.

[leantorres73]

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

[jy-metservice]

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

@leantorres73 I used the following ACL. (Note: I have altered our account ID and bucket name). I had to give access to both the cfn-exec-role + the cfn-deploy role

{
    "Version": "2012-10-17",
    "Id": "AccessControl",
    "Statement": [
        {
            "Sid": "CDKToolkitAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-cfn-exec-role-123456789-us-east-1",
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-deploy-role-123456789-us-east-1"
                ]
            },
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:PutObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::our-bucket-name",
                "arn:aws:s3:::our-bucket-name/*"
            ]
        }
    ]
}

[juadavard]

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

@leantorres73 I used the following ACL. (Note: I have altered our account ID and bucket name). I had to give access to both the cfn-exec-role + the cfn-deploy role

{
    "Version": "2012-10-17",
    "Id": "AccessControl",
    "Statement": [
        {
            "Sid": "CDKToolkitAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-cfn-exec-role-123456789-us-east-1",
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-deploy-role-123456789-us-east-1"
                ]
            },
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:PutObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::our-bucket-name",
                "arn:aws:s3:::our-bucket-name/*"
            ]
        }
    ]
}

This worked for me

[jy-metservice]

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

@leantorres73 I used the following ACL. (Note: I have altered our account ID and bucket name). I had to give access to both the cfn-exec-role + the cfn-deploy role

{
    "Version": "2012-10-17",
    "Id": "AccessControl",
    "Statement": [
        {
            "Sid": "CDKToolkitAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-cfn-exec-role-123456789-us-east-1",
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-deploy-role-123456789-us-east-1"
                ]
            },
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:PutObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::our-bucket-name",
                "arn:aws:s3:::our-bucket-name/*"
            ]
        }
    ]
}

This worked for me

It's been a while since I worked on this! At a superficial level, there are a few differences between the IAM policy I wrote, and the one you've used however. I'd look at those differences first of all

[rdjogo]

No intention to nitpick on this, just trying to be precise, to check if I'm missing something.

I'm facing the same issue, so I'd like to know if the ACLs you refer to are indeed Bucket ACLs or is it Bucket Policy? Since my bucket has Object Ownership enforced, ACLs are disabled, and I added the above specified permissions within the bucket policy, but to no avail.

When I turn off the object ownership enforcement, ACLs are enabled, but the interface doesn't allow to specify them in this format, but just to enter grantee and check the boxes for read write on bucket/objects...

Thanks!

[jy-metservice]

No intention to nitpick on this, just trying to be precise, to check if I'm missing something.

I'm facing the same issue, so I'd like to know if the ACLs you refer to are indeed Bucket ACLs or is it Bucket Policy? Since my bucket has Object Ownership enforced, no ACLs are disabled, and I added the above specified permissions within the bucket policy, but to know avail.

When I turn off the object ownership enforcement, ACLs are enabled, but the interface doesn't allow to specify them in this format, but just to enter grantee and check the boxes for read write on bucket/objects...

Thanks!

The permissions I wrote above are applied to the S3 bucket as a bucket policy.

Public Access (Block all public access: ON: Access: Bucket and objects not public)

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

The object ownership config for that bucket is: Object Ownership Object writer

The bucket ACL is this (via aws s3api get-bucket-acl --bucket obscured-bucket-name):

{
    "Owner": {
        "DisplayName": "amazon.OBSCURED",
        "ID": "OBSCURED"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "amazon.OBSCURED",
                "ID": "OBSCURED",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]

Hope that helps

[rdjogo]

Thanks, it definitely helped! What didn't help is that I was at first working with the wrong bucket.

Just a note, since I was working with cross account pipeline, I had to add the following permissions in the key policy of the kms key used for assets S3 bucket in the account where the pipeline resides, for the deploy role in the "destination" account:

"Action": [ "kms:Decrypt", "kms:DescribeKey", "kms:GenerateDataKey" ],

[raymondsquared]

yes, attaching s3:Get* policy to the cdk-xxx-deploy-role-xxx solved the issue for me