Creating more than 10 environments fails with Resource limit exceeded for LogResourcePolicy

[proof-nicholas]

Hi,

We use copilot to create preview environments for developer feature branches. Everything has been going well until we tried to create the 11th environment for our app and ran into an error where the CloudFormation stack failed with the error:

Resource handler returned message: "Resource limit exceeded. (Service: CloudWatchLogs, Status Code: 400, Request ID: ed664c86-8c50-4121-8ce9-07d36e240f8d)" (RequestToken: 4994a1fd-835f-1e8e-e211-861e199d9690, HandlerErrorCode: ServiceLimitExceeded)

It looks like we are hitting AWS hard quota limits for CloudWatch Logs Resource Policies:

Up to 10 CloudWatch Logs resource policies per Region per account. This quota can't be changed.

It appears creating an environment includes this snippet in the CloudFormation template:

LogResourcePolicy:
    Metadata:
      'aws:copilot:description': 'A resource policy to allow AWS services to create log streams for your workloads.'
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: !Sub '${AppName}-${EnvironmentName}-LogResourcePolicy'
      PolicyDocument:
        Fn::Sub: |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Sid": "StateMachineToCloudWatchLogs",
                "Effect": "Allow",
                "Principal": {
                  "Service": ["delivery.logs.amazonaws.com"]
                },
                "Action": [
                  "logs:CreateLogStream",
                  "logs:PutLogEvents"
                ],
                "Resource": [
                  "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/copilot/${AppName}-${EnvironmentName}-*:log-stream:*"
                ],
                "Condition": {
                  "StringEquals": {
                    "aws:SourceAccount": "${AWS::AccountId}"
                  },
                  "ArnLike": {
                    "aws:SourceArn": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*"
                  }
                }
              }
            ]
          }

Does this policy really need to be per environment? Could we not have one per application (although I guess this would still be a problem for those in the community who have more than 10 applications)?

What options do I have to workaround this?

copilot version: v1.33.2

[proof-nicholas]

I was able to workaround this by adding an override to each environment to remove the LogResourcePolicy resource:

- op: remove
  path: /Resources/LogResourcePolicy

Then use the following AWS CLI command to create a more generic resource policy that is shared by all environments within the app (make sure to edit this command first to replace the placeholders w/ your account details [e.g. <account-id>]):

aws logs put-resource-policy --policy-name "copilot-app-LogResourcePolicy" \
--policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"StateMachineToCloudWatchLogs\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"delivery.logs.amazonaws.com\"},\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Resource\":\"arn:aws:logs:<region>:<account-id>:log-group:/copilot/*:log-stream:*\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"<account-id>\"},\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:logs:<region>:<account-id>:*\"}}}]}"

[libert-xyz]

We are facing a similar issue. Any other workaround that doesn't involve adding an override?

[chrisschaub]

Same issue here; shouldn't copilot create 1 shared policy and re-use it within a profile?

[libert-xyz]

@proof-nicholas how do you specify the manually created resource-policy in copilot?

[zeeshanpolaris]

I just ran in to this issue. I have the same question on how to specify the manually created resource policy in copilot?

[proof-nicholas]

@proof-nicholas how do you specify the manually created resource-policy in copilot?

For us, we only have one application but with multiple environments because we use it to deploy preview environments for feature branches. Consequently, I just ran the command manually once using the aws cli, i.e. not using copilot.

[zeeshanpolaris]

/Resources/LogResourcePolicy

Thank you. This worked for me.

[guvkon]

I was able to workaround this by adding an override to each environment to remove the LogResourcePolicy resource:

- op: remove
  path: /Resources/LogResourcePolicy

Then use the following AWS CLI command to create a more generic resource policy that is shared by all environments within the app (make sure to edit this command first to replace the placeholders w/ your account details [e.g. <account-id>]):

aws logs put-resource-policy --policy-name "copilot-app-LogResourcePolicy" \
--policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"StateMachineToCloudWatchLogs\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"delivery.logs.amazonaws.com\"},\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Resource\":\"arn:aws:logs:<region>:<account-id>:log-group:/copilot/*:log-stream:*\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"<account-id>\"},\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:logs:<region>:<account-id>:*\"}}}]}"

Thank you very much. This worked for me.

[tyoc213]

I think I hit the limit resource limit exceeded. (Service: CloudWatchLogs...

copilot env deploy -n prod-env
✔ Proposing infrastructure changes for the appi-prod-env environment.
- Creating the infrastructure for the appi-prod-env environment.              [update rollback complete cleanup in progress]  [21.8s]
  The following resource(s) failed to create: [InternetGateway, Cluster,                                                               
   EnvironmentHostedZone, AddonsStack, VPC, LogResourcePolicy, CustomRes                                                               
  ourceRole].                                                                                                                          
  - A CloudFormation nested stack for your additional AWS resources                    [delete in progress]                            [21.7s]
    - A bucket policy to deny unencrypted access to the bucket and its contents        [not started]                                   
    - An Amazon S3 bucket, ask-kb-refs, for storing and retrieving objects             [delete in progress]                            [2.8s]
      Resource creation cancelled                                                                                                      
  - An ECS cluster to group your services                                              [delete in progress]                            [7.6s]
    Resource creation cancelled                                                                                                        
  - An IAM role to manage certificates and Route53 hosted zones                        [delete in progress]                            [7.6s]
    Resource creation cancelled                                                                                                        
  - Delegate DNS for environment subdomain                                             [not started]                                   
  - A Route 53 Hosted Zone for the environment's subdomain                             [delete in progress]                            [7.6s]
    Resource creation cancelled                                                                                                        
  - A security group to allow your containers to talk to each other                    [not started]                                   
  - Request and validate an ACM certificate for your domain                            [not started]                                   
  - An Internet Gateway to connect to the public internet                              [delete complete]                               [0.0s]
    Resource creation cancelled                                                                                                        
  - A resource policy to allow AWS services to create log streams for your workloads.  [delete complete]                               [4.2s]
    Resource handler returned message: "Resource limit exceeded. (Service:                                                             
     CloudWatchLogs, Status Code: 400, Request ID: a84ba9a5-19bc-4826-8adf                                                             
    -03e7bc802237)" (RequestToken: aef3454e-7e30-6cb5-3fd8-77d9242568db, H                                                             
    andlerErrorCode: ServiceLimitExceeded)                                                                                             
  - Private subnet 1 for resources with no internet access                             [not started]                                   
  - Private subnet 2 for resources with no internet access                             [not started]                                   
  - A custom route table that directs network traffic for the public subnets           [not started]                                   
  - Public subnet 1 for resources that can access the internet                         [not started]                                   
  - Public subnet 2 for resources that can access the internet                         [not started]                                   
  - A private DNS namespace for discovering services within the environment            [not started]                                   
  - A Virtual Private Cloud to control networking of your AWS resources                [delete complete]                               [2.8s]
    Resource creation cancelled                                                                                                        
✘ deploy environment prod-env: describe stack events appi-prod-env-AddonsStack-GOF3A7KQNEJ6: ValidationError: Stack [appi-prod-env-AddonsStack-GOF3A7KQNEJ6] does not exist
    status code: 400, request id: fc6097cc-bf77-4d54-93b9-945bc25b04e2