Attaching WAF to ALB via an env addon is complicated

[lgarvey]

Because the environment load balancer is only provisioned when the first service is deployed, it’s not possible to initially deploy the environment with the waf config.

So, to deploy we have to follow this sequence:

1. Deploy the environment
2. Deploy a first service
3. Add the environment waf addon cfn
4. Redeploy the environment

I haven’t yet tried deploying multiple environments, but I imagine this will become onerous - we’d have to remember to follow the above sequence each time a new environment is deployed.

Then deleting the service and environment also requires a special order.

1. Delete the waf addon con
2. Deploy the environment
3. Delete the service
4. Delete the environment

Ideally the WAF config would only apply if the PublicLoadBalancer resource exists. Is there a way to achieve this?

[huanjani]

Related: #4422

[huanjani]

Hi, @lgarvey!

I can see that this process is cumbersome. What if you add a Condition (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) in your env addon CFN template?

[lgarvey]

I will test adding a condition to enable waf per env, but presumably it will still fail because of the !Ref PublicLoadBalancer in the addons.parameters.yml file.

Parameters:
  LBARN: !Ref PublicLoadBalancer

[lgarvey]

A condition doesn't seem to help deploy an env

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  LBARN:
    Type: String
Mappings:
  EnvWAFConfigurationMap:
    # Create an entry for each environment

    prod:
      WebACLArn: 'arn:aws:wafv2:eu-west-2:[redacted]'

    staging:
      WebACLArn: 'arn:aws:wafv2:eu-west-2:[redacted]'

    training:
      WebACLArn: 'arn:aws:wafv2:eu-west-2:[redacted]'

    dev:
      WebACLArn: 'arn:aws:wafv2:eu-west-2:[redacted]'

Conditions:
  HasLB: false  #!Not [!Equals [!Ref LBARN, ""]]

Resources:
  WebACLAssociation:
    Type: 'AWS::WAFv2::WebACLAssociation'
    Condition: HasLB
    Properties:
      WebACLArn: !FindInMap [EnvWAFConfigurationMap, !Ref Env, WebACLArn]
      ResourceArn: !Ref LBARN

Results in the following vague CFN error and nothing useful in cloudtrail:

ResourceNotReady: failed waiting for successful resource state: Nested change set arn:aws:cloudformation:eu-west-2:[redacted]:changeSet/copilot-8173ce84-778f-4d6a-a78b-c78c8bce87b8-AddonsStack-18HK2MAKC67OJ/fd395038-68e7-4c8e-8e0e-2d73cd1bd1fb was not successfully created: Currently in FAILED.

[iamhopaul123]

Hello @lgarvey. Agreed it is not a pleasant experience. We could potentially have multiple ways of mitigating this problem:

• Allow users to enable ALB in env manifest
• Allow users to bring their own ALB (related to https://github.com/aws/copilot-cli/issues/1457)
• We have native support for WAF

We'll update here and please stay tune and keep track of this issue. Thank you!

[lgarvey]

Hi,

That sounds great to me. We will solve the problem using boto3 in the meantime. We're building a copilot-helper command line app to add some additional features and simplify some aspects of copilot so that it's easier to use across our department which has multiple product/development teams supported by cross-cutting SRE & Platform teams.

We envisaged the WAF ACL ARN being put into the environment manifest file: https://github.com/uktrade/copilot-tools/blob/main/commands/copilot_cli.py#L222

Thanks!

On Thu, 15 Jun 2023 at 17:30, Penghao He @.***> wrote:

Hello @lgarvey https://github.com/lgarvey. Agreed it is not a pleasant experience. We could potentially have multiple ways of mitigating this problem:

• Allow users to enable ALB in env manifest
• Allow users to bring their own ALB (related to #1457 https://github.com/aws/copilot-cli/issues/1457)
• We have native support for WAF

We'll update here and please stay tune and keep track of this issue. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/aws/copilot-cli/issues/4988#issuecomment-1593382746, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB2CY3E6FLNHKVKMSFWPQDXLM2EBANCNFSM6AAAAAAZGGINZA . You are receiving this because you were mentioned.Message ID: @.***>

[afgallo]

Just adding a reference to this for consideration: https://github.com/aws/copilot-cli/discussions/4968

[KollaAdithya]

Hey @lgarvey !

Just a follow up on this

I will test adding a condition to enable waf per env, but presumably it will still fail because of the !Ref PublicLoadBalancer in the addons.parameters.yml file.

Parameters:
  LBARN: !Ref PublicLoadBalancer

why do you think this will fail because of !Ref PublicLoadBalancer

[lgarvey]

There isn't a PublicLoadBalancer resource until the first service is deployed. So if that config is sat in the copilot/environments/addons directory, the initial copilot env deploy fails.

[KollaAdithya]

Yeah that's true until your first service is deployed. There will not be any PublicLoadBalancer available. Currently, you need to follow the following pattern always in order to attach WAF via env Addons.

1. Deploy the environment without Addons.
2. Deploy the service
3. Attach the environment Addon for WAF
4. Redeploy the environment with Addons.

These are the Addons that i used to attach WAF to LoadBalancer. a. In addons.parameter.yml

Parameters:
  LBARN: !Ref PublicLoadBalancer

b. In addonstemplate.yml

Parameters:
  App:
    Type: String
    Description: Your application’s name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  LBARN:
    Type: String
    Description: ARN of the Load balancer.
    Default: “”
Conditions:
  HasLB: !Not [!Equals [!Ref LBARN, ""]]
Resources:
  WEBACL:
    Type: AWS::WAFv2::WebACL
    Condition: HasLB
    Properties:
      Name: myWebACL
      DefaultAction:
        Block: {}
      Scope: REGIONAL
      VisibilityConfig:
        CloudWatchMetricsEnabled: true
        MetricName: BlockedRequests
        SampledRequestsEnabled: true
  ALBWebACLAssociation:
    Type: AWS::WAFv2::WebACLAssociation
    DependsOn: WEBACL
    Properties:
      ResourceArn: !Ref LBARN
      WebACLArn: !GetAtt WEBACL.Arn

[lgarvey]

Yes, and then it also imposes a specific order when deleting the stack, and we'd have to remember to follow the steps every time an environment is created or deleted: manually move the waf/addons.params file out of the copilot/environments/addons directory, deploy the env, deploy the service, move the files back and redeploy the env.

I work on the SRE/platform team for a UK public sector organisation and we're looking to replace Cloudfoundry/Gov UK Paas with Copilot. Because the solution will be used by development teams with support from the central platform team, we're looking to reduce the complexity as much as possible.

For now we'll probably manage the WAF configuration outside of Copilot.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.

[bathsundeep-graticule]

Parameters:
  LBARN: !Ref PublicLoadBalancer

@KollaAdithya How does this work? Maybe I'm misunderstanding something but the environment stack is not exporting PublicLoadBalancer as an output

https://github.com/aws/copilot-cli/blob/87308f4a50fe22dd3e8c85b9ea567bdca51053de/templates/environment/versions/cf-v1.0.0.yml#L192

[Lou1415926]

hey @bathsundeep-graticule ! I believe @KollaAdithya's example was for environment addons. In this case, PublicLoadBalancer is a resource created in the environment template, the same stack as the addon's parent stack, meaning that you can get its ARN just by !Ref PublicLoadBalancer.

If you need the public load balancer's ARN in a service addon though, that'd be a cross-stack reference, and you are right you'd need the ARN in the environment stack's output. While we indeed do not output the ARN, you could construct the ARN from what we do outputs: https://github.com/aws/copilot-cli/issues/2254#issuecomment-861658890!