NLB Security validation error when deploying NLB for frontend service.

dmateos commented 1 year ago

Currently getting this error when trying to deploy a frontend service with version 1.30.0

deploy service frontend to environment dev: deploy service: check if changeset is empty: create change set copilot- for stack -dev-frontend: ValidationError: [/Resources/NLBSecurityGroup/Type/SecurityGroupIngress] 'null' values are not allowed in templates

Could this be related to the recent change allowing NLB's to support SG's AWS have released?

Relevent manifest entries:

http: false nlb: port: 80 network: vpc: placement: "private"

Lou1415926 commented 1 year ago

@dmateos Does your VPC only contain private subnets, not public subnets? For example, you might have an environment manifest that looks like this:

network:
  vpc:
     subnets:
        # no "public" field
        private:
            <some configuration>

dmateos commented 1 year ago

@Lou1415926

That does look to be the issue, so it seems you can only do a ALB without public subnets not a NLB.

Im trying to fit this in a multi account architecture where we have a networking account with the nat gateways, ALB to route to a NLB in the deployment account which is connected to the network account via a TGW.

Because our deployment accounts have no NAT GW etc its all handled by a TGW on the private subnet.

A work around seems to be let it deploy a private ALB (Backend service with http "/" defined), create a NLB with the ALB as a target group manually then point the ALB in the other account to that.

dmateos commented 1 year ago

I guess my new question is, can we make it possible to deploy a backend service with a NLB in a fully private subnet the same we can with an ALB.

The idea is i want the staitc ip addresses so i can add them to the ALB in my network account across the transit gateway.

Lou1415926 commented 1 year ago

@dmateos Ah, that makes sense. I take it that your NLB does not receive any traffic over internet - only traffic from the VPC (or over TGW) right? Unfortunately this is not nicely built into Copilot yet - the only built-in NLB today is internet-facing. However, what you are looking for is still definitely possible with some effort!

Using YAML Patch

I assume when you opened the issue, you were deploying the NLB with a Load-Balanced Web Service. You can try the following:

On the same LBWS, comment out the nlb field. Run copilot svc deploy.
Uncomment the nlb field, run copilot svc override. Do not run copilot svc deploy until the 4th step [1].

try the yaml patch feature with the following yaml patch file:

- op: replace
path: /Resources/PublicNetworkLoadBalancer/Properties/Scheme
value: internal
- op: replace
path: /Resources/PublicNetworkLoadBalancer/Properties/Subnets
value:
Fn::Split:
  - ","
  - Fn::ImportValue: !Sub '${AppName}-${EnvName}-PrivateSubnets'

Run copilot svc deploy This should create the NLB as an internal NLB, placed in private subnets.

Alternatively, you can replace step 1. and 2. with:

Delete the LBWS by copilot svc delete --env <the env you were trying to deploy into>
Run copilot svc override. Do not run copilot svc deploy until the 4th step [1].

There are two things to notice: [1] The reason why I asked that copilot svc deploy not to be run until step 4, was because otherwise you would likely get an error that says:

Resource handler returned message: "The following target groups cannot
be associated with more than one load balancer: arn:aws:elasticloadba
lancing:ap-northeast-1:568623488001:targetgroup/test-n-NLBTa-MLLTSYRUT
RCJ/342af9aab333bedf (Service: ElasticLoadBalancingV2, Status Code: 40
0, Request ID: 4f6357ea-546a-41d6-bf98-c0fc6e153d7b)" (RequestToken: f
3795b64-8f1c-6b1a-d348-79eb5a98f979, HandlerErrorCode: ServiceLimitExc
eeded)

at step 4. This is a result of the mechanism of how CloudFormation re-creates resources, plus the limitation that target groups have. [2] This requires you to still deploy the NLB with a Load-Balanced Web Service. If you need to deploy the NLB with a Backend Service, please let me know - I can help you with that too.

dmateos commented 1 year ago

Hi @Lou1415926

Wow awesome thanks for the response, i will try that out, it looks workable.

At the moment i have deployed it as a "backend" service so i can make the private ALB, however im happy to make it a Load-Balanced Web Service aslong as i can use the private NLB.

[2] This requires you to still deploy the NLB with a Load-Balanced Web Service. If you need to deploy the NLB with a Backend Service, please let me know - I can help you with that too.

Just out of interest how hard is this option vs as a Load-Balanced web-service.

Lou1415926 commented 1 year ago

Just out of interest how hard is this option vs as a Load-Balanced web-service.

In general, this would involve creating all NLB-relevant resources as an addon to your service, and use yaml patch to modify the main stack to use some value from the addon.

The addons template itself isn't too difficult to figure out:

You can write a manifest with your desired nlb config
Run copilot svc package --upload-assets against that manifest
Copy&paste the resources starting from PublicNetworkLoadBalancer into addons/nlb.yml

This is roughly the addons template you will need. There are several modifications that you need to do on top of this though:

Add ⬇️ on top of the addons template

Parameters:
App:
Type: String
Env:
Type: String
Name:
Type: String

Replace all ${AppName}, ${EnvName} and ${WorkloadName} with ${App}, ${Env}, ${Name}, respectively.
Finally, just read through the whole addons template to make sure that the configuration looks good to you.

For example, to create the stack that's more or less equivalent to ⬇️ manifest (as of Copilot v1.30.1):

nlb:
  port: 81/tcp

I would have an addon that looks like this:

Parameters:
  App:
    Type: String
  Env:
    Type: String
  Name:
    Type: String
Resources:
  PublicNetworkLoadBalancer:
    Metadata:
      'aws:copilot:description': 'A Network Load Balancer to distribute public traffic to your service'
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internal
      Subnets:
        Fn::Split:
          - ","
          - Fn::ImportValue: !Sub '${App}-${Env}-PrivateSubnets'
      Type: network
  NLBListener:
    Metadata:
      'aws:copilot:description': 'A TCP listener on port `81` that forwards traffic to your tasks'
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref NLBTargetGroup
          Type: forward
      LoadBalancerArn: !Ref PublicNetworkLoadBalancer
      Port: 81
      Protocol: TCP
  NLBTargetGroup:
    Metadata:
      'aws:copilot:description': 'A target group to connect the network load balancer to your service on port 81'
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 81
      Protocol: TCP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 60 # ECS Default is 300; Copilot default is 60.
      TargetType: ip
      VpcId:
        Fn::ImportValue: !Sub "${App}-${Env}-VpcId"
  NLBSecurityGroup:
    Metadata:
      'aws:copilot:description': 'A security group for your network load balancer to route traffic to service'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow access from the network load balancer to service
      SecurityGroupIngress:
        - CidrIp: 10.0.0.0/24
          Description: Ingress to allow access from Network Load Balancer subnet
          FromPort: 81
          IpProtocol: TCP
          ToPort: 81
        - CidrIp: 10.0.1.0/24
          Description: Ingress to allow access from Network Load Balancer subnet
          FromPort: 81
          IpProtocol: TCP
          ToPort: 81
      Tags:
        - Key: Name
          Value: !Sub 'copilot-${App}-${Env}-${Name}-nlb'
      VpcId:
        Fn::ImportValue: !Sub "${App}-${Env}-VpcId"
  # NLBDNSAlias: # You don't need this if you do not care about accessing the nlb via. svc-nlb.env.app.domain.com
  #   Metadata:
  #     'aws:copilot:description': 'The default alias record for the network load balancer'
  #   Type: AWS::Route53::RecordSetGroup
  #   Properties:
  #     HostedZoneId:
  #       Fn::ImportValue: !Sub "${App}-${Env}-HostedZone"
  #     Comment: !Sub "Default NetworkLoadBalancer alias for service ${Name}"
  #     RecordSets:
  #       - Name: !Join
  #           - '.'
  #           - - !Sub "${Name}-nlb"
  #             - Fn::ImportValue: !Sub "${App}-${Env}-SubDomain"
  #             - ""
  #         Type: A
  #         AliasTarget:
  #           HostedZoneId: !GetAtt PublicNetworkLoadBalancer.CanonicalHostedZoneID
  #           DNSName: !GetAtt PublicNetworkLoadBalancer.DNSName
Outputs:
  NLBTargetGroupId:
    Value: !Ref NLBTargetGroup

and a YAML patch that looks like this:

- op: add
  path: /Resources/Service/DependsOn/-
  value: AddonsStack
- op: add
  path: /Resources/Service/Properties/LoadBalancers/-
  value:
    ContainerName: <container that receives the traffic, likely the service name>
    ContainerPort: 81
    TargetGroupArn: 
      Fn::GetAtt: 
        - AddonsStack
        - Outputs.NLBTargetGroupId
- op: add
  path: /Resources/TaskDefinition/Properties/ContainerDefinitions/<index of the container that receives the traffic>/PortMappings/-
  value:
    ContainerPort: 81
    Protocol: tcp

There is one thing to notice❗I got the addons template above based on copilot-v1.30.1 svc package. However, recently AWS announced support for security group for NLB, which allows a more fine-grind ingress/egress control. copilot-v1.30.1 doesn't have this, but copilot.v1.31.0 probably will. You can either a) stick to what copilot-v1.30.1 gives you, which is completely fine; b) modify the template yourself to add the security group for NLB c) wait for copilot v1.31.0 to run svc package

If you need to give your NLB a nice DNS name (e.g. by using nlb.alias) - it could be a bit more work, especially for maintenance. Given your use case - that you wanted to allowlist the static IP address - I assumed you didn't need the alias right?

In general, this isn't too bad if you don't need alias - I would give it a try if time permits! Here are a few reasons why you'd choose this (backend+addons+yaml patch) over the other (LBWS+NLB+yaml patch):

LBWS is positioned as public services, whereas Backend is positioned as private. Therefore, it is possible that certain implementation around LBWS assumes that it is public. In the meantime, your LBWS is basically used as a private service, therefore, it is difficult for me to guarantee that future changes in LBWS won't break your YAML patch.
The YAML patch that I gave here, for the other option, assumes that the logical ID of the NLB is "PublicNetworkLoadBalancer". As a part of the effort to support security group for NLB, we might need to change the logical ID of that resource. If this happens, you will probably need to update the YAML patch, replacing "PublicNetworkLoadBalancer" with the new logical ID, which will be clearly printed out for you at the time!

Here are a few reasons why you'd go with the other option (LBWS+NLB+yaml patch):

Easier
Handle alias

If you go with LBWS+NLB+yaml path......

I assumed when you first opened the issue, you were deploying a Load-Balanced Web Service, is that correct? You probably already did this - but I wanted to mention that you might want to add http: false to your manifest to turn off public traffic, because it seems that your service is expected to be completely private.

gj02ib65 commented 1 year ago

Thank you for the work to, I assume, test, and certainly for writing up this explanation. I have a use case for internal NLB and am excited to see this option. I would also add a thumbs up to having an Internal NLB defined in the manifest like an internal ALB.

dmateos commented 1 year ago

Wow amazing amount of info, thank you so much

dannyrandall commented 1 year ago

Just wanted to link a few existing feature requests for internal NLB support: https://github.com/aws/copilot-cli/issues/3840, https://github.com/aws/copilot-cli/issues/5131. I'll close this one in favor of those two! Please give a 👍 on those issues!😊

aws / copilot-cli

NLB Security validation error when deploying NLB for frontend service. #5271

Using YAML Patch

If you go with LBWS+NLB+yaml path......