Spike: Rate limiting incoming traffic at ALB

[nikolai-efimov]

User Story - Business Goal

To ensure business continuity in case of spikes in traffic and/or DDOS attacks

• [x] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)

As a system I want to rate limit incoming traffic before it hits app So that so that the system is protected from being overwhelmed during spikes of incoming traffic

As a system I want to rate limit incoming traffic before it hits app So that so that the system is protected from DDOS attacks

Additional Info and Resources

Initial searched showed that most likely the solution in our case will be AWS WAF, which works well with ELB

• https://aws.amazon.com/blogs/security/three-most-important-aws-waf-rate-based-rules/
• https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html
• https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-rate-based-request-limiting.html

Acceptance Criteria & Checklist

Please keep in mind, we're talking about the ALB layer, not the app layer for this research.

• [x] Verify that AWS WAF is the right approach (other possibilities might be Nginx or Kong)
• [x] Technical Documentation has been created and linked in the glossary
• [x] Learnings have been shared with the team, and decisions have been documented
• [ ] Tickets for next steps have been identified and created

[mjones-oddball]

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @justaskdavidb2 @k-macmillan @kalbfled @ldraney @nikolai-efimov

[npmartin-oddball]

Corey will be picking this up. His assigned buddy will be Cris.

[cris-oddball]

From Corey:

Looked at the ALBs we have with Cris in the AWS console, and reached out to Kyle to clarify some problems we are currently facing with regard to rate limiting. AWS WAF should provide a low maintenance, AWS native way to solve a lot of the problems in this ticket. I will look at drafting up a proposal and sharing it out tomorrow.

[cris-oddball]

On hold for Corey to complete some on-boarding, will pick it up again tomorrow, Friday 7/25, at the latest.

[cris-oddball]

Spike documentation has been added to the team repo. Meeting has been scheduled for next Weds - this ticket is on hold until then.

[coreycarvalho]

Findings have been presented to the team. @k-macmillan and I reached out to the team that maintains the VA reverse proxy to see if our WAF solution will be suitable for controlling public traffic too.

Expect to see an ADR created tomorrow.

[coreycarvalho]

ADR has been created. The team decided that we can close this out even though we are still waiting for some information on how the reverse proxy works.

Once I get that information, I am going to create the tickets necessary to start working on this.

[coreycarvalho]

After speaking with the rev proxy team, it looks like we might not get the public IP forwarded on to us in a way that can be easily detected. We might get it via the X-Real-IP header, but that might not be a reliable way to consistently get a public IP.

Regardless, rev proxy said that there are multiple DDoS prevention methods at multiple layers before us from the VA and from them. Since this is the case, we can focus on our internal use case of using WAF to prevent self DDoSing and an extra layer of API protection.