Implement load-balancer + WAF for automated DDoS prevention

[dunkOnIT]

While we don't expect this to yield results overnight, it creates an infrastructure that (as far as I understand) makes it much easier for us to selectively deny requests overloading our servers.

Full discussion here, with relevant sections quoted below:

b) we switch to the AWS load balancer and attach a WAF (web application firewall) to it. There we can define rules like if X amount of traffic comes from IP Y for Z amount of time block it.

As we are planning to switch to a load balanced approach anyway, b) seems the most reasonable. Each rule check does come with a price though, which we need to look into. As it depends on traffic we might save some money from not serving the Malicious requests and it will even itself out.

It would not be much effort to set up the infrastructure (maybe an hour), I think the biggest change we need to think about how the load balancer integrates with the web sockets. ALB does support Web sockets, but we need to change the SSL Termination to the ALB and think about correct values for idle connection timeout settings.

[dunkOnIT]

@FinnIckler assigning you here as you seem to know what's going on. Not as high priority as the automated instance upgrades/downgrades in my view, but probably still nice to set up - especially if it isn't too much effort (famous last words lol)

Feel free to pushback/suggest reassignment as needed.

[FinnIckler]

We did implement the load balancer and it feels like it already helped a bit with traffic. Looking at the logs our 500 count is 0.1%! Implementing WAF will need approval from Budgeting