API scalability: Analyze ECS and ALB configuration & make recommendation

[lizkrznarich]

Service/repository new-deployment

Describe the current state/issue Currently, ROR has 2 container instances running the API app in prod and 1 each in staging and dev. Each env has 1 ALB handling requests to the container(s). There is no auto-scaling to increase the number of container instances or ALBs. While ALB and app CPU capacity appear to be less of a bottleneck during peak usage than Elasticsearch CPU, increased resourcing during peak usage would likely help to prevent any issues in the future.

Describe the desired state/solution Research and make a recommendation for changes to ECS and ALB configuration to add redundancy and scale resourcing horizontally. Related to roadmap item https://github.com/ror-community/ror-roadmap/issues/124

Additional notes

• Search rate and CPU usage are somewhat predictable, but there are large spikes and a general increase as new integrators start using ROR APIs

[lizkrznarich]

• Auto-scale ECS tasks up to 3(?) based on CPU usage (80%) https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-autoscaling-targettracking.html

• Multiple ALB instances are not needed at this time, as ALBs automatically scale. Also, since we have target groups in 2 availability zones, 2 ALB nodes already exist. We could consider serving API requests to internal applications (ror-app, future testing tools, etc) from an internal endpoint/port. In that case, it may make sense to use 2 separate ALBs. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html