API PRD instance load balancing and scaling rules

[logstar]

@blackdenc - Could you specify the current load balancing and dynamic scaling rules for the OpenPedCan API production instance? The production instance will soon be published.

Following are the rules that I remembered, which might be incorrect:

• There are 5 instances always running.
• Concurrent API HTTP requests will be routed to different instances for load balancing.
• New instances will be spawned if CPU utilization is above certain threshold. As R plumber HTTP server handles request sequentially, the CPU utilization for handling multiple requests will always stay low, so probably no instance will be spawned by the scaling rule.
• The maximum number of instances is 10.
• Each instance has 10GB mem and 4 CPU cores as specified in the Jenkins file.

Related to #55 and #27.

cc @chinwallaa

[devbyaccident]

There are 5 instances always running.

Yes, that is configured in this parameter: https://github.com/PediatricOpenTargets/OpenPedCan-api/blob/0a7046b1dedc7a7b954400edae7a45b4d60a8a98/Jenkinsfile#L14

Concurrent API HTTP requests will be routed to different instances for load balancing.

We don't have the module this is setup in public, but yes, I can confirm that we do have these instances setup as ECS tasks behind an application load balancer that will route traffic to available instances.

New instances will be spawned if CPU utilization is above certain threshold. As R plumber HTTP server handles request sequentially, the CPU utilization for handling multiple requests will always stay low, so probably no instance will be spawned by the scaling rule.

We scale based on the ECSServiceAverageCPUUtilization metric, which is measuring the CPU utilization across all tasks in the service, meaning that if each task stays busy for a prolonged period of time, i.e, a steady stream of users, new tasks will be spawned to handle the load. More info on how AWS calculates that metric can be found here.

The maximum number of instances is 10.

Correct, here is where that parameter is defined: https://github.com/PediatricOpenTargets/OpenPedCan-api/blob/0a7046b1dedc7a7b954400edae7a45b4d60a8a98/Jenkinsfile#L15

Each instance has 10GB mem and 4 CPU cores as specified in the Jenkins file.

Correct, here are the values we pass into the creation of the ECS task: https://github.com/PediatricOpenTargets/OpenPedCan-api/blob/0a7046b1dedc7a7b954400edae7a45b4d60a8a98/Jenkinsfile#L16-L19

[logstar]

@blackdenc - Thank you for the specifications on the load balancing and scaling rules.

@chinwallaa - I was wondering if there is any other information about load balancing and scaling rules that would need to be specified.