Experiment with Signed API without ELB

[Siegrift]

It should be possible to deploy Signed API without an load balancer and expose its public IP. Note, that we never use the IP directly as we have the CDN put on top of it. Using the ELB seems extraneous because there is just a single instance it forwards the requests to.

This could decrease the AWS costs by a large margin as can be seen on the invoice from January.

The ELB returns 502 when the service is unavailable or under heavy load. I am suspicious whether it works correctly, because we see 502 errors but AWS reports low CPU and memory usage. But it serves as a natural SPAM protection because it will guard the service from potential spam attacks.

[bbenligiray]

Why is this on hold?

[Siegrift]

I added the label after I opened https://github.com/api3dao/signed-api/pull/218 which was the prototype, but with some heavy downsides. I added "on hold" label, because I wasn't sure whether we want to proceed with this or not.

On the call (3 weeks ago) we talked about trying https://stackoverflow.com/questions/57940616/static-outbound-ip-for-aws-ecs-fargate-task so I removed the label and will try it out.

[Siegrift]

I've looked at the solution described in the link and I am not sure how it can be applied to our problem.

The problem discussed in the issue is to make the outbound traffic from the service come from a static IP address. For that one can use NAT gateway which translates the private IP address of the service with the static IP of the NAT gateway.

One of the comments suggests whitelisting the whole subnet, but that seems to refer to the target service (called by the AWS service that needs to have the static IP). In fact, our service allows traffic from any IP.

The problem we have is that service IP changes each time a task is redeployed, but we need it to be static to use it with CDN. To assign static IP, it seems we need to use ELB.

[bbenligiray]

Yeah I see, https://stackoverflow.com/a/73038635 only gives you an outbound static IP

[Siegrift]

I discussed the problem with Stefi and we came up with an alternative, that I like the most. Have a single load balancer that distributes the traffic to different services based on path. This is currently done by CDN (but it's costly).

With this new format, we would have a CDN (for caching) that forwards the traffic to ALB (because it checks target healthiness, and reliably forwards traffic to the service). The costs for load balancer should be much less, because only 1 will be running (and one of the things we're paying is hourly rate). The CDN bill should decrease as well (because we pay for each evaluated path forward rule) and this is not a cost in ALB.

The downside is that the deployments will not be isolated anymore and will share the VPC (and thus need to be in the same region). If one region is not enough for deployments, we can deploy the same stack (ALB + services) on another region.

Few links to support the claim:

• https://medium.com/cognitoiq/how-cognitoiq-are-using-application-load-balancers-to-cut-elastic-load-balancing-cost-by-90-78d4e980624b
• https://www.reddit.com/r/aws/comments/yc61zo/is_it_normal_for_elb_to_cost_so_much/

[Siegrift]

All Signed APIs under API3 are redeployed as proposed here