search

warp-contracts / gateway

A meta-gateway for Arweave SmartWeave interaction transactions.
30 stars 10 forks source link

Deployment via ECS #96

Open ppedziwiatr opened 2 years ago

ppedziwiatr commented 2 years ago

Current configuration:

  1. main instance: EC2 on t3.medium: https://eu-north-1.console.aws.amazon.com/ec2/home?region=eu-north-1#InstanceDetails:instanceId=i-0eda8ef302509bb14

Runs in node cluster mode (https://nodejs.org/api/cluster.html#cluster), started via PM2 (https://pm2.keymetrics.io/docs/usage/cluster-mode/) with -i max. In case of t3.medium this effectively means 2 nodes.

image

Requires:

Additional notes: Only one worker in cluster runs the gateway tasks (e.g. transactions syncing, transactions verification) All workers in cluster run the http server and handle both the POST and GET requests. This is handled via gateway.lock file.

  1. replica instance: EC2 on t3.micro: https://eu-north-1.console.aws.amazon.com/ec2/home?region=eu-north-1#InstanceDetails:instanceId=i-0d80ce350fba0431c Also runs in node cluster mode, with the --replica param. This effectively means, that node only serves data - i.e. serves only "GET" requests and does not run any tasks/jobs.

  2. ALB - https://eu-north-1.console.aws.amazon.com/ec2/home?region=eu-north-1#LoadBalancers:sort=loadBalancerName sw-gw-alb with sw-gw-target-group https://eu-north-1.console.aws.amazon.com/ec2/home?region=eu-north-1#TargetGroup:targetGroupArn=arn:aws:elasticloadbalancing:eu-north-1:731675056359:targetgroup/sw-gw-target-group/d11c3b67c5addb3a This group has both the above EC2 instances connected.

  3. Cloudfront - https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=eu-north-1#/distributions/E2UWDPG250O74Q - https://d1o5nlqr4okus2.cloudfront.net with the alb and main instance node origins connected. All the 'post' requests are sent to the main node. The rest is sent to the ALB. Some caching policies are enabled for certain endpoints - https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=eu-north-1#/distributions/E2UWDPG250O74Q/behaviors This cloudfront distribution is connected to the gateway.redstone.finance domain.

Max-Levitskiy commented 2 years ago

Planned changes:

  1. We don't need pm2 anymore. ECS will manage horizontal scale across cluster nodes.
  2. We try to read all secrets and configurations from environment variables. If an environment variable doesn't exist, use current behavior. In the future, we should use some configuration library, which gives files/environment configuration from the box.
  3. gateway.lock file - we should definitely change this approach. It doesn't work in multi-node/zone/region environments. The easiest way for now - we can run two different ECS services. The first one is for the "master" node, which will have 1 instance, and an environment variable says it should do additional work. And the second one is for "worker" nodes with horizontal autoscaling.
ppedziwiatr commented 2 years ago

So in current env. there are in fact 3 possibilites:

  1. main node that runs all the tasks AND serves both the POST and GET requests
  2. main node that serves both the POST and GET requests (when gateway.lock file is present)
  3. replica node, that serves only GET requests (when run with --replica param)

But yeah, in ECS env this must be handled by some kind of flag/env. variable.

If we remove the sequencer from the equation (i.e. from the gateway node) - https://github.com/warp-contracts/gateway/issues/89 - the above could be reduced to:

  1. main node that runs all the tasks AND serves both the POST and GET requests
  2. replica nodes, that serves both the POST and GET requests

The sequencer-related endpoint is currently the only one, that cannot be scaled horizontally "easily".

Max-Levitskiy commented 2 years ago

Ok, so for compete this story, we need to complete #97 and #89 before.