worker service can expose a port

charlierudolph commented 6 years ago

Extracted from #685

The env var passed to other services would be

`<SERVICE>_HOST`

Local development value	ftp-api:2121
Local production value	ftp-api:21
AWS production value	ftp-api.local:65000

Useful for services that need to communicate directly with non-web services. note that for worker services we need to have them query the ec2metadata api and update their route53 record to match its scheduled EC2 box.

charlierudolph commented 6 years ago

Requires putting a script on each docker image

charlierudolph commented 6 years ago

Sample terraform: we would probably update the name and role to what we prefer

resource "aws_iam_role_policy" "allow_dns_changes" {
  name = "ecs-allow-dns-changes-dbp-${module.stack.environment}"
  role = "${element(split("/", module.stack.iam_role), 1)}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

Sample script: we would probably make the whole name an environment variable and thus would need a couple env variables, AWS_INTERNAL_ZONE_ID and SELF_HOSTNAME.

#!/usr/bin/env bash

set -euo pipefail

INSTANCE_IP="$(curl -fsSL http://169.254.169.254/latest/meta-data/local-ipv4)"

aws route53 change-resource-record-sets --hosted-zone-id "$INTERNAL_ZONE_ID" --change-batch '{
  "Comment": "Update nats record to point to this container",
  "Changes": [
      {
          "Action": "UPSERT",
          "ResourceRecordSet": {
              "Name": "nats.'$INTERNAL_ZONE_NAME'.",
              "Type": "A",
              "TTL": 30,
              "ResourceRecords": [
                  {
                      "Value": "'$INSTANCE_IP'"
                  }
              ]
          }
      }
  ]
}'

This script requires curl / awscli to be installed on the machine.

So when a worker service exposes a port we automatically add the necessary terraform. Adding the script may be a manual thing if we need to add it to the production docker image, but would need some way to disable it locally. Could have a command that generates the script for them and then they update their dockerfile with all the needed stuff.

We could see if we can somehow wrap the users production docker image building a new one that installs the deps, and updates the cmd to run the ip script first and then the default cmd. Not sure if that is possible.

Alternative / resource: https://github.com/vagalume/Route53-Updater-for-ECS

@hugobho @alexdavid thoughts?

hugoesthere commented 6 years ago

Just to confirm, we'd be adding the "route53:ChangeResourceRecordSets" rule to our existing ecs role policy right: https://github.com/Originate/exosphere/blob/master/terraform/aws/ecs-cluster/iam.tf#L23? This would give every service permission to modify route53. I'm not sure there's an easy way just to allow worker services that access. @alexdavid ?

Originate / exosphere

worker service can expose a port #869

`<SERVICE>_HOST`