Connecting to many domains and cloning distributions

Another question, might be unrelated and too specific: My application needs also to be connected to 50 domains. With API Gateway it was easy, I would just create an ACM certificate for them and map them to the same API that serverless created.

Now with CloudFront it is a bit more complicated. Each ACM certificate can have 10 domains. That means I will need 5 CloudFront distributions at minimum. So my options are two:

Somehow using serverless.yml create 5 distributions, 5 S3 buckets, 5 lambda functions (10 actually). So everytime I deploy it deploys all of them so they are in sync. It's messy, but should work.
Deploy only to one distribution like it is now, and then clone it through AWS. The cloned versions will use the same S3 Bucket and Lambda functions that the original used. It's clean. The problem is that if I deploy to the original distribution, the Lambda functions version updates, and I need to update it also across the cloned distributions somehow.

I would rather go with option 2, as it's way cleaner. I just don't think this component supports this, and if not, how can I develop this. I'm wondering if someone faced this problem of managing multiple (many) domains and how did they solve this? As I said, before with API Gateway it was extremely easy.

Multi-domain support will be incredibly useful.

For those who are looking for a solution, I created two scripts: aws_add.js which let you add a cloned distribution, and aws_update.js which updates all the cloned distributions upon deployment. I'm also using multi-env with serverless based on this: https://github.com/serverless-nextjs/serverless-next.js/blob/master/packages/serverless-components/nextjs-component/examples/multiple-instance-environment/serverless.js.

What I do is pretty simple - every deploy, I fetch all the cloned distributions and I update their settings based on the original distribution. So all the distributions share the same S3 Origin and same Lambda functions. The aws_add.js is there just to create manually some cloned distributions

aws_add.js

just call it with node aws_add.js

require("dotenv").config({ path: `${__dirname}/.env.local` });
var AWS = require("aws-sdk");
var cloudfront = new AWS.CloudFront({
  accessKeyId: process.env.AWS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_KEY,
});

const readline = require("readline");
const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

let dist_id, domain, certificate;

rl.question("Is it production? [Y/n]:\n", (env) => {
  dist_id =
    env.trim().toLowerCase() == "y" || env.trim().length == 0
      ? process.env.DIST_ID_PROD
      : process.env.DIST_ID_DEV;
  rl.question("Enter domain:\n", (url) => {
    domain = url.trim();
    rl.question("Enter ACM for this domain:\n", (acm) => {
      certificate = acm.trim();
      rl.close();
    });
  });
});

rl.on("close", () => {
  main();
});

async function main() {
  try {
    let dist_config = await new Promise((resolve, reject) => {
      cloudfront.getDistributionConfig({ Id: dist_id }, function (err, data) {
        if (err) reject(err);
        else resolve(data.DistributionConfig);
      });
    });

    if (domain && domain.length) {
      dist_config.Aliases = {
        Quantity: 1,
        Items: [domain],
      };
    }

    if (certificate && certificate.length) {
      dist_config.ViewerCertificate = {
        ACMCertificateArn: certificate,
        SSLSupportMethod: dist_config.ViewerCertificate.SSLSupportMethod,
        MinimumProtocolVersion:
          dist_config.ViewerCertificate.MinimumProtocolVersion,
      };
    }

    dist_config.CallerReference = new Date().getTime().toString();

    let tags = {
      Items: [
        {
          Key: "clone",
          Value: dist_id,
        },
      ],
    };

    let dist = await new Promise((resolve, reject) => {
      cloudfront.createDistributionWithTags(
        {
          DistributionConfigWithTags: {
            DistributionConfig: dist_config,
            Tags: tags,
          },
        },
        function (err, data) {
          if (err) reject(err);
          else resolve(data.Distribution);
        }
      );
    });

    console.log(
      `\n\nADDED DISTRIBUTION\n===================\n\bID:\n${dist.Id}\nDomain:\n${dist.DomainName}`
    );
  } catch (err) {
    console.log(err.message);
  }
}

aws_update.js

require("dotenv").config({ path: `${__dirname}/.env.local` });
var Promise = require("bluebird");
var AWS = require("aws-sdk");
var cloudfront = new AWS.CloudFront({
  accessKeyId: process.env.AWS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_KEY,
});

var args = process.argv.slice(2);

const dist_id =
  args[0] == "production" ? process.env.DIST_ID_PROD : process.env.DIST_ID_DEV;

main();

async function main() {
  try {
    console.log(`UPDATING DISTRIBUTIONS\n===================\n`);

    let original_dist = await new Promise((resolve, reject) => {
      cloudfront.getDistributionConfig({ Id: dist_id }, function (err, data) {
        if (err) reject(err);
        else resolve(data);
      });
    });

    const list = await new Promise((resolve, reject) => {
      cloudfront.listDistributions({ MaxItems: "1000" }, function (err, data) {
        if (err) reject(err);
        else resolve(data.DistributionList);
      });
    });

    let clones = [];

    await Promise.each(list.Items, async (item) => {
      const tags = await new Promise((resolve, reject) => {
        cloudfront.listTagsForResource({ Resource: item.ARN }, function (
          err,
          data
        ) {
          if (err) reject(err);
          else resolve(data.Tags);
        });
      });

      tags.Items.map((tag) => {
        if (tag.Key == "clone" && tag.Value == dist_id) clones.push(item);
      });
    });

    await Promise.each(clones, async (clone) => {
      let clone_config = await new Promise((resolve, reject) => {
        cloudfront.getDistributionConfig({ Id: clone.Id }, function (
          err,
          data
        ) {
          if (err) reject(err);
          else resolve(data);
        });
      });
      clone_config.DistributionConfig.DefaultCacheBehavior =
        original_dist.DistributionConfig.DefaultCacheBehavior;
      clone_config.DistributionConfig.CacheBehaviors =
        original_dist.DistributionConfig.CacheBehaviors;

      return new Promise((resolve, reject) => {
        cloudfront.updateDistribution(
          {
            DistributionConfig: clone_config.DistributionConfig,
            IfMatch: clone_config.ETag,
            Id: clone.Id,
          },
          function (err, data) {
            if (err) reject(err);
            else {
              console.log(
                `Updated ${
                  clone.Aliases.Items.length
                    ? clone.Aliases.Items[0]
                    : clone.DomainName
                }`
              );
              resolve();
            }
          }
        );
      });
    });

    console.log(`\n\nDONE\n===================\n`);
  } catch (err) {
    console.log(err.message);
  }
}

And finally:

package.json

{
  "name": "server-crm",
  "version": "0.3.0",
  "private": true,
  "scripts": {
    "devOld": "next dev -p 5000",
    "dev": "node server.js",
    "build": "next build",
    "start": "next start",
    "deployDev": "serverless --stage=development && node aws_update.js",
    "deployProd": "serverless --stage=production && node aws_update.js production"
  }
}

I think it makes sense to support multiple domain records pointing to a single distribution - for example if you have localized subdomains or a similar use case. There are a few other similar issues e.g https://github.com/serverless-nextjs/serverless-next.js/issues/620. It should be relatively straightforward to support it. For now, I think a workaround is to actually forgo the domain input in this component and use something like Terraform to manage your Route53 domain records / CloudFront alternate domain names directly. Personally, I am doing that since my Route53 records and app are on different accounts anyway.

For having multiple CloudFront distributions, that's definitely a very interesting use case. It seems like it could add a lot of complexity to the component. Purely hypothesis, but I'm wondering if it could also be simplified using Terraform?

Define X CF distributions in Terraform, and point their lambda_function_association and origin to those of the original CF distribution (you don't need to import this to Terraform).
After the serverless component has updated, run terraform apply and Terraform might be able to detect new Lambda@Edge versions (you may need to import the Lambda itself into Terraform so it knows about it?) and update the X CF distributions accordingly?

I think it makes sense to support multiple domain records pointing to a single distribution - for example if you have localized subdomains or a similar use case. There are a few other similar issues e.g #620.

In my specific case I can't have multiple domains point to the same distribution. The domains have different ACM SSL certificates, and cannot be combined into a single certificate (because they are completely independent, and I don't want them to be dependent on the same certificate. I don't want that if one domain fails to authorize the certificate by removing the DNS records, it will fail the whole certificate). Each distribution is allowed to have only one SSL certificate. This case is actually not that specific, because when the domains are unrelated to each other (not subdomains of the same domain), you will need one-to-one relationship between the certificates and the distributions.

Yup, I think I understand your use case - just wanted to refer to a sort of related issue, which may possibly have similar implementation details.

@dphang, do you have an example of how you have setup terraform to work with serverless? I aim to have something like this:

dev.website.com
www.dev.website.com (S3 redirect with CF distribution)
staging.website.com
www.staging.website.com (S3 redirect with CF distribution)
etc.

(Note: I'm not concerned yet with having dev and staging, etc. in a single yaml, only a single stage dev for now. I'll figure out the multi-stage later)

I can setup the Route53 and ACM records with terraform, and the CF distribution, S3 bucket redirect etc for the www->apex redirect, and even the main CF distribution with ACM initialised for using with serverless next. But when I try to use the distribution id for CF in serverless next, I just get ping-pong conflicts back and forth. The serverless next deploy also creates a new origin set, which if I need to update the terraform, terraform will simply delete it.

I've also tried creating the bucket origin with the terraform CF distribution, and specifying that in the bucketName input with limited success. The core problem is the master of the IaC. If I setup with terraform, I then update with serverless NextJS, but that means I can no longer use terraform again to make changes again due to the conflicts.

I've tried use the domain, certificateArn, aliases, and all other combinations but without success. As you've referenced a number of times you use terraform, it would be appreciated if you could share an example. Thanks in advance.

@damo78

Thanks for sharing. I think mine is simpler. I just have one domain pointing to the CF distribution URL. What I do is Terraform now manages the ACM cert (wildcard cert) and Route53 record (just one pointing to CF distribution URL). The serverless next.js component manages the CF distribution (I do not put it in Terraform, I'll explain why in a bit).

ACM:

resource "aws_acm_certificate" "example_certificate" {
  domain_name               = "example.com"
  subject_alternative_names = ["*.example.com"]
  validation_method         = "DNS"
}

Route53:

resource "aws_route53_record" "record" {
  zone_id = zone.primary.zone_id
  name    = "test.example.com"
  type    = "A"

  alias {
    name                   = "distributionUrl"
    zone_id                = "distributionZoneId"
    evaluate_target_health = false
  }
}

I am manually inputting the CF distribution data in the Route53 record since I do not want Terraform to manage it at all. The "problem" I found is this component tends to be more destructive than TF, but only be because it tries to abstract most of the configuration away from you and set some sensible defaults i.e it creates various cache behaviors and other inputs by default (e.g if you did not specify).

Whereas TF only tries to keep your infrastructure in sync with attributes you've specified/included, but does not touch attributes you may have manually updated outside of TF.

So I suspect you may have a similar problem. Try to let the component manage the CF distribution entirely if possible. So you can do one of two things:

Use S3 to sync and manage the .serverless state (example in the README), so you don't need to specify distributionId anywhere else. This component completely manages the distribution.
You can keep using distributionId input (and managing the distribution in Terraform), but I think you need to figure out all the CF attributes you are modifying or are implicitly modified by the component, and do not specify those in the Terraform resource. Basically, keep it minimal. For example, don't specify any other origin behaviors than the default (as that's managed by this component). But I think you have to specify an origin and default_cache_behavior and viewer_certificate at the very least, so that has to be the same as ones set by the component.

I've been using (1) for my app with good success. I haven't tried (2), I think if TF config for the distribution is minimal (just the required attributes, and those are in sync with this component's defaults), then it may work well - since TF should not be destructive to attributes you don't specify. Though it may create a bit more work if you update values - e.g if you update the default cache behavior in the component, you must update it in TF config.

TL;DR: I manage the CF distribution completely with this component due to its more destructive behavior when updating its state. Terraform manages the cert, Route53 records, etc.

Hope it helps.

PS: for your www to apex redirect, why not try using the domainRedirects input, which will redirect in the handler? Then you would have one fewer distribution to manage (although it sounds like you already manage it well in Terraform). Although the caveat is you can't redirect /_next/static/* URLs since that behavior does not have the Lambda handler attached to it.

Thanks @dphang. Ultimately I've been aiming for this to be a totally IaC solution with CI/CD pipeline so any manual config management would let it down. This is why I gave up on using this component for the CF distribution, certificate and route 53. I tried so many combinations of domain, certificateArn, domainRedirects, etc. and gave it up as there was always a missing element (for now). I'll relook at these in the future.

I believe I may have now had some success using terraform for the two distributions (www redirect) and main nextjs, ACM and Route 53, whilst using the distributionId and explicit bucketName in the component. These parameters I plan to pass/retrieve between TF and SLS NextJS using AWS parameter store, but in the meantime my limited component spec is like this:

serverless-nextjs-app:
  component: '@sls-next/serverless-component@1.18.0-alpha.5'
  inputs:
    cloudfront:
      distributionId: E262927EXAMPLE
      aliases: ['dev.website.com'] # note this will use AWS SSM parameter
    bucketName: explicit-bucket-name-from-terraform # note this will use AWS SSM parameter
    name:
      defaultLambda: some-name-default-lambda
      apiLambda: some-name-api-lambda
    runtime:
      defaultLambda: 'nodejs12.x'
      apiLambda: 'nodejs12.x'

They key recent change that made this work was the recent fixing of aliases I saw you point out elsewhere, which a few days ago didn't work.

What I have had to do with the terraform is clone the cloudFront/s3 module I was using (Forked version of cloudposse/terraform-aws-cloudfront-s3-cdn) so that I could add a lifecycle { ignore_changes} block - terraform doesn't allow this to be dynamic so I couldn't use the same module for both. Currently I have this working using ignore_changes = all, but I expect with work I can be more specific. It essentially ignores anything the serverless nextjs component updates it with, but should allow any TF-related updates I need to do.

The ACM and Route 53 is created in a separate terraform module and is complete before the distribution is created. So, basically after creation in terraform, from then on the serverless component takes up the management of the CF config. I'm sure there'll be some danger in here, but I'll keep trying.

Once I have a robust end to end solution working I'll come back and post. Thanks again for your prompt response and please keep up the great work!

@damo78, yep, IaC is way more maintainable, so I agree there. The component itself is obviously not a full-fledged IaC so it probably could be improved to be more compatible when used with proper IaC providers. I am also thinking on how we can improve its flexibility.

In other words, maybe something like this can build all the deployment artifacts, but you can provide your own TF config (and are responsible for all of CloudFront configuration) and then the component can have a lightweight mode that just updates the necessary pieces: Lambda@Edge + S3 on each deployment. A couple of others have asked to separate build + deploy step, so I think it has enough traction (that will be the first step to get there).

Thanks for the feedback - we will consider all this in the future.

serverless-nextjs / serverless-next.js