Figure out how to serve an AWS Lambda function with a Function URL from a custom subdomain

[simonw]

[This issue thread provides a blow-by-blow account of how I figured out the way to serve an AWS Lambda function from a custom domain, using CloudFront and ACM]

I think Cloudfront is the way to do this:

• https://aws.amazon.com/blogs/networking-and-content-delivery/using-amazon-cloudfront-with-aws-lambda-as-origin-to-accelerate-your-web-applications/
• https://serverlessland.com/patterns/cloudfront-lambda-urls

My previous notes on how I shipped the Lambda Function URL are here: https://til.simonwillison.net/awslambda/asgi-mangum

[simonw]

In /Users/simon/Dropbox/Development/datasette-on-lambda-prototype on my computer, using tips from https://til.simonwillison.net/awslambda/asgi-mangum

I added datasette-x-forwarded-host to the requirements.txt file there and ran:

docker run -t -v $(pwd):/mnt \
  public.ecr.aws/sam/build-python3.9:latest \
  /bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"

Then:

rm -rf function.zip
(cd lib; zip ../function.zip -r .)

But which function is it? https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions helped me identify it as https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/my-test-lambda-python-function?tab=code

So:

aws lambda update-function-code \
  --function-name my-test-lambda-python-function \
  --zip-file fileb://function.zip

But having deployed that EVERY page is a 500 server error (if you skip the cache).

[simonw]

I forgot to add lambda_function.py and fixtures.db to the zip file.

zip function.zip lambda_function.py
zip function.zip fixtures.db
aws lambda update-function-code \
  --function-name my-test-lambda-python-function \
  --zip-file fileb://function.zip

[simonw]

That deploy worked but it did not fix the bug - even though https://lambda-demo.datasette.io/-/plugins shows the new plugin.

[simonw]

https://lambda-demo.datasette.io/-/asgi-scope seems to indicate that x-forwarded-host is not provided by Cloudfront:

 'headers': [[b'sec-fetch-mode', b'navigate'],
             [b'x-amzn-tls-version', b'TLSv1.2'],
             [b'sec-fetch-site', b'none'],
             [b'x-forwarded-proto', b'https'],
             [b'x-forwarded-port', b'443'],
             [b'dnt', b'1'],
             [b'x-forwarded-for', b'24.5.172.176'],
             [b'sec-fetch-user', b'?1'],
             [b'via',
              b'2.0 549a5eaa264d3b997d6acfdba72f56d0.cloudfront.net (CloudFr'
              b'ont)'],
             [b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'],
             [b'x-amzn-trace-id', b'Root=1-633a1410-62bde31d28b9714b06d45b27'],
             [b'host',
              b'fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws'],
             [b'upgrade-insecure-requests', b'1'],
             [b'accept-encoding', b'gzip'],
             [b'x-amz-cf-id',
              b'kfdCiG-Q0XGPjb-Efmws8N2BbCJF-slwm8f_ZbSvQVGWDMP2bj02Iw=='],
             [b'sec-fetch-dest', b'document'],
             [b'user-agent', b'Amazon CloudFront']],

[simonw]

https://aws.amazon.com/premiumsupport/knowledge-center/configure-cloudfront-to-forward-headers/ says:

1. Follow the steps to create a cache policy using the CloudFront console.
2. Under Cache key settings, for Headers, select Include the following headers. From the Add header dropdown list, select Host.

[simonw]

Tried editing behavior and adding this:

[simonw]

That seemed to break everything: https://lambda-demo.datasette.io/ now returns:

{"Message":null}

[simonw]

AccessDeniedException looks relevant.

[simonw]

Removing Host fixed it. Adding Host broke it again.

[simonw]

Definitely proven to myself that having Host as a header that gets sent through breaks the application and causes every page to return a AccessDeniedException error.

[simonw]

I turned on a whole bunch of those extra headers:

I tried to do even more but it gave me an error (tucked away at the bottom of the page where I initially missed it) complaining I had selected too many.

Now https://lambda-demo.datasette.io/-/asgi-scope has this:

 'headers': [[b'cloudfront-viewer-country', b'US'],
             [b'x-amzn-tls-version', b'TLSv1.2'],
             [b'sec-fetch-site', b'none'],
             [b'cloudfront-viewer-postal-code', b'94018'],
             [b'x-forwarded-port', b'443'],
             [b'sec-fetch-user', b'?1'],
             [b'via',
              b'2.0 e2d7efb4a6fe4a49c212c47079f43f9c.cloudfront.net (CloudFr'
              b'ont)'],
             [b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'],
             [b'cloudfront-viewer-country-name', b'United States'],
             [b'host',
              b'fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws'],
             [b'upgrade-insecure-requests', b'1'],
             [b'cloudfront-viewer-city', b'El Granada'],
             [b'sec-fetch-mode', b'navigate'],
             [b'x-forwarded-proto', b'https'],
             [b'dnt', b'1'],
             [b'x-forwarded-for', b'24.5.172.176'],
             [b'cloudfront-viewer-country-region', b'CA'],
             [b'cloudfront-viewer-time-zone', b'America/Los_Angeles'],
             [b'x-amzn-trace-id', b'Root=1-633a1788-3196e3a46e240cf84721ba58'],
             [b'cloudfront-viewer-longitude', b'-122.47390'],
             [b'cloudfront-viewer-latitude', b'37.51410'],
             [b'cloudfront-viewer-country-region-name', b'California'],
             [b'accept-encoding', b'gzip'],
             [b'x-amz-cf-id',
              b'g9dQ7qE_E8mL3_-7A118ZbnSlEfkNb0jRHzDM59JY3ehcnir4GD4qA=='],
             [b'sec-fetch-dest', b'document'],
             [b'user-agent', b'Amazon CloudFront']],

That latitude/longitude is on the edge of the small town where I live, so pretty accurate!

[simonw]

These look relevant to my Host problem:

• https://serverfault.com/questions/888714/send-custom-host-header-with-cloudfront
• https://brianschiller.com/blog/2018/10/24/cloudfront-host-header

[simonw]

Aha! Here's a Twitter thread where someone notes that Host breaks things, but X-Forwarded-Host is not supported (when using Cloudfront and API Gateway): https://twitter.com/matthieunapoli/status/1288444807011065856

[simonw]

From this PR it looks like the answer may be to set x-forwarded-host using a CloudFront function.

• https://github.com/getlift/lift/pull/44

[simonw]

Apparenty CloudFront Functions are NOT the same thing as Lambda@Edge: https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/

[simonw]

OK I'm going to make a function.

function handler(event) {
    event.request.headers["x-forwarded-host"] = event.request.headers["host"];
    return request;
}

[simonw]

ARN is:

arn:aws:cloudfront::462092780466:function/Set-X-Forwarded-Host-Header

[simonw]

Then you have to "publish" it (I forgot to do this):

Now you can associate it from the functions page:

I hope "Viewer Request" (as opposed to "Viewer Response") was the right guess!

[simonw]

https://lambda-demo.datasette.io/ now says:

% curl -i 'https://lambda-demo.datasette.io/'
HTTP/2 503 
server: CloudFront
date: Sun, 02 Oct 2022 23:21:57 GMT
content-type: text/html
content-length: 999
x-cache: FunctionExecutionError from cloudfront
via: 1.1 f85d379725bf31eb2428acfa2b9da6e6.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: rYuOIEz9U6vm04p5xFfqri6xSJ3FihQKrupKTIP-CZ_Nh1aKJ3TtSw==

So it didn't like my new function.

[simonw]

I tried the "Test" feature and it showed a useful error:

Fixed the implementation to:

function handler(event) {
    event.request.headers["x-forwarded-host"] = event.request.headers["host"];
    return event.request;
}

[simonw]

That fixed the error but https://lambda-demo.datasette.io/-/asgi-scope still shows no x-forwarded-host header.

[simonw]

Trying this instead:

That caused a 503 error again.

[simonw]

I switched it back to Viewer Request.

[simonw]

Saw this in the "Test" view:

The CloudFront function returned an invalid value: request.headers is malformed. invalid request header, key 'x-forwarded-host', keyMultiValue is not an object

[simonw]

I think this might be it:

function handler(event) {
    event.request.headers["x-forwarded-host"] = {
        value: event.request.headers["host"].value
    };
    return event.request;
}

Clue was here: https://www.honeybadger.io/blog/aws-cloudfront-functions/

That seems to work in the test mode:

That worked!!

 'headers': [(b'cloudfront-viewer-country', b'US'),
             (b'x-amzn-tls-version', b'TLSv1.2'),
             (b'sec-fetch-site', b'none'),
             (b'cloudfront-viewer-postal-code', b'94018'),
             (b'x-forwarded-port', b'443'),
             (b'sec-fetch-user', b'?1'),
             (b'via',
              b'2.0 f7aef728fd226cb808d34cb93114336c.cloudfront.net (CloudFr'
              b'ont)'),
             (b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'),
             (b'x-forwarded-host', b'lambda-demo.datasette.io'),
             (b'cloudfront-viewer-country-name', b'United States'),
             (b'host', b'lambda-demo.datasette.io'),
             (b'upgrade-insecure-requests', b'1'),
             (b'cloudfront-viewer-city', b'El Granada'),
             (b'sec-fetch-mode', b'navigate'),
             (b'x-forwarded-proto', b'https'),
             (b'dnt', b'1'),
             (b'x-forwarded-for', b'24.5.172.176'),
             (b'cloudfront-viewer-country-region', b'CA'),
             (b'cloudfront-viewer-time-zone', b'America/Los_Angeles'),
             (b'x-amzn-trace-id', b'Root=1-633a1fcd-3d0392966fa468423e62cd20'),
             (b'cloudfront-viewer-longitude', b'-122.47390'),
             (b'cloudfront-viewer-latitude', b'37.51410'),
             (b'cloudfront-viewer-country-region-name', b'California'),
             (b'accept-encoding', b'gzip'),
             (b'x-amz-cf-id',
              b'4n8H9CDFEVLWnnCAV41QZaTAGzT4T52bqF3Dy4GZHDUBdxu8rScnBA=='),
             (b'sec-fetch-dest', b'document'),
             (b'user-agent', b'Amazon CloudFront')],

[simonw]

And now the "suggested facets" links on https://lambda-demo.datasette.io/fixtures/compound_three_primary_keys go to the right place.

[simonw]

It looks like the homepage / doesn't set a cache-control: max-age= header at all, and as a result Cloudfront defaults to caching it for 24 hours.

% curl -s -i https://lambda-demo.datasette.io/ | head -n 15
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 2675
date: Sun, 02 Oct 2022 23:35:07 GMT
x-amzn-requestid: de5693c7-47f4-403b-8606-83fab9827158
link: https://lambda-demo.datasette.io/.json; rel="alternate"; type="application/json+datasette"
x-amzn-trace-id: root=1-633a202b-251a20ca5abdeb9f59160250;sampled=0
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 f85d379725bf31eb2428acfa2b9da6e6.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: aaAXEqtcMhgfOnbQgD8ur8HmR1jxZNfYW2_r1OTnZTgK6RlV3FNdpQ==
age: 131

<!DOCTYPE html>

Note the age: 131 header there, showing it was cached 131 seconds ago.

Other pages on the site with cache-control headers max out at 5 seconds as they should:

% curl -s -i 'https://lambda-demo.datasette.io/fixtures/facetable' | head -n 15
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 34484
date: Sun, 02 Oct 2022 23:37:46 GMT
x-amzn-requestid: b1ce10b8-4275-4df6-9680-b9dcab21ef3f
referrer-policy: no-referrer
cache-control: max-age=5
link: https://lambda-demo.datasette.io/fixtures/facetable.json; rel="alternate"; type="application/json+datasette"
x-amzn-trace-id: root=1-633a20ca-0539b22874f7aa8443da0663;sampled=0
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 52a50599e55838e3cced4f5e481dca9e.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: hQGjCckm7Je4yN6oktuf57vgTGD87mgfii4PKjKaq1UOucRcXwntaQ==
age: 3