Configurable header forwarding

[46bit]

Hi 18F! At GDS we use a fork of your CDN service broker on the GOV.UK PaaS. We recently added a provision option that gives users control over headers being forwarded to their applications. Hopefully our implementation can be of use to you in addressing #46.

Background

Amazon CloudFront forwards a very limited set of headers by default. The existing broker also forwards the Host header when a service is set to use the default origin (i.e., when forwarding to CloudFoundry's router.)

When making requests to the origin, CloudFront's caching mechanism associates HTTP requests with their response. The more variation within the forwarded request, the fewer cache hits and the less effective the cache. Limiting the headers forwarded is therefore key to cache performance—but in our experience applications often require headers that CloudFront does not forward by default.

Amazon allow specifying (whitelisting) 10 non-default headers to forward. Given we already forward Host to CloudFoundry, this leaves 9 headers that can potentially be configured. CloudFront also supports forwarding all headers using * as a header name, which entirely disables caching.

Our usecase

We've had two requests to forward custom headers:

1. A heavily-cacheable application detects the output format (e.g., JSON or XML) using the request's Accept header. The Accept header is not forwarded by default.
2. An application needs many or all headers to be forwarded and is not cacheable.

(1) was addressed by allowing users to whitelist extra headers to be forwarded. However (2) requires forwarding more than 9 headers and for this purpose we have allowed use of the * wildcard to forward all headers.

The headers provisioning option

This commit adds an optional headers field to the provisioning details, which can optionally be given as an array of strings. For instance:

• {"domain": "example.com", "headers": []} or omitting headers altogether will forward the Host header and Amazon's default headers.
• {"domain": "example.com", "headers": ["Accept"]} will forward the Accept header to the app, as well as Host and Amazon's default headers.
• {"domain": "example.com", "headers": ["*"]} will forward all headers to the app. In this case, Amazon disables caching.

We validate the contents of headers to avoid manual debugging:

• No more than 9 header names can be provided;
• The same header name cannot be provided more than once;
• The wildcard header name * cannot be accompanied by other header names.

Final notes

For further details you can see the README changes, our Pull Request for this at alphagov/paas-cdn-broker#5, and the relevant changes to our tech docs at alphagov/paas-tech-docs#53.

[wslack]

@46bit this is so exciting to see! I use the broker all of the time and just so I'm clear; this code is about which headers to accept, not adding new headers on its own - right?

[46bit]

@wslack Woo :-) Yes, it's to control which HTTP headers will be forwarded by CloudFront, rather than adding new header values.

You've raised a good point: we could have a more descriptive name for the setting than headers. We chose to stay consistent with your existing cookies setting, which controls cookie forwarding.

[mogul]

@46bit are more commits coming on this one? I'd like to get it merged.

[46bit]

@mogul Thanks for the reminder, yes indeed!

[46bit]

Yay! Sorry I didn't deal with the headers count--I tried several ways of clearing it up, but I wasn't happy code for varying between 9 and 10 based on the presence of Host.