[RFC] Proxy url includes to avoid CORS issues

[apmorton]

The current implementation of #include <http://url.com/header.h> has a few issues - the biggest of which is not being able to include files from websites that don't have CORS headers configured.

I think we could use cloudfront and lambda@edge to proxy and cache these requests to get around the CORS issues without a major risk of abuse using the technique shown here: https://aws.amazon.com/blogs/networking-and-content-delivery/dynamically-route-viewer-requests-to-any-origin-using-lambdaedge/

We may also consider using cloudflare workers instead (https://workers.cloudflare.com/). The biggest vector for abuse on cloudfront would be consuming excess bandwidth to run our bill up. On cloudflare we would not need to be concerned because there are no bandwidth charges. The free tier (plus 5$ a month for the workers addon) would be sufficient.

To further curb potential misuse we could do any combination of:

• wrap response in json or some other encoding (which would make it unsuitable for anyone trying to use it as their own cdn)
• limit cors allowed domains to our own (although this would make it impossible for unofficial CE installations to use if they wanted)
• always serve text/plain MIME type, which would prevent someone from using us to proxy malicious html/javascript/whatnot
• use a custom user agent on upstream requests so we can be identified and blocked if desired
• limit upstream response sizes

[partouf]

I'm wondering if we could not also just build it into CE as a separate call. That way we'll also have the thing under source control, and we'll notice sooner if it's being abused.

[apmorton]

My biggest concern with that is serving arbitrary content under our own domain. We can take steps outlined above to mitigate the risk, but in the end we would be letting anyone proxy arbitrary content through the main domains.

If we host this on a separate cloudfront or cloudflare endpoint it could operate under a separate domain and share zero trust roots in browsers with the main site.

Under cloudfront we could leave it as blahblahblah.cloudfront.net (or just register another throwaway domain like ce-proxy.net or something) - I'm not entirely sure how cloudflare works here, we may need to register a separate domain if they don't provide a default for us to use.

[mattgodbolt]

Why not "just" have CE do this in-process; as in the CE process goes out and fetches? (or initiates a call to something which does). The fact the client does it is a horrible hack. On the server we get the "wrap response in JSON" for free.