Switch JSON source away from API and over to /index.json

[SirRawlins]

We now provide a JSON version of the status page data for every status page direct from the pages root url. Such as the one at http://status.sorryapp.com/index.json

This is served up through out Static Assets CDN so it much, much more performant than hitting the dynamic API.

We should switch the plugin to use this new datasource as it'll be far more performant and scale a great deal better.

[SirRawlins]

The easiest way to go about dropping this into place will be to just switch out the API endpoint variable and the parser logic so it still makes two requests for brand and notices.

Then we can extend on this by bringing the two requests together so they're pulled in just the once upon page load and cached.

[SirRawlins]

One initial worry with this now I've given the idea some thought is that we need to be able to make the request over HTTPS to avoid browsers throwing up warnings about unsecured content being included.

The only way I can see us doing this is to access to bucket direct using the AWS url, and not our own. This means the AWS wildcard certificate should work.

I'm going to look and see if this is viable or not.

[SirRawlins]

So a quick look at this we can do it - https://s3-eu-west-1.amazonaws.com/status.sorryapp.com/index.json

However this is on the proviso that it'll break if you add or change your CNAME at any point, as we'd need to resolve the active bucket, as redirects do not work, they move the traffic to the new domain and therefore break the SSL connection.

[SirRawlins]

The other downside to this is that it reveals the underlying infrastructure in the codebase and in the AJAX requests as we're making visible direct requests to AWS S3, rather than masking through our own domain/api.

[SirRawlins]

I think I'm worrying a little too much about this, if it works, it works, it'll be damn fast, rock solid etc.

[SirRawlins]

The first issue I've now stumbled across is that the buckets, or at least the JSON don't currently support cross-origin requests which is causing jQuery to bork.

We need to look at cross-domain request policy on the buckets, and see how we can best add that. I do know that they're able to support it, it's just not something we're currently configuring on our buckets.

[SirRawlins]

I've now added a simple default CORS config to our test bucket manually to see how this behaves.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

We''ll have to wait and see if this does what we want, we can then later figure out how we add this to all buckets related to the application.

[SirRawlins]

Not having a great deal with joy using that configuration. I'm going to try an alternative which we currently have set on our assets bucket and seems to work nicely.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
    </CORSRule>
</CORSConfiguration>

[SirRawlins]

O.k so that simplified option also did not work. I have now after some Googling found an option which works, and we need to include allow all headers option like so:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

This CORS config means we're now able to make a request to the bucket for the JSON without it shitting it's pants.

We're now getting an error which complains about the returned data structure which is a new shape to what it used to be, we'll need to rework the JS to have this parse it better.

[SirRawlins]

O.k so I've got the plugin now drawing from the new source which is great, and it seems nice and quick too which is fantastic news.

Before I go much further, or think about wrapping up the first experimental release we need to solve the bucket CORS issue on the application side, and ensure all the buckets are configured with the correct policy so the plugin will work moving forward.

[SirRawlins]

After sitting and thinking for a short while about this new implementation I'm not entirely happy with it.

We would be making the following sacrifices if we do this:

• We expose S3 as the underlying storage.
• We are not able to migrate away from S3.
• Customers show change their CNAME would break their plugin.
• If S3 buckets are not created their is no fallback mechanism (this might not be bad, don't want fallback hammering us.)
• We cannot tell when used have installed the plugin through the admin/ui

Some of these, namely the last one we can probably live with, however others are going to cause is all kinds of issues down the line.

[SirRawlins]

Current thoughts on a possible solution:

Stick some form of CDN middleman into the mix. Perhaps this caches this read-only data for a short period, or not at all, but acts as a nice SSL endpoint for the bucket.

We could then have a single new API subdomain, like api-ro.sorryapp.com or something like that, with a single S3 bucket on the back end which could take a hammering.

Only worry is that a single S3 bucket cannot take as much traffic as multiple S3 buckets, so we compromise ourselves in that respect.

[SirRawlins]

The other option is that we change the redirect setup on the S3 buckets, so we can always access the JSON in the root/original bucket should we wish to, over SSL.

The downside here is that we still end up hitting the S3 endpoint direct, and not through our own DNS is a bit of a tie-in to the platform, and feels like a bad idea.

[SirRawlins]

I've had some time to think overnight about this, and I now have a clearer idea of a possible option, but it'll depend on redirect options on S3.

In this new scenario, the plugin would continue to hit the sorryapp.com domain directly, and NOT the aws/s3 endpoint. This would keep the plugin flexible and un-prone to changes of infrastructure or CNAMES down the line.

The S3 bucket would then detect the request from the plugin, and redirect to the underlying AWS url, so that the request can be served over SSL.

This will work very nicely, but it means S3 needs to be able to redirect based on either a HTTP Header we pass, or by using some special redirect object which the plugin hits rather than the default index.json

[SirRawlins]

O.k so the first challenge with this approach is that hitting https:// on the S3 bucket doesn't work. S3 just sits and times the request out.

Potentially we could hit the non-ssl connection page from the plugin and get a redirect to the secure option, however, my guess is that the browser will get upset about doing this.

[SirRawlins]

I'm running some quick tests on hitting the http endpoint with a redirect to https however the browser doesn't like it - it get's upset and blocks the request, so that's likely to be a no-go. Going to keep experimenting, see what I can come up with.

[SirRawlins]

O.k so after some more experimenting it certainly looks like AJAX/CORS is pretty locked down and doesn't like any form of redirect on the requests, so this is pretty non-viable as an option, which is a damn shame.

[SirRawlins]

One possible option which popped into my head, is to the a HEAD request, which might not have to be over SSL, to get the redirect data, and then make the GET request to the correct URL.

[SirRawlins]

Here's some data on HEAD requests should we with to pull data on it.

https://ochronus.com/http-head-request-good-uses/

[SirRawlins]

HEAD doens't give us any data on it's redirect status so that's not going to be any help for my idea.

hmmm....

[SirRawlins]

O.k after running in circles over this Robin and I had a discussion this morning whereby we've said we're happy to make sacrifices in order to get it working, as we can always improve and deal with back-compat issues at a later date, but for now performance gains are most vital.

So, plugin will hit S3 endpoint directly, which means it is allowed to travel over SSL.

In order to get this working we need to make some minor changes on the app end, enabling CORS on the buckets, and also perhaps versioning the JSON files, so we can at least have a fixed endpoint?

[SirRawlins]

The app is now up to scratch for this. However we are no longer providing /index.json as an endpoint. This endpoint is now versioned, which gives us better change protection moving forward.

//s3-eu-west-1.amazonaws.com/example.sorryapp.com/api/v1.json

Is now the correct endpoint url to be using for the JSON.

[SirRawlins]

Been through a QA session with Rob1 and all looking good. I'm going to merge in the changes and wrap up a release.