Proposal: Updated SDK client options for new DNS structure

lmars commented 6 months ago

This is a proposal for updating SDK client options to support our new DNS structure.

Background

Default Endpoints

By default, SDKs currently send REST and realtime requests to the following primary endpoints:

rest.ably.io (default primary REST endpoint)
realtime.ably.io (default primary realtime endpoint)

When there are problems using these default primary endpoints, SDKs switch to using a set of five default fallback endpoints for both REST and realtime traffic:

a.ably-realtime.com
b.ably-realtime.com
c.ably-realtime.com
d.ably-realtime.com
e.ably-realtime.com

Environment Endpoints

As part of the Active Traffic Management feature, we have the concept of an "environment" which results in 7 endpoints distinct from the default ones described above:

<env>-rest.ably.io (primary REST endpoint)
<env>-realtime.ably.io (primary realtime endpoint)
<env>-a-fallback.ably-realtime.com
<env>-b-fallback.ably-realtime.com
<env>-c-fallback.ably-realtime.com
<env>-d-fallback.ably-realtime.com
<env>-e-fallback.ably-realtime.com

SDKs are configured to use these environment endpoints by setting the environment client option.

For example, an SDK configured with environment="acme" will use the following endpoints:

acme-rest.ably.io (primary REST endpoint)
acme-realtime.ably.io (primary realtime endpoint)
acme-a-fallback.ably-realtime.com
acme-b-fallback.ably-realtime.com
acme-c-fallback.ably-realtime.com
acme-d-fallback.ably-realtime.com
acme-e-fallback.ably-realtime.com

Custom Endpoints

Customers can request to use endpoints under a domain they control (see here), in which case we agree a set of client-facing endpoints which they CNAME to the Ably controlled environment endpoints above.

For example, a customer who controls acme.com might use the following endpoints:

realtime.ably.acme.com (primary REST and realtime endpoint)
a-fallback.ably.acme.com
b-fallback.ably.acme.com
c-fallback.ably.acme.com
d-fallback.ably.acme.com
e-fallback.ably.acme.com

SDKs are configured to use these custom endpoints by setting the restHost, realtimeHost, and fallbackHosts client options, for example:

{
  restHost: "realtime.ably.acme.com",
  realtimeHost: "realtime.ably.acme.com",
  fallbackHosts: [
    "a-fallback.ably.acme.com",
    "b-fallback.ably.acme.com",
    "c-fallback.ably.acme.com",
    "d-fallback.ably.acme.com",
    "e-fallback.ably.acme.com",
  ],
}

DNS Restructure

We have recently completed a restructuring of our DNS, which includes altering the terminology used.

Terminology Change

The term "environment" has been replaced internally with either "routing policy" or "cluster" to distinguish between customers who use Active Traffic Management and those which have a dedicated cluster respectively. For the purposes of this proposal, the term "routing policy" is equivalent to the term "environment" used in the Background section above (i.e. it refers to a set of DNS records used for routing traffic).

Top Level Domain Changes

DNS now falls under different domains for prod traffic (i.e. all customer traffic) versus nonprod traffic (i.e. anything that is not customer traffic, like traffic from SDK tests, from website staging, or from development clusters).

Primary DNS has moved from the ably.io domain to the ably.net domain for prod traffic, and to ably-nonprod.net for nonprod traffic. Fallback DNS continues to exist under the ably-realtime.com domain for prod traffic, whereas nonprod fallback DNS exists under the ably-realtime-nonprod.com domain.

DNS continues to exist under the ably.io domain, although they are now CNAMEs to the equivalent DNS in ably.net or ably-nonprod.net.

Routing Policy Endpoints

We have dropped the split between REST and realtime endpoints. There is now a single primary endpoint which handles both types of traffic for a given routing policy, and there continues to be five fallback endpoints, so six endpoints in total for a given routing policy.

For routing policies which handle prod traffic, the six endpoints are:

<policy>.realtime.ably.net (primary endpoint)
<policy>.a.fallback.ably-realtime.com
<policy>.b.fallback.ably-realtime.com
<policy>.c.fallback.ably-realtime.com
<policy>.d.fallback.ably-realtime.com
<policy>.e.fallback.ably-realtime.com

and for nonprod traffic:

<policy>.realtime.ably-nonprod.net (primary endpoint)
<policy>.a.fallback.ably-realtime-nonprod.com
<policy>.b.fallback.ably-realtime-nonprod.com
<policy>.c.fallback.ably-realtime-nonprod.com
<policy>.d.fallback.ably-realtime-nonprod.com
<policy>.e.fallback.ably-realtime-nonprod.com

Default Endpoints

The default primary and fallback endpoints now fall under the "main" routing policy, and have a structure which is consistent with non-default routing policies. The default endpoints are now:

main.realtime.ably.net (primary endpoint)
main.a.fallback.ably-realtime.com
main.b.fallback.ably-realtime.com
main.c.fallback.ably-realtime.com
main.d.fallback.ably-realtime.com
main.e.fallback.ably-realtime.com

Custom Endpoints

Custom endpoints continue to be structured in a way described in the Background section above, although they now CNAME DNS records in ably.net rather than ably.io.

We do however suggest that customers use a well-known set of subdomains under their custom domain (i.e. realtime.<domain> for the primary endpoint, [a-e]-fallabck.<domain> for the fallback endpoints) which will allow us to infer the endpoints from a single client option.

Proposal

To support this updated DNS structure, I propose the following:

add a routingPolicy string client option to use non-default routing policy endpoints
deprecate the environment client option, although continue to accept it and use the old DNS structure when provided
add a customDomain string client option to use canonically named custom endpoints
add a primaryHost client option to use a non-canonically named custom primary endpoint, and continue to use fallbackHosts to set custom fallback endpoints
deprecate the restHost and realtimeHost client options in favour of primaryHost
add a nonprod boolean client option to use nonprod TLDs (i.e. ably-nonprod.net and ably-realtime-nonprod.com), defaults to false
update all SDKs to set routingPolicy="main" by default (perhaps do this sometime after delivering the above changes)

If there's agreement, I will formalise this within the spec.

Migration

A customer who previously used an Environment Endpoint with an environment="acme" client option will now use an equivalently named Routing Policy Endpoint with a routingPolicy="acme" client option.

A customer who previously used a canonically named Custom Endpoint by using the following client options:

{
  restHost: "realtime.ably.acme.com",
  realtimeHost: "realtime.ably.acme.com",
  fallbackHosts: [
    "a-fallback.ably.acme.com",
    "b-fallback.ably.acme.com",
    "c-fallback.ably.acme.com",
    "d-fallback.ably.acme.com",
    "e-fallback.ably.acme.com",
  ],
}

will now use a single customDomain="ably.acme.com" client option.

A customer who previously used a non-canonically named Custom Endpoint by using the following client options:

{
  restHost: "streaming.acme.com",
  realtimeHost: "streaming.acme.com",
  fallbackHosts: [
    "a-fallback.streaming.acme.com",
    "b-fallback.streaming.acme.com",
    "c-fallback.streaming.acme.com",
    "d-fallback.streaming.acme.com",
    "e-fallback.streaming.acme.com",
  ],
}

will now use primaryHost instead of restHost and realtimeHost:

{
  primaryHost: "streaming.acme.com",
  fallbackHosts: [
    "a-fallback.streaming.acme.com",
    "b-fallback.streaming.acme.com",
    "c-fallback.streaming.acme.com",
    "d-fallback.streaming.acme.com",
    "e-fallback.streaming.acme.com",
  ],
}

Developers who previously used the nonprod sandbox endpoint with an environment="sandbox" client option will now use routingPolicy=sandbox and nonprod=true client options.

graham-russell commented 6 months ago

As mentioned the other day. We will need to add parameters for HTTP & Websocket internet-up endpoints. These can be inferred for routing policies but specified for Custom CNAMEs.

AndyTWF commented 6 months ago

Some SDKs like ably-js still have even older client options host and wsHost. It probably isn't worth a major breaking version to remove them, but worth considering to avoid the ambiguity of three different schemes in that SDK?

Looks good though! 🎉

AndyTWF commented 6 months ago

Specifically for ably-js those two deprecated client options are being removed in v2 anyway, so all good!

ably / specification