Document inband channel occupancy events

[SimonWoolf]

As described in https://gist.github.com/paddybyers/d350f491d0807360620e37eb4d3af66c .

NB: per the comment I added on permissions, in order to keep our public permissions system simple (well, not 'simple', but not too much more complex than it already is), we are not documenting the second bullet point of that comment (channel-metadata:publishers). Our public API will be that you need the channel-metadata permission to get streamed occupancy events.

┆Issue is synchronized with this Jira Task by Unito

[mattheworiordan]

Updated at https://gist.github.com/mattheworiordan/43c2a2ea9263573987c83a45cf89fbe4

[SimonWoolf]

Not clear whether we intend to actually document this; and if so whether we're documenting it for everyone to use or only for enterprise customers. There's no mechanism at the moment to limit it in that way; currently anyone who knows about it can use it. cc @kavalerov

[SimonWoolf]

my recommendation is that we document this and make it available to everyone, and emphasize it as the best way to get occupancy if you happen to already be subscribed to a channel over polling the channel status api (because polling is inefficient) or using the channel.lifecycle metachannel (which doesn't scale well as you accumulate more and more channels).

No point making this an enterprise-only feature if it just means other customers will fall back to getting the same information in a way that is more expensive for us to service (polling the channel status rest api)

[kavalerov]

Updated formatting on Matt's gist: https://gist.github.com/kavalerov/f3bd37b7d76467653a50c260f8055623

[SimonWoolf]

Also I'd argue we should remove this ^ from the application settings. There are use-cases for sending occupancy events for all channels to the central channel.lifecycle metachannel, but they are relatively niche, and it doesn't scale to large numbers of channels. IMO dangling this as a pushy enterprise upsell (which is what putting it prominently in application settings is doing) as if it's the best way to get occupancy is giving it more prominence than it deserves

[kavalerov]

Was the original thinking behind putting it there just to upsell Enterprise, or were there any other considerations (if you know)?

[SimonWoolf]

shrug. added in https://github.com/ably/website/pull/1890 which was mainly about adding the channel-metadata capability to API keys, doesn't look like there was much discussion.

What's the next step to push this forward? Now Mark has left we don't have a dedicated docs writer - is devrel taking over that, or does one of the realtime team need to translate this to a docs pr?

[ably-sync-bot]

➤ Ivan Kavalerov commented:

We are actually going to have a tactical regroup regarding documentation early next week I believe, so unless this is extremely critical (and I think it is important, but not critical, but correct me if I am wrong) - I think we will be tackling it after that meeting.

[mattheworiordan]

Yes it's about upselling. It's a very common UX pattern to disable features users don't have access to drive upgrade discussions.

If it's not clear what that does we should fix that.

[SimonWoolf]

@mattheworiordan My understanding was that the main reason we disabled the feature by default was because for 95% of users it would be a bad idea to enable it. Exposing every occupancy change for every channel as an event on the central lifecycle metachannel is... I mean, it does have its usecases (which is why it was implemented), but since it doesn't scale well as your channel count increases the usecases that it's the best choice for are relatively rare. (And enabling it for people who don't actively need it ruins the lifecycle channel's utility as a lifecycle channel, by flooding it with messages that could well overflow the channel's capacity and lead to it dropping actual lifecycle events, etc.).

So my understanding was that it's enterprise-only because we don't want to enable it for anyone unless we've discussed their usecase with them beforehand and made sure this is actually the best solution for them, and we can only justify the time investment of that discussion for enterprise customers. Which I'd argue means we giving it such a prominent position as an upsell is an odd choice - "upgrade to enterprise to get this feature, which once you upgrade we will tell you you probably shouldn't use"..?

Now that we have occupancy events on the channel itself (which is what this issue is for), that's a better choice for the vast majority of people, it avoids the scaling issues

[kavalerov]

@SimonWoolf are connections one of the members of metrics.category? It is shown in the examples in the gist, but not mentioned here: https://www.ably.io/documentation/realtime/channel-metadata#occupancy

[SimonWoolf]

Yes it is. It's in there under "+ See an example ChannelDetails JSON object", but should be in the table. That table is also wrong for another reason (misses out the metrics nesting; again the example json object is correct).

[SimonWoolf]

We are going to do this: see decision outcome and plan in https://github.com/ably/ideas/issues/430#issuecomment-729027125 . Docs repo tasks:

• Document inband channel occupancy events as a feature that is available to everyone, as described in https://gist.github.com/paddybyers/d350f491d0807360620e37eb4d3af66c

• Reconsider the channel metadata API docs and the tutorial) to move away from a presentation that frames polling the REST channel status api as the non-enterprise version of the enterprise-only realtime channel occupancy events sent on the lifecycle channel. Remove all mention from the docs of the latter 'feature'. New parts of the channel metadata API are:

  • Inband channel occupancy events, as the primary mechanism for getting occupancy updates in realtime
  • Lifecycle channels, as a centralised way of finding out about channels opening and closing
  • A channel occupancy reactor rule source, to get occupancy updates from a set of channels in a centralised way (that doesn't involve attaching to all of them) -- this does not exist yet, but is planned, and at least initially will be enterprise-only or a business upsell. So this will be a 'get in touch if you want this' type thing
  • The channel status REST api, which should emphasize that it is for one-off updates, and if you want occupancy events in realtime you should use one of the above mechanisms and not poll the REST api.
  • Channel enumeration, which should be de-emphasized, and made very clear that it is for occasional use by servers, and concurrent attempts to enumerate channels will just be refused

• Have a new page as the main source of information on metachannels, which mostly aren't really part of the channel metadata API (the channel lifecycle channel kindof is, but the other metachannels definitely aren't), and afaict are on that page at the moment just because there's no other place for them.

At Ivan's suggestion, will leave the implementation of all this as an onboarding execise for @tbedford with he joins at the beginning of December.

[ably-sync-bot]

➤ Tony Bedford commented:

A simple example in JavaScript showing how to subscribe to inband events would be very useful. The only example I have seen is the old way of doing things.

[SimonWoolf]

A simple example in JavaScript showing how to subscribe to inband events would be very useful. The only example I have seen is the old way of doing things.

There's a 1-to-1 mapping between the old way and the new way for all channel params, so analagous to any of the other ones on https://www.ably.io/documentation/realtime/channels/channel-parameters/overview#using-parameters-ably .

Explicit example:

const channel = realtime.channels.get('foobar', { params: {occupancy: 'metrics' } });
channel.subscribe('[meta]occupancy', (msg) => {  console.log(msg.data); });

[ably-sync-bot]

➤ Tony Bedford commented:

Also clarify difference between occupancy vs presence

[ably-sync-bot]

➤ Tony Bedford commented:

Note to add on meta channel occupancy (not the recommended approach): "However, since this involves funnelling occupancy updates for every channel in the app to a single metachannel, it does not scale well as the number of channels grows, so has limited use cases. Almost always, either inband channel occupancy events (for situations where the entity wanting to know the occupancy is attached to the channel) or occupancy rules (where it isn't) are better fits."

[ably-sync-bot]

➤ Tony Bedford commented:

Probably should change this https://www.ably.io/tutorials/channel-occupancy-events ( https://www.ably.io/tutorials/channel-occupancy-events ) tutorial to use inband occupancy events.

[ably-sync-bot]

➤ Tony Bedford commented:

Now on staging. Just need to do a website PR to bump docs after I’ve done some more checks on staging.

[ably-sync-bot]

➤ Tony Bedford commented:

Now on live website