MQTT Topic Hierarchy

[joepavitt]

With our own MQTT Broker offering now, we are looking to extend value here by displaying the known topic hierarchy for messages being sent to the relevant Team's clients.

Technical Discussion Points

### Pre-requisites
- [ ] Establish method for extracting hierarchy

UX Design Phases

Phase 1 - Basic View

View-mode only, render the topic hierarchy for the broker. Note that the view now has two tabs: "Hierarchy" and "Clients".

[joepavitt]

RE: Extraction methods, @hardillb has done preliminary investigations here, and found that we can utilise the EMQX API in order to return a topic hierarchy. This would be the first piece of our MQTT architecture that requires broker/brand-specific code, but the value here is important and we should proceed with that.

The alternative, non-brand specific option was to setup clients to subscribe to # for each team's broker, and record the hierarchy ourselves, but computationally, this would be excessive.

[hardillb]

EMQX provides the following API endpoint

https://docs.emqx.com/en/emqx/v5.8/admin/api-docs.html#tag/Topics/paths/~1topics/get

This requires an admin API token to authenticate and replies with the following structure:

{
  "data": [
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/d/DeqNr3g6p5/command"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/a/rGymXb1wWp/command"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/d/K6jN3Z4ybv/command"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/a/NvBmA71zAo/command"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/p/152fb007-7eb4-4e41-ad3b-8f946d11108b/in/foo"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/+/d/+/logs/heartbeat"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/+/d/+/response/96ffc27e-b60d-42da-83f9-1eb42baf4ba1"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "$share/platform/ff/v1/+/d/+/status"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "$share/platform/ff/v1/+/l/+/status"
    },
    {
      "node": "emqx@node1.emqx.com",
      "topic": "ff/v1/32E4NEO5pY/p/+/out/foo"
    }
  ],
  "meta": {
    "count": 10,
    "limit": 100,
    "page": 1,
    "hasnext": false
  }
}

This includes all topics on the broker, so will need filtering for a give team's internal topics ( ff/v1/[team id]/c) and have the prefix removed.

This could be exposed via an api endpoint that takes the team as an argument.

[joepavitt]

@hardillb for server-side work here, how much time would you need?

In the case of ff/v1/32E4NEO5pY/p/+/out/foo - the plus represents just any string, right?

[hardillb]

@joepavitt The hardest part will be sorting out the correct sharing of the secrets for the API keys.

Other than that it should just be a http client that can handle walking the pages to get all the data and then filtering it (probably best to filter it on each response to help limit amount of data need to be held in ram).

I can probably write a first version of the code this afternoon, but the logistics of the other bit will take longer

[joepavitt]

Thanks Ben, and in the case of the +, just so I understand for the Hierarchy view, do we get information on what strings have populated that content, or just that the subscription is listening to +?

[hardillb]

yes, MQTT wildcards come in 2 flavours

• + matches any value at a single level (between 2 /) in the topic
• # matches 0-n levels, but can only appear at the end of a topic. e.g. foo/bar/# is valid and matches both foo/bar/baz and foo/bar/baz/qux but foo/#/bar is not valid

The data returned includes both the topics that messages are published to and the subscription patterns that clients have requested.

So in the case of a subscription pattern with a wildcard, we should see duplicates in the data, in that we will see the subscription pattern and and topics which matched that patter, which in turn will provide the actual matches.

For the topic view, I suggest we filter out obvious subscription patterns (those with wildcards) and only show the actual data (or at least have it togglable)

[hardillb]

Just been testing this some more, and the EMQX endpoint might not actually give us what we need.

It's only showing subscribed topic patterns, not the topics messages are published on (which makes sense, from a broker design point of view) but the built in topic viewer in the EMQX dashboard had suggested it was both.

[hardillb]

The alternative approach of having the topic tree populate live using a MQTT WebSocket client is the other option, but carries the risk of the browser having to receive (and then discard) all the message payloads and also potentially deal with the high rate of messages.