[Proposal] Add a generic device-based addressing scheme for "WAN" networks

About

Based on my thoughts shared at GH-132, I am trying to generalize a few new requirements around data acquisition from TTN into another design principle of Kotori. On this matter, I am thinking about finally adding a device-based addressing/endpoint scheme, additionally to the channel-based approach Kotori is offering from the very beginning.

Please note while referring to TTN and HTTP within this proposal, the functionality is meant to be generally available, also for MQTT-based data acquisition.

Relation to the TTN integration

In this way, we will have a more generic foundation to build upon, and can specialize it by adding specific concerns around TTN on top of it. From a different perspective, the TTN WAN support feature would not take any significant detour from what Kotori will be offering anyway.

Requirements recap

We identified two more sensible variants to submit measurement/metric/telemetry data to Kotori.

Address information (device identifier) is encoded into the URL, but not using classic "URL path"-based semantics.
Address information (device identifier) is encoded into the message payload, in whatever format that may be.

Proposal

So, my proposal to implement those variants would be to introduce a new endpoint infix, but not special to TTN, which would just be /d, which means "device". If this would be followed by a device identifier, the device identifier would be picked from the URL. When there is no device identifier, and the data will be submitted straight to the realm address, the machinery will attempt to decode it from the message payload.

Device identifier in URL

http:/api/{realm:mqttkit-1}/d/{device_id:.*}/{slot:data}

realm  ....^^
special identifier "d" .....^
device identifier .............^^
well-known slot identifier "data" ............^^

Example:

# HTTP variant
https://daq.example.org/api/mqttkit-1/d/itest-foo-bar/data

# MQTT variant
mqttkit-1/d/itest-foo-bar/data.json

Device identifier in message payload

http:/api/{realm:mqttkit-1}/d/{slot:data}

realm  ....^^
special identifier "d" .....^
well-known slot identifier ....^^

Example:

# HTTP variant
https://daq.example.org/api/mqttkit-1/data

# MQTT variant
mqttkit-1/data.json

Feature request

The original feature request from @thiasB is:

My idea is that the topic should be derived from the TTN devID via replace(/-/g, '/') + "/data". It means that the devID must have 4 components that correspond to the MQTT topic levels, separated by the - character instead of the / character, which is disallowed on that level. This has the advantage that you can put all your devices in the same TTN application, and still have control over the entire topic.

-- https://community.hiveeyes.org/t/tts-ttn-daten-an-kotori-weiterleiten/1422/35

As it is only a proposal yet, I will be happy to take your thoughts into consideration.

/cc @einsiedlerkrebs, @thiasB, @u-l-m-i, @ClemensGruber, @wetterfrosch, @MKO1640

Thoughts about channel isolation and authentication

Problem

We will have to take care that this implementation does not accidentally weaken Kotori's security concept, which is mostly based around limiting access to certain topics by path prefix, and mapping that to channel user/owner ACLs in Mosquitto, which are reflected on the second path level of the MQTT topic, after the realm component, which is mqttkit-1 in the example below.

You will recognize that it is not possible to apply an ACL rule on the MQTT topic mqttkit-1/d/owner-*, as wildcards are usually applied to MQTT topics only on the path segment level, and not mid-segment. ^1

If we don't apply any countermeasures here, direct-device-addressing will open the door to access the write path to anyone else's channels, even if they would be protected by a MQTT full-channel ACL.

Example

Those examples demonstrate the MQTT topics for the same data channel, while using different addressing schematics, classical channel-based vs. the new device-based scheme.

# Channel-based MQTT WAN addressing.
mqttkit-1/owner/foo/bar/data.json

# Device-based MQTT WAN addressing.
mqttkit-1/d/owner-foo-bar/data.json

A typical Mosquitto ACL item currently looks like this.

user <username>
topic readwrite mqttkit-1/owner/#

Solutions

I see two possible solutions.

When enabling this feature in the corresponding Kotori .ini file, the operator would need to permit it for specific owners only, like
```
[channel]
permit_device_addressing = ["acme", "peter"]
```
We may allow to explicitly configure *, in order to tear down that wall completely, but otherwise, allowing no user to use direct-addressing, would be a sane choice by default.
Adjust the specification. This effectively shrinks the headroom to two path components. It feels a bit cleaner, but I don't know if it is possible to implement. Also, it would defy @thiasB idea to run the application for the whole hiveeyes realm.
```
# Device-based MQTT WAN addressing.
mqttkit-1/owner/foo-bar/data.json
```

Conclusion

Currently, for @thiasB use case, I see no other choice than 1., adding a permit_device_addressing configuration option. While it is sad that the feature would not be completely maintenance-free then, I see no other chance to hold up the minimal amount of channel isolation/protection Kotori offers.

Please do not hesitate to share any ideas you might have about this detail. Thanks.

daq-tools / kotori