Expose NetID and TenantID on user facing API

jpmeijers commented 2 years ago

Summary

I see something similar has already been added for the internal APIs: https://github.com/TheThingsNetwork/lorawan-stack/pull/4726

A Gateway ID is unique per Tenant ID per ... Identity Server - or I just say per NetID.

Currently one can use a Tenant ID and API Key to talk to https://.eu1.cloud.thethings.industries/api/v3/. There one can obtain a list of gateways. Because the request is done using the Tenant ID, we already know the tenant under which these gateway IDs are unique.

There is however no API to obtain the NetID.

Of course one can make the assumption that cloud.thethings.industries is always NetID 000013, but it would be better if one can confirm this via an API call.

Why do we need this?

Confirm which NetID is being used by a tenant.

What is already there? What do you see now?

...

What is missing? What do you want to see?

An API endpoint to provide the NetID, TenantID, and maybe the Cluster ID if that makes sense.

Environment

TTS v3.16.2

How do you propose to implement this?

...

How do you propose to test this?

...

Can you do this yourself and submit a Pull Request?

No

KrishnaIyer commented 2 years ago

@jpmeijers: We'd like to know what is the purpose of exposing the Tenant ID/Net ID? What would you do with that if it's exposed?

jpmeijers commented 2 years ago

Yes, it is indeed a very specific scenario I have. Let me try and explain.

I have one big database with measurements that came in from different TTS instances. Data is tagged with the Tenant ID and NetID of the device that sent the message, as well as the gateway, and the gateway's Tenant ID and NetID. This info is available in the standard uplink message json.

I could have also used the ClusterID, but there's still confusion if a TenantID on one cluster is a unique network from the same TenanID on another cluster. So I decided not to use the ClusterID.

Example:

{
   "end_device_ids":{
      "device_id":"cricket-002",
      "application_ids":{
         "application_id":"jpm-crickets"
      },
   },
   "uplink_message":{
      "rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"eui-60c5a8fffe71a964",
               "eui":"60C5A8FFFE71A964"
            },
         },
         {
            "gateway_ids":{
               "gateway_id":"packetbroker"
            },
            "packet_broker":{
               "forwarder_net_id":"000013",
               "forwarder_tenant_id":"ttnv2",
               "forwarder_cluster_id":"ttn-v2-eu-3",
               "forwarder_gateway_id":"eui-647fdafffe007a1f",
            },
         }
      ],
      "network_ids":{
         "net_id":"000013",
         "tenant_id":"ttn",
         "cluster_id":"ttn-eu1"
      }
   }
}

Device: 000013/ttn/jpm-crickets/cricket-002 (NetID/TenantID/AppID/DevID) Gateway 1: 000013/ttn/eui-60c5a8fffe71a964 (NetID/TenantID/GatewayID) Gateway 2: 000013/ttnv2/eui-647fdafffe007a1f

This scheme allows me to uniquely identify any device or gateway on any TTS instance (or any LoRaWAN network), and allows me to use all metadata obtained via the Packet Broker.

I also keep the last heard time of each gateway to know if it is online or not. Gateway statuses are obtained from three sources:

Uplink messages - if a messages is received via a gateway, that specific gateway is online.
Packet Broker mapper api.
TTS API using Api Key (/api/v3/gateways) - for private gateway or networks that are not connected to the PB.

The first two of these identify gateways with all the IDs including the TenantID and NetID.

The TTS API only identifies gateway using the GatewayID. https://www.thethingsindustries.com/docs/reference/api/gateway/#message:Gateway

So what I need is a way to know that the gateway info obtained via the API belongs to a specific Gateway that was seen via an uplink message. The TenantID is provided to me along with the API Key to know which endpoint to talk to, so I could use this for the TenantID part. But the NetID is still missing.

I see there is an API to get a DevAddr (/api/v3/ns/dev_addr), but this requires extra permissions, and I'm concerned that I might be causing internal NS confusion when I let it generate unused DevAddrs.

adriansmares commented 2 years ago

The NetID is not a better separator than the cluster ID in the context of the comment above - keep in mind that both The Things Stack Cloud and The Things Stack Community Edition share the same NetID. To make matters worse, you may see non TTSC/TTSCE deployments that use that NetID.

We added at some point in the past (https://github.com/TheThingsNetwork/lorawan-stack/pull/5408) extra cluster identifiers - the cluster address and the tenant address:

      "network_ids": {
        "net_id": "000013",
        "tenant_id": "tti",
        "cluster_id": "staging1",
        "cluster_address": "staging1.cloud.thethings.industries",
        "tenant_address": "tti.staging1.cloud.thethings.industries"
      }

The addresses are real FQDN, so as long as you trust DNS to be authoritative, they should really serve as globally unique identifiers (and just discard any .local TLD I suppose).

jpmeijers commented 2 years ago

Two issues I have with this:

The same tenant on multiple clusters is considered the same network. Ex jpmeijers.eu1.cloud.thethings.industries and jpmeijers.au1.cloud.thethings.industries is the same network. When one logs into the two consoles you see the same list of applications, devices and gateways. That's why it's the same network. I therefore have to identify a network based on the NetID, just like the PacketBroker does.
TTN Mapper had to make a choice when V3 launched, and changing the way networks are identified is either impossible or will mean a lot of migrations need to be written. For now we stick to TenantId@NetId.

We added at some point in the past (https://github.com/TheThingsNetwork/lorawan-stack/pull/5408) extra cluster identifiers

Is this exposed on an API somewhere?

TheThingsNetwork / lorawan-stack