Include gateway identifier in metadata

johanstokking commented 3 years ago

Summary

Include the gateway identifier in metadata.

Why do we need this?

For Home Networks to identify gateways in the Forwarder network.

What is already there? What do you see now?

No field for a gateway identifier in UplinkMessage.

What is missing? What do you want to see?

There should be a field, probably a structure with a friendly name, description and EUI. All optional.

How do you propose to implement this?

Add to the UplinkMessage proto message.

Can you do this yourself and submit a Pull Request?

Yes

tkerby commented 3 years ago

Any updates on this one? It's preventing a few applications from moving to v3 for me.

matthijskooijman commented 3 years ago

I see that the uplink message JSON now indeed includes the gateway id, i.e. the rx_metadata element for the gateway has somthing like:

            "gateway_ids":{
               "gateway_id":"mjs-gateway-5",
               "eui":"1DEF5A0000000202"
            },

However, it seems that gateways that are connected to the v2 backend and connect to v3 using the packet broker (or at least that's what I understand) only have:

            "gateway_ids":{
               "gateway_id":"packetbroker"
            },

Multiple entries with the same id exist when multiple v2 gateways have received the messages.

Are there any plans to also show actual ids for such gateways? Or is this technically infeasible?

johanstokking commented 3 years ago

Are there any plans to also show actual ids for such gateways? Or is this technically infeasible?

It's technically not feasible and I think it's also not desired. The gateawy_id shown in the RX metadata is the gateway in the network that you're connected to. You'll always know which gateway that is.

When networks offload traffic to Packet Broker, they may include the plain gateway ID, its hash or no ID, and/or optionally the EUI. This could overlap with names in the network that gateways are connected to, i.e. there could be a gateway mjs-gateway-5 in your network, but also in my The Things Stack open source that I connect to Packet Broker and also call mjs-gateway-5. The gateway ID packetbroker is there to ensure that the gateway is not a local gateway, that ID is blacklisted so it will not cause any confusion.

Then, for The Things Stack, from the compatibility commitment:

We will not break the API towards gateways and applications within the major version. This includes how gateways communicate (with Gateway Server) and how applications work with data (with Application Server)

So we cannot replace the current behavior as applications already depend on the current behavior. We can only do this in The Things Stack V4.

matthijskooijman commented 3 years ago

Thanks for clarifying, I see the problem. However, I wonder: Wouldn't it be possible to add another id? It already seems there are multiple parts to the id (though I'm not entirely sure what guarantees these id parts have, like if they should all be unique or not?). i.e. something like:

            "gateway_ids":{
               "gateway_id":"packetbroker",
               "original_gateway_id": "mjs-gateway-5"
            },

Edit: Or what @jpmeijers suggests below, putting the id inside the "packetbroker" dictionary would also be fine

jpmeijers commented 3 years ago

The current implementation also does not solve my original problem: uniquely identifying a coverage area. ie. a unique gateway. The combination of tenant+cluster+gateway ID should provide this.

Question: Is forwarder the network to which the gateway is connected to, and home the network the device is registered on?

What I currently see in the metadata:

"rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"packetbroker"
            },
            "packet_broker":{
               "message_id":"01F3MHC7W44N2KR4301XNTSVE5",
               "forwarder_net_id":"000013",
               "forwarder_tenant_id":"ttn",
               "forwarder_cluster_id":"ttn-v2-eu-3",
               "home_network_net_id":"000013",
               "home_network_tenant_id":"ttn",
               "home_network_cluster_id":"ttn-eu1",
               "hops":[

What I want to see:

"rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"packetbroker"
            },
            "packet_broker":{
               "message_id":"01F3MHC7W44N2KR4301XNTSVE5",
               "forwarder_net_id":"000013",
               "forwarder_tenant_id":"ttn",
               "forwarder_cluster_id":"ttn-v2-eu-3",
               "forwarder_gateway_id":"mjs-gateway-5",
               "forwarder_gateway_eui":"1DEF5A0000000202",
               "home_network_net_id":"000013",
               "home_network_tenant_id":"ttn",
               "home_network_cluster_id":"ttn-eu1",
               "hops":[

Also, the way we identify a tenant and cluster here is not consistent with how the same "domain" is identified via the headers:

  'X-TTS-DOMAIN' => 'eu1.cloud.thethings.network',

How about we use the same naming for both?

I don't see how adding this additional metadata will break the API.

johanstokking commented 3 years ago

Question: Is forwarder the network to which the gateway is connected to, and home the network the device is registered on?

Yes

What I want to see:

[...]

This is how it is supposed to be exactly. It's just that the forwarders running at V2 edges aren't updated yet to include the metadata. We are doing that now.

Also, the way we identify a tenant and cluster here is not consistent with how the same "domain" is identified via the headers:
  'X-TTS-DOMAIN' => 'eu1.cloud.thethings.network',
How about we use the same naming for both?

Hmm, what would you propose then?

jpmeijers commented 3 years ago

Hmm, what would you propose then?

Maybe a forwarder_network_domain field which contains something like eu1.cloud.thethings.network? This will make sense for any TTS V3 cluster. Not sure how this will work in the future if we have other network servers talking via the packet broker. But any domain name will make sense and unqiuely identify the forwarder network - even non-TTS ones that does not have the concept of a tenant. Something like "forwarder_network_domain": "ns.lacuna.space" makes sense too.

johanstokking commented 3 years ago

All NetID + tenant ID is unique, and so are gateway IDs and EUIs within those. Tenant IDs are optional. Most networks won't use tenants, like Lacuna probably. We will, however, allow The Things Stack Open Source users to use 000013 NetID with their own tenant ID and we (that is, TTN, the authority of tenants in 000013) are responsible for making sure that tenants IDs within 000013 are unique.

Domain names aren't adding to uniqueness. It's just another value with another meaning, and Packet Broker doesn't use domain names for authenticating networks, while it does use NetID + tenant ID so those values are guaranteed to be correct. If we allow Packet Broker Agent to report the locally configured domain name, that can also be a local name like thethings.local in an on-premises network in a printer shop, with no internet accessible Console, while they are a client for Packet Broker.

jpmeijers commented 3 years ago

Getting a little off-topic with this, but not sure where else to discuss this.

When we have this metadata in the json object:

               "forwarder_net_id":"000013",
               "forwarder_tenant_id":"ttn",
               "forwarder_cluster_id":"ttn-v2-eu-3",
               "home_network_net_id":"000013",
               "home_network_tenant_id":"ttn",
               "home_network_cluster_id":"ttn-eu1",

And we get this header at the same time as the json object via the webhook call:

'X-TTS-DOMAIN' => 'eu1.cloud.thethings.network',

How would one use the forwarder_tenant_id and the forwarder_cluster_id to match it up with a X-TTS-DOMAIN? Is a simple concatenation like this good enough?

forwarder_tts_domain := forwarder_tenant_id + "." + forwarder_cluster_id
home_tts_domain := home_network_tenant_id + "." + home_network_cluster_id

// Just as example, as doing this test doesn't make sense
if tts_domain == headers["X-TTS-DOMAIN"] {
  // Gateway is in same network as device
}

// Expecting this to be true
if tts_domain == headers["X-TTS-DOMAIN"] {
  // Device network should be home network
}

But ttn.ttn-eu1 is not equal to eu1.cloud.thethings.network!

johanstokking commented 3 years ago

What are you trying to achieve?

Strictly speaking you (only) can assume the following:

X-Tts-Domain suffix .cloud.thethings.network is The Things Stack Community Edition (The Things Network)
X-Tts-Domain suffix .cloud.thethings.industries is The Things Stack Cloud (The Things Industries)
home_network_net_id 000013 and home_network_tenant_id ttn is The Things Network
home_network_net_id 000013 with any other home_network_tenant_id can be anything: The Things Stack Cloud, The Things Stack Enterprise, The Things Stack Open Source, even ChirpStack using Passive Roaming and using address space of TTN

If you want to determine whether the message was picked up by a community network gateway, check the packetbroker gateway RX metadata and consider (3) above.

If you want to determine whether a user configured the TTN Mapper integration from TTN or TTSC, check the X-Tts-Domain suffix.

Combining the two: if you want to determine whether the message was picked up by community gateways and delivered to you by the community network, check both RX metadata and X-Tts-Domain.

The X-Tts-Domain request header is the originating public domain of the calling The Things Stack network and is intended for the application layer to enqueue downlink messages. That's it.

Also, home_network_cluster_id is informative only and we don't guarantee any stable relationship between that and the domain name.

jpmeijers commented 3 years ago

Scenario 1: Device on private network is received by public community network gateways. TTS-Mapper-ttn-to-ttc

Scenario 2: Device on public community network is received by a private network's gateway. TTS-Mapper-ttc-to-ttn

Note: I see the different TTN clusters as different networks. Should I see them as a single network? For me the Australia cluster is a completely standalone network, separate form the EU cluster.

Another note: I could not test the second scenario properly, as it seems like Public => Private works, but the exact opposite forwarding via the Packet Broker, Private => Public does not work.

johanstokking commented 3 years ago

Note: I see the different TTN clusters as different networks. Should I see them as a single network? For me the Australia cluster is a completely standalone network, separate form the EU cluster.

They are the same. With V3, au1.cloud.thethings.network is operated by TTI and is like any other TTN region.

Need to deduce that the gateway is connected to the-box.cloud.thethings.industries by using only these fields.

You get the 000013 and the-box. Those are guaranteed to be unique. Why do you need the host name?

Another note: I could not test the second scenario properly, as it seems like Public => Private works, but the exact opposite forwarding via the Packet Broker, Private => Public does not work.

By default private networks are not contributing to TTN. We're working on Console support to configure this (https://github.com/TheThingsNetwork/lorawan-stack/issues/3771). For now, use CLI to configure a routing policy from your private network to TTN: https://www.thethingsindustries.com/docs/reference/packet-broker/configure/routing-policies/

jpmeijers commented 3 years ago

They are the same. With V3, au1.cloud.thethings.network is operated by TTI and is like any other TTN region.

For the first time tonight I logged into the au1 console, and I was surprised to see all my applications listed. I thought these were completely separate. But I now agree that they can be seen as the same network.

Using the 000013 and tenant_id to identify a network makes sense from the LoRaWAN Alliance perspective.

The problem however here is that the _net_id and the tenant_id are not reported in either the json object or the headers_ when a packet is received directly (not via packet broker). See attached log. So if we need to use the net_id+tenant_id to identify a unique network, that info should always be available in the metadata. And in that case you are right that the domain name is redundant. But at this point the X-Tts-Domain/MQTT host is the only always available identifier.

TTS sample webhook data.txt

Private networks contributing to TTN: some of the TTI customers specifically want to do this to allow community members to map their networks for them. This is the main reason I need to identify the forwarding network for packet broker gateways.

johanstokking commented 3 years ago

So if we need to use the net_id+tenant_id to identify a unique network, that info should always be available in the metadata. And in that case you are right that the domain name is redundant. But at this point the X-Tts-Domain/MQTT host is the only always available identifier.

I see. Filed issue here: https://github.com/TheThingsNetwork/lorawan-stack/issues/4076

matthijskooijman commented 3 years ago

I'm still missing gateway ids for ttnv2 gateways in the MQTT data. @johanstokking previously said:

This is how it is supposed to be exactly. It's just that the forwarders running at V2 edges aren't updated yet to include the metadata. We are doing that now.

Is this being tracked anywhere?

jpmeijers commented 3 years ago

@johanstokking can we open this issue again? As far as I can see the required data is still nor available in the metadata.

lesinigo commented 3 years ago

You probably already know this, but the same also applies to data received with the web hook integration:

{
    "uplink_message": {
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "packetbroker"
                },
                "packet_broker": {
                    "message_id": "xxxxxxxxxxxxxxxxxx",
                    "forwarder_net_id": "000013",
                    "forwarder_tenant_id": "ttn",
                    "forwarder_cluster_id": "ttn-v2-eu-4",
                    "home_network_net_id": "000013",
                    "home_network_tenant_id": "ttn",
                    "home_network_cluster_id": "ttn-eu1",

... and no gateway EUI anywhere.

johanstokking commented 3 years ago

This is probably a configuration issue. This is not a Packet Broker API issue so this issue can stay closed.

We're looking into it and and get this fixed.

cc @KrishnaIyer @michalborkowski96

packetbroker / api