Report gateway network - Githubissues

jpmeijers commented 4 years ago

Summary

When I receive a json object from the webhooks integration I would like to see which networks the gateways belong to that received the packet. Maybe report a "namespace" field per gateway. Currently I believe "packet-broker" is used as gateway ID when data is received by another network.

Why do we need this?

Gateway IDs are only unique per network. We need a way to distinguish between gateways on different networks. Personally for me I'd like to collect statistics per gateway, but this can also be useful to "blacklist" unreliable gateways.

What is already there? What do you see now?

On my own network I see Gateway ID, Gateway EUI and Antenna Index. This allows me to uniquely identify a specific piece of hardware and antenna that received the packet. I am unsure what is reported when data is received via the packet broker as my private test network is not connected to the packet broker.

What is missing? What do you want to see?

I would like to have an indication if the packet is received by a gateway on my network (maybe TTN) or by a gateway on a different network. An indication of the network would be nice, even if it is just a hash. This will allow me to know that the gateway that received the packet is not on TTN but on a private network. Ideally I want to see the Network ID, Gateway ID and Antenna Index of every gateway that received the packet. This will allow me to uniquely identify any gateway.

Environment

...

How do you propose to implement this?

...

Can you do this yourself and submit a Pull Request?

No

jpmeijers commented 4 years ago

This feature request can also be interpreted as a Documentation Request. Currently it is unclear what would happen if a packet is received by another network and forwarded to my network. While writing a webservice to handle webhook integration requests I need to be able to handle this case. It would be better to handle it now during the architecture design, rather than to have to add it to a production system later.

johanstokking commented 4 years ago

We changed the way Packet Broker metadata is reported since The Things Stack 3.8. The gateway ID will still be packet-broker, but there will be metadata from Packet Broker included as well.

Still, this does not include a (pointer to) a gateway. It could definitely be beneficial – please file an issue in https://github.com/packetbroker/api as it needs an API change there.

For mapping specifically, we have mapping and mapping services API prepared. We don't have an implementation yet; I can imagine that TTN Mapper will implement these services and networks can report gateway status, location and visibility that way, instead of via traffic.

johanstokking commented 4 years ago

Since this issue is requesting specifically the originating network of a packet, this issue can technically be closed as this is now in 3.8 already. TTI Cloud Hosted will be updated shortly to make this effective. Please update or close the issue if there's anything else needed at the moment.

jpmeijers commented 4 years ago

Two different issues here.

Mapping: I like the way we are going with the Packet Broker API's you linked. At some point we should standardise what we are going to use. I don't have the resources to change the mapper every time there is a change on TTN's side.
Reporting originating network: Good to hear this is already in, and this is the actual topic of this issue. I don't have a Things Stack instance that connects to the packet broker, so I do not have a way to see this in action. Could you provide an example of what the JSON looks like that a webhook endpoint will receive?

Currently I receive this from a v3.8.1 stack:

{
   "end_device_ids":{
      "device_id":"cricket-001",
      "application_ids":{
         "application_id":"jpm-crickets"
      },
      "dev_addr":"<redacted>"
   },
   "correlation_ids":[
      "as:up:01E853V3KAEV9SDE37PCPHD1XD",
      "gs:conn:01E853PP0QZD3SNGXNABQMWMR5",
      "gs:uplink:01E853V3BPY5KCJB328NRQE7XS",
      "ns:uplink:01E853V3BQYR58M8KFT89SHHHG",
      "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01E853V3BQJDKVH13Q2V97C7HK"
   ],
   "received_at":"2020-05-12T18:55:05.834106433Z",
   "uplink_message":{
      "f_port":1,
      "f_cnt":56643,
      "frm_payload":"<redacted>",
      "decoded_payload":{
         "gps_0":{
            "altitude":128.3800048828125,
            "latitude":-33.<redacted>,
            "longitude":18.<redacted>
         }
      },
      "rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"pisupply-shield",
               "eui":"B827EBFFFED88375"
            },
            "timestamp":147142404,
            "rssi":-61,
            "channel_rssi":-61,
            "snr":10.2,
            "uplink_token":"Ch0KGwoPcGlzdXBwbHktc2hpZWxkEgi4J+v//tiDdRCE7pRG",
            "channel_index":7
         }
      ],
      "settings":{
         "data_rate":{
            "lora":{
               "bandwidth":125000,
               "spreading_factor":7
            }
         },
         "data_rate_index":5,
         "coding_rate":"4/5",
         "frequency":"867900000",
         "timestamp":147142404
      },
      "received_at":"2020-05-12T18:55:05.591560247Z"
   }
}

johanstokking commented 4 years ago

Mapping: I like the way we are going with the Packet Broker API's you linked. At some point we should standardise what we are going to use. I don't have the resources to change the mapper every time there is a change on TTN's side.

There are two stable APIs: The Things Stack V3 (JSON and gRPC) for LoRaWAN networking and applications, and Packet Broker API for transport and infrastructure.

These APIs are complementary. Packet Broker mapping services can be used to get gateway locations and their visibility, regardless of traffic and link performance. We'll use it to show a map of all gateways, including TTN's public gateways and TTI customer networks who want to share their gateway locations for peering. Link performance, essentially the core of TTN Mapper, is not part of Packet Broker mapping services; you still use The Things Stack APIs for that, but you could show gateways without needing to discover them via traffic.

Reporting originating network: Good to hear this is already in, and this is the actual topic of this issue. I don't have a Things Stack instance that connects to the packet broker, so I do not have a way to see this in action. Could you provide an example of what the JSON looks like that a webhook endpoint will receive?

I'll get back on this shortly.

jpmeijers commented 4 years ago

Just a small comment.

but you could show gateways without needing to discover them via traffic.

I have never discovered gateways using traffic. It is however something I am currently considering as people complain that their gateways take too long to show up on the map.

The important thing is to have the ability to associate traffic coming in with some kind of unique gateway identifier. And using this identifier the gateway details can be requested from the Packet Broker API. I'm just concerned that the identifier I currently use (gateway ID) is not unique.

johanstokking commented 4 years ago

I have never discovered gateways using traffic. It is however something I am currently considering as people complain that their gateways take too long to show up on the map.

I meant gateway locations reported in gateway metadata on uplink traffic, regardless of the ones showing up on the map, right?

The important thing is to have the ability to associate traffic coming in with some kind of unique gateway identifier. And using this identifier the gateway details can be requested from the Packet Broker API. I'm just concerned that the identifier I currently use (gateway ID) is not unique.

Within a tenant ID they will be unique. But TTI/TTN cannot guarantee that open source The Things Stack or self hosted The Things Enterprise Stack implementations do not use conflicting tenant IDs. We will, however, ensure that TTN, TTI Cloud Hosted and TTI Dedicated Cloud Hosted is unique, and we promise best effort to keep TTI Self Hosted unique too.

pe1mew commented 3 years ago

For operating the community gateways it is mandatory to monitor each individual community gateway. This is done with a specific application where a node is sending data to the network. In V2 we can identify each individual gateway as part of the metadata that is provided and monitor operation. With V3 this information is lost.

As a result of this we cannot monitor our community gateways anymore with our own application. For our community is mandatory to have this gateway information available.

johanstokking commented 3 years ago

For operating the community gateways it is mandatory to monitor each individual community gateway. This is done with a specific application where a node is sending data to the network. In V2 we can identify each individual gateway as part of the metadata that is provided and monitor operation. With V3 this information is lost.

As a result of this we cannot monitor our community gateways anymore with our own application. For our community is mandatory to have this gateway information available.

OK. So what do you need to have here; the gateway ID as known to the forwarding network?

pe1mew commented 3 years ago

For our community monitoring, we require gateway ID or gateway EUI. For TTNMapper additional the gateway location in latitude, longitude is mandatory. This may be clarified or commented by @jpmeijers

jpmeijers commented 3 years ago

Latitude and Longitude could potentially be looked up elsewhere, so for me the Gateway ID and/or Gateway EUI as specified on the forwarding network would be great.

I currently see a tenant field in the packet broker metadata. I assume we can use this to identify the forwarding network.

johanstokking commented 3 years ago

For our community monitoring, we require gateway ID or gateway EUI.

It will be in the namespace of the forwarder (NetID + tenant ID), but yes, we can include this.

The reason why we didn't do that so far, is that forwarders may not want to expose gateway IDs publicly, as they may reveal exact locations, customer names, etc. So I guess we need to make this optional. We'll come up with something for the mapping features.

I currently see a tenant field in the packet broker metadata. I assume we can use this to identify the forwarding network.

Yes. TTN community will be NetID 000013 and tenant ID ttn.

TheThingsNetwork / lorawan-stack

Report gateway network #2507

Summary

Why do we need this?

What is already there? What do you see now?

What is missing? What do you want to see?

Environment

How do you propose to implement this?

Can you do this yourself and submit a Pull Request?