search

TheThingsNetwork / lorawan-stack

The Things Stack, an Open Source LoRaWAN Network Server
https://www.thethingsindustries.com/stack/
Apache License 2.0
978 stars 307 forks source link

Resend packets stored in Storage Integration #6723

Open KellyM001 opened 11 months ago

KellyM001 commented 11 months ago

Summary

Support the ability to (re)send packets stored in Storage Integration out all enabled integrations.

Current Situation

If we lose the ability to process uplink packets in real time because our backend system fails for any reason, we have the ability to retrieve those packets from Storage Integration (which is a great resilience feature btw) via an API. However our backend system is designed to process uplink packets from enabled Application integrations only. And so if we wanted to process packets stored in Storage Integration, we would need to design, implement, and maintain other processes to retrieve those packets from Storage Integration and reinject them into our data pipeline.

In our particular situation the data pipeline consists of our MQTT subscribers subscribing to the TTS MQTT broker, and from the MQTT subscriber the packets are duplicated and sent to multiple other locations. If we have an outage in our backend, then to process any packets lost during an outage we would need to:

  1. Install, configure, and maintain our own MQTT broker.
  2. Modify our existing MQTT subscribers to also subscribe to topics in our MQTT broker.
  3. Design and create a process to retrieve the packets from the Storage Integration via it's API and then inject those packets into our MQTT broker.

Why do we need this? Who uses it, and when?

This is purely to minimise the amount of data processing services we need to support in our backend platform. If there was a way we could request packets stored in Storage Integration to be resent via existing Application integrations, then it would save TTI customers having to build their own processes to retrieve and process packets lost during a backend outage.

Proposed Implementation

I'm not familiar with the TTS architecture regarding Applications and integrations, but if an API end point (or a Console page or CLI command) could be provided that accepted a time range to get data from Storage Integration and resend those packets out existing Application integrations that would be extremely useful. Input parameters would be time range, Organisation, Application, and possibly Device (although if there is a backend outage then I expect uplink packets from all devices in the Application would need to be resent).

Contributing

Code of Conduct

KrishnaIyer commented 11 months ago

Thanks for the detailed issue. We will discuss this internally and get back to you on this.