LoRaWAN retry packets with wrong timestamp being forwarded to Habhub

[MedadRufus]

When flying over areas with lots of LoRaWAN gateways, some TTN gateways seem to send the data later than other gateways to the MQTT broker. These late packets are have the attribute "is_retry":true. This is a problem because these have a later timestamp than the original message and they all get sent to habhub with the wrong timestamp.

For example in the image below, packet 301 is reported to arrive at 3 different times: 14:30:59,14:31:00 and 14:31:04. Only the first is correct.

Therefore, when calculating speed of the balloon, due to the wrong timestamps, the speed readings are incorrect.

I suggest 2 solutions.

1. filter out all MQTT packets that contain the "is_retry":true attribute. Dont use the data from these retry packets. It is likely that the data contained within already came in a packet earlier A packet with this retry attribute looks like this.

{
  "app_id": "icss_lora_tracker",
  "dev_id": "icspace22",
  "hardware_serial": "00B76CF36C7CFA8A",
  "port": 99,
  "counter": 286,
  "is_retry": true,
  "payload_raw": "AHMJZwFn/4UCiAe5pgDdPxA6TAMCJyYEARM=",
  "payload_fields": {
    "analog_in_3": 100.22,
    "barometric_pressure_0": 240.7,
    "digital_out_4": 19,
    "gps_2": {
      "altitude": 10635,
      "latitude": 50.6278,
      "longitude": 5.6639
    },
    "temperature_1": -12.3
  },
  "metadata": {
    "time": "2020-10-19T13:03:31.730922239Z",
    "frequency": 868.3,
    "modulation": "LORA",
    "data_rate": "SF8BW125",
    "airtime": 154112000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-fcc23dfffe0f4c7b",
        "timestamp": 2089330388,
        "time": "2020-10-19T13:03:31.096209Z",
        "channel": 1,
        "rssi": -111,
        "snr": -4.8,
        "rf_chain": 0
      }
    ]
  }
}

Conversely, a packet that contains the correct timestamp and not the retry attribute looks like this:

{
  "app_id": "icss_lora_tracker",
  "dev_id": "icspace22",
  "hardware_serial": "00B76CF36C7CFA8A",
  "port": 99,
  "counter": 287,
  "payload_raw": "AHMJhwFn/3QCiAe4mwDfGhA0DAMCKCkEARI=",
  "payload_fields": {
    "analog_in_3": 102.81,
    "barometric_pressure_0": 243.9,
    "digital_out_4": 18,
    "gps_2": {
      "altitude": 10619,
      "latitude": 50.6011,
      "longitude": 5.7114
    },
    "temperature_1": -14
  },
  "metadata": {
    "time": "2020-10-19T13:05:18.016203852Z",
    "frequency": 868.5,
    "modulation": "LORA",
    "data_rate": "SF8BW125",
    "airtime": 154112000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-689e19fffe8e615b",
        "timestamp": 2149128972,
        "time": "2020-08-23T11:40:30.367695Z",
        "channel": 2,
        "rssi": -108,
        "snr": -3.8,
        "rf_chain": 0
      }
    ]
  }
}

2. The second solution is to use a timestamp from the packet data from the balloon itself. Most balloon trackers have a RTC or a GPS that can tell it the real time. This can be send down in the packet to give an unambiguous timestamp. Relying on the gateway to give the correct timestamp is insufficient because I have seen rogue gateways give the wrong timestamp altogether. The ttnhabhub bridge already has options to parse the timestamp with the SodaqOne raw packet parser.

[bertrik]

I completely agree with the first point, let's change it to ignore the "is_retry" packets.

Regarding the second point, isn't it better to use the TTN network time? I would say that the TTN time (in JSON: metadata.time) is probably the most reliable, not influenced by individual gateway owners, is complete in the sense that is has both a date and a time-of-day (the payload only sends time, not date) and is not affected by things like GPS issues on the payload, low battery on the payload, RTC drift on the payload, etc.

What do you think?

[MedadRufus]

Strictly speaking, option 1 is not the best because it discards some gateway metadata that gets displayed/logged by habhub, namely the gateway id. Better will be to send the retry data with a corrected timestamp to habhub. But to be honest, I don't think it is worth preserving.

I need to check if TTN time (in JSON: metadata.time) is correct, even for retry messages. I have extensive logs of all data from ICSPACE22 launched today. I have attached them in this post. I will look through them. I have also attached it in this post

mqtt_log_data-1-13.txt

I have seen some packets on the TTN console with a wrong timestamp, but with no "is_retry":true attribute. It may be a seperate issue, with rogue TTN gateways.

[MedadRufus]

I find that parasitic retry messages have the wrong metadata.time timestamp. It is simply easier to discard all retry messages. And then use the data and metadata.time timestamp from the first message exclusively.

[bertrik]

I committed a fix for this, and used your examples in a basic unit test, thanks!

[MedadRufus]

Thanks @bertrik. I will run the latest code tomorrow. But this time I don't think we will have the retry packet issue; There are far fewer gateways over the Mediterranean where ICSPACE22 will be flying!

Very happy to contribute to this project, albeit in a very simple way with log data.

Regards Medad