Down frequency gets corrupted between MQTT → Semtech UDP packets

lukecyca commented 7 months ago

Some background to my issue and troubleshooting is located in this forum post: https://forum.chirpstack.io/t/dragino-lps8n-with-mqtt-forwarder-uses-wrong-downlink-frequency

I believe I've isolated this to the chirpstack-mqtt-forwarder. The frequency specified by the LNS is modified when passed on to the gateway.

Here is a JOIN_ACCEPT MQTT packet from the LNS:

{
  "phyPayload": "IOjGEhU69v5h54skpaMbdSROCQVfcyzw+AKLl5fsZM8t",
  "txInfo": {
    "frequency": 923300000,
    "power": 21,
    "modulation": {
      "lora": {
        "bandwidth": 500000,
        "spreadingFactor": 12,
        "codeRate": "CR_4_5",
        "polarizationInversion": true
      }
    },
    "timing": {
      "delay": {
        "delay": "6s"
      }
    },
    "context": "PQcQKQ=="
  }
}

And the corresponding semtech_udp TxPk:

{
  "imme": false,
  "rfch": 0,
  "powe": 21,
  "ant": 0,
  "brd": 0,
  "tmst": 1028873064,
  "tmms": null,
  "freq": 268.435632,
  "modu": "LORA",
  "datr": "SF10BW500",
  "codr": "4/5",
  "fdev": null,
  "ncrc": null,
  "ipol": true,
  "prea": null,
  "size": 33,
  "data": "IOjGEhU69v5h54skpaMbdSROCQVfcyzw+AKLl5fsZM8t"
}

Notice that the frequency is correct in the MQTT packet, and incorrect in the Semtech UDP packet.

It's always 268.abcdef where the decimals change a bit each time.

Version: chirpstack-mqtt-forwarder 4.1.3

Here is my config:

[logging]
  level="info"
  log_to_syslog=false

[backend]
  enabled="semtech_udp"

  [backend.semtech_udp]
    udp_bind="0.0.0.0:1700"

[mqtt]
  topic_prefix="us915_0"
  json=true
  server="wss://mqtt.test.example.com/mqtt"
  username="redacted"
  password="redacted"
  ca_cert="/etc/ssl/cert.pem"
  tls_cert=""
  tls_key=""

brocaar commented 7 months ago

That is strange indeed... Two questions:

What is the Dragino firmware version that you are using?
Could you capture the UDP data using tcpdump (see https://www.chirpstack.io/docs/guides/connect-gateway.html#tcpdump)? Just to confirm if the MQTT forwarder sends the wrong data, or if the UDP interprets the JSON payload incorrectly.

That would be really helpful for debugging this issue further.

brocaar commented 7 months ago

An additional question, do have the same issue with json=false in your configuration?

lukecyca commented 7 months ago

Thanks for your response!

Here is the output of tcpdump:

E..>>C@.@..i.............*.=.....@A..'.8{"rxpk":[{"jver":1,"tmst":351666419,"time":"2024-02-16T22:59:10.554631Z","chan":7,"rfch":1,"freq":903.700000,"mid": 8,"stat":1,"modu":"LORA","datr":"SF10BW125","codr":"4/5","rssis":-109,"lsnr":-16.2,"foff":-10653,"rssi":-96,"size":23,"data":"AAEBAAAAQUComT+J0XxBQKgV26fGGAw="}]}
14:59:10.558335 IP localhost.1700 > localhost.52744: UDP, length 4
E.. >D@.@.......................
14:59:10.914613 IP localhost.1700 > localhost.44932: UDP, length 267
E..'>R@.@..q...............&....{"txpk":{"imme":false,"rfch":0,"powe":21,"ant":0,"brd":0,"tmst":356666416,"tmms":null,"freq":268.435568,"modu":"LORA","datr":"SF10BW500","codr":"4/5","fdev":null,"ncrc":null,"ipol":true,"prea":null,"size":33,"data":"IPD9e+P9+GKQh1/rVz4uB4n0U74WsQhS/RFj6FS+BuTG"}}
14:59:10.916055 IP localhost.52744 > localhost.1700: UDP, length 44
E..H>S@.@..O.............4.G.....@A..'.8{"txpk_ack":{"error":"TX_FREQ"}}
14:59:10.929708 IP localhost.1700 > localhost.44932: UDP, length 267
E..'>T@.@..o...............&....{"txpk":{"imme":false,"rfch":0,"powe":21,"ant":0,"brd":0,"tmst":357666416,"tmms":null,"freq":268.435536,"modu":"LORA","datr":"SF12BW500","codr":"4/5","fdev":null,"ncrc":null,"ipol":true,"prea":null,"size":33,"data":"IPD9e+P9+GKQh1/rVz4uB4n0U74WsQhS/RFj6FS+BuTG"}}
14:59:10.995979 IP localhost.52744 > localhost.1700: UDP, length 44
E..H>Z@.@..H.............4.G.....@A..'.8{"txpk_ack":{"error":"TX_FREQ"}}
14:59:14.143265 IP localhost.44932 > localhost.1700: UDP, length 12

That suggests that the incorrect frequency is coming from chirpstack-mqtt-forwarder's side of the UDP connection, as opposed to it being misinterpreted on the gateway's side.

What is the Dragino firmware version that you are using?

The above traces have all been with lgw-openvpn-5.4.1699323660 however I had the same symptoms with v5.4.1704801796.

do have the same issue with json=false in your configuration?

Yes, I had the same symptoms prior to enabling json=true for troubleshooting.

Two other clues that may be relevant:

I am getting a lot of UPLINK_MIC Errors, and manual inspection of the MIC bytes show they differ by only a single bit in one of the bytes when comparing the payload using tcpdump, and the values shown in Chirpstack's LoRaWAN frames UI. This happens in about 4/5 packets.
I also just tried setting up this gateway and node on TTN using the Semtech UDP protocol, and it worked without issue.

brocaar commented 7 months ago

Thanks, that is very helpful. Note that there is also something odd with the tmst. The difference between RX and TX should always be a multiple of 1_000_000 (x number of seconds), which is also not the case. I will try to reproduce this issue on one of my Dragino gateways next week.

lukecyca commented 6 months ago

I've switched to the chirpstack-gateway-bridge (running on a server, not the gateway) and everything is working for me now. I think I'll continue with this architecture as it suits my use case just fine.

I'll leave this issue open. Just wanted to let you know that it's no longer impacting me.

brocaar commented 6 months ago

Just to let you know, I am able to reproduce your MIC issue on my Dragino gateway as well. This issue seems to be limited to the Dragino package. I have tested the same version on one of my Kerlink gateways, and there it worked as expected. I suspect something went wrong during the cross-compiling for the Dragino architecture of the latest version(s).

brocaar commented 6 months ago

Compiling the same code, but with the latest Rust toolchain (nightly, as Rust has moved MIPS support to tier 3) no longer results in the MIC error and it seems the data is no longer corrupted. I have created a test-build for you:

https://www.dropbox.com/scl/fo/jtqxrcz9ly24t6gg903gw/h?rlkey=jjqnh15los7av781zas7vnqt0&dl=0

Recommended way to upgrade:

opkg remove chirpstack-mqtt-forwarder (remove old version)
reboot
copy the .ipk from dropbox onto your gateway (/tmp folder)
opkg install ...

Else you might run into errors related to the limited amount of storage that is available.

lukecyca commented 6 months ago

Thanks. I got this installed, but this test build doesn't seem to have support for the wss:// mqtt uri scheme. The previous build did (release v4.1.3).

Is it much trouble to enable that in your test build? That's what I've got set up in my test environment. If it's more than 15min of trouble for you, let me know and I'll just stand up a different MQTT server instead.

brocaar commented 6 months ago

You are right, the websocket feature flag is missing, I'm now looking into this.

brocaar commented 6 months ago

This adds websocket support: https://github.com/chirpstack/chirpstack-mqtt-forwarder/commit/351091f28647b8d0b5f28b0f8615a2c1be8323eb.

(I have tested this with both plaintext as websockets over TLS)

I will do a some more testing and if all looks fine, I will create a new release.

brocaar commented 6 months ago

This was released as v4.2.0.

chirpstack / chirpstack-mqtt-forwarder

Down frequency gets corrupted between MQTT → Semtech UDP packets #50