gatway-bridge suddenly stops accepting packets from semtech udp

[trauter]

Is this a bug or a feature request?

Bug

What happened?

We are using a Kerlink Wirnet Station with Semtech Packet Forwarder and lora-gatway-bridge installed locally (currently version 3.3).

We have some testing with really high network load. After some hours to days, the gateway-bridge seems to stop working out of the sudden. Restarting the gateway-bridge-process (even without restarting semtech packet forwarder) solves the problem.

We did not dig deeper yet, but the log does not provide any further information. It seems that a downlink message from MQTT is not forwarded to UDP and a Uplink Message (ACK) from UDP is not forwared to MQTT: After that, the system does not accept any new UDP messages (and after one minute, the gatway unsubscribes because of that).

What version are your using?

3.3

Could you share your log output?

Here is everything ok

Dec 11 10:09:33 Wirnet local2.notice root: time="2019-12-11T10:09:33Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK Dec 11 10:09:33 Wirnet local2.notice root: time="2019-12-11T10:09:33Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushData Dec 11 10:09:33 Wirnet local2.notice root: time="2019-12-11T10:09:33Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=dev/gateway/7276ff000b0317a9/event/up uplink_id=3103e09a-dc57-40ac-9028-004d7c6618bc Dec 11 10:09:33 Wirnet local2.notice root: time="2019-12-11T10:09:33Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK Dec 11 10:09:34 Wirnet local2.notice root: time="2019-12-11T10:09:34Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:55627" protocol_version=2 type=TXACK Dec 11 10:09:34 Wirnet local2.notice root: time="2019-12-11T10:09:34Z" level=info msg="integration/mqtt: publishing event" downlink_id=e9fc23f4-c669-43a1-aa75-c34d905424cb event=ack qos=0 topic=dev/gateway/7276ff000b0317a9/event/ack Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushData Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=dev/gateway/7276ff000b0317a9/event/up uplink_id=28ec863b-72b4-496b-9c2d-e96ec3dfdd3c Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushData Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=dev/gateway/7276ff000b0317a9/event/up uplink_id=406be0b1-985f-4c9a-8572-0b71e6b4595a Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushData Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=dev/gateway/7276ff000b0317a9/event/up uplink_id=e49605a0-5354-47e3-8624-3c60ae800a1d Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushData Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=info msg="integration/mqtt: publishing event" event=up qos=0 topic=dev/gateway/7276ff000b0317a9/event/up uplink_id=d284688b-d676-4fbf-a52a-62faeb78435c Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: sending udp packet to gateway" addr="127.0.0.1:56482" protocol_version=2 type=PushACK

starting from here, 2 incoming packets are not forwarded anymore (up + down). After that, the system stops working

Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=info msg="integration/mqtt: downlink frame received" downlink_id=9d1d70de-7785-41a8-824e-94741851209b gateway_id=7276ff000b0317a9 Dec 11 10:09:35 Wirnet local2.notice root: time="2019-12-11T10:09:35Z" level=debug msg="backend/semtechudp: received udp packet from gateway" addr="127.0.0.1:55627" protocol_version=2 type=TXACK Dec 11 10:10:21 Wirnet local2.notice root: time="2019-12-11T10:10:21Z" level=debug msg="backend/semtechudp: cleanup gateway registry" Dec 11 10:11:21 Wirnet local2.notice root: time="2019-12-11T10:11:21Z" level=debug msg="backend/semtechudp: cleanup gateway registry" Dec 11 10:11:21 Wirnet local2.notice root: time="2019-12-11T10:11:21Z" level=info msg="integration/mqtt: unsubscribe topic" topic="dev/gateway/7276ff000b0317a9/command/#"

[brocaar]

Could you test again with the latest version?

[brocaar]

Closing this issue. In the latest versions there have been a couple fixes which I believe will fix this issue too.