helium / router

router combines a LoRaWAN Network Server with an API for console, and provides a proxy to the Helium blockchain
Apache License 2.0
69 stars 32 forks source link

Missing Uplinks #817

Closed RTK221B closed 2 years ago

RTK221B commented 2 years ago

Skylab has been noticing immense packet loss. They have a potential 5000 device deployment stuck due to this. They’re seeing 30-40% loss in uplinks. There's a probability that there's an overflow of data not getting forwarded to the console. Sensors that were working prior to the LHS update are no longer working. They have observed that the data was reaching the hotspot but is being dropped on the way to the LNS. They're also not seeing join attempts or accepts. The current gateways they are using are LongAP and Kerlink. Devices have also been rebooted/power cycled.

jdgemm commented 2 years ago

Let's see if the new miner update has had a positive impact on their situation like we've seen with other users: https://engineering.helium.com/2022/07/14/miner-hotspot-release.html

Net-Metrics commented 2 years ago

I'm still seeing this on some devices, others behave as expected. Don't know if it could be related but thought I'd post just in case it points towards something.

My devices with missing uplinks are the same devices that have the LinkADRReq/LinkADRAns Looping issue

https://github.com/helium/router/issues/791

On a device that behaves as expected I see the following

image

This device successfully transmits almost all of it's uplinks, you can see 1 missed uplink at the start of this, every other one was received

image

Contrast this with the device that has issues

image

Received uplinks, this sensor missed 15 uplinks over the same period

image