search

helium / router

router combines a LoRaWAN Network Server with an API for console, and provides a proxy to the Helium blockchain
Apache License 2.0
70 stars 31 forks source link

Console seems to not report all hotspots receiving the uplink #726

Closed dmamalis closed 2 years ago

dmamalis commented 2 years ago

  1. Are you using Helium hosted Console or running your own open source Console? We are using a private console hosted by Helium.

  2. If you are on Helium hosted Console, which url are you currently on? https://console-vip.helium.com

  3. If you are experiencing bugs in Helium hosted Console, please specify your Organization name and Device ID (ex. 3483050b-284d-47e2-8eda-e087fc03a753). Organization: trackplast Device ID: 9df3b703-a75e-439a-bf90-d30b16bf812a although this seems to affect multiple devices.

Please describe your issue: There seems to be an issue with console reporting the number of hotpsots receiving each uplink. The Join Request is reported to be receivied by multiple hotspots as seen on the picture below. Once the Join Accept is received by the end node, then an uplink fires up within the next few seconds. This uplink is reported to be received by only one hotspot although it is transmitted in the exact same way (transmission power, SF12 etc) and the environment is exactly the same.

image

Another indication that the message is trully being received by multiple hotspots but the console fails to report those is that each message is reported to be seen by a different hotpsot, all of which is in the list of the initial group of hotspots which receive the Join Request. image

Occasionally, the uplink message is reported to be received by two hotspots, but never more than that. image

vicmgs commented 2 years ago

Hi, this is expected behavior. Please see https://docs.helium.com/use-the-network/console/multi-packets/#configuring-multiple-packets to see all more uplink reports.

vicmgs commented 2 years ago

apologies, will look into this further

dmamalis commented 2 years ago

@vicmgs as described on the opening text sometimes the report shows that 2 hotspots receive the message. The multiple packets settings is set to all available and is applied to all devices through the label

image image image

vicmgs commented 2 years ago

it looks like all the events in the test-001 device have been purged since they were older than 7 days. Do you have another device currently experiencing this issue?

dmamalis commented 2 years ago

@vicmgs the data from test-001 is about 7 hours old. I will try to find a way to send data in a few minutes with another device (test-adeunis)

dmamalis commented 2 years ago

The only difference between test-adeunis and all the rest of the devices, is that test-adeunis is a created device with the keys issued by helium console, while all the rest of the devices are imported, thus the keys have been randomly autogenerated by an external script. This is probably irrelevant, however I leave it here as a possible cause

vicmgs commented 2 years ago

Looks like the adeunis is hearing from more than 1 or 2 hotspots consistently. Do you have any other devices suffering the issue to investigate? possibly one of the imported ones?

dmamalis commented 2 years ago

Unfortunately not at the time being as I am not in the office due to time difference. However, all "bottle-1xx" devices are on the field transmitting every 6 hours.Some of them will eventually transmit within the next minutes although, since they are on the field, we have no control of the environment, thus I can't tell you which will transmit and which will be within adequate coverage.

The issue however is consistent: all bottle-1xx devices, plus the test-001, test-002 devices were imported and then the label was applied on them. All 52 devices had the same issue.

If you have 7 days record, the data should be there. The devices where flushed and deployed a few days back.

It feels like some settings is not applied correctly but on the console/server side, rather than the GUI side. Moreover, we have an MQTT integration which saves all responses on a TSDB and it shows the exact same behaviour. This means that this is not a console GUI issue but rather a server side issue, maybe on the deduplication stage (?)

vicmgs commented 2 years ago

Agree with your assessment, everything on the UI looks correct. Moving this issue to router repo for further investigation.

dmamalis commented 2 years ago

@vicmgs behold live data from test-002. Also, a quick correction. test-xxx have autogenerated keys and not imported as I initially noted, so we can rule this idea out. image

vicmgs commented 2 years ago

If you click the "late" checkbox, you can see that other hotspots are hearing the uplink but dropping them. Deferring to router team to figure out why that is happening.

dmamalis commented 2 years ago

Is there any way those Late messages can be forwarded to the MQTT endpoint? This would partially solve our problem until the issue is reviewed and hopefully resolved. This project tries to exploit reception metadata in order to geolocate devices. If the multiple packets arrive on the console with the hotspot name and timestamp, then I suppose they can be forwarded to another endpoint too. then this would bring back some confidence to the whole project.

image image

macpie commented 2 years ago

Seems like you are trying to ack on every message which make the window on time small (so we are able to downlink). If you don't care too much about the ack I would stop so that your window of time can be bigger and you should get more packets in there.

dmamalis commented 2 years ago

@macpie first things first the image is from a test device which does send a lot of messages. The normal operation of the rest of the devices is to send a single message every 6 hours. The message is indeed a confirmed uplink with a LinkCheck request. If the device has adequate coverage it will drop it's TX power, this is an extra mechanism on top of the ADR.

No matter what, asking for a downlink is a normal process in LoRaWAN. As I understand, latency issues delay this functionality. This is something we can live with, knowing that the light hotspots will eventually make things better.

However, as I aforementioned, since the router delivers the late messages on the console, it would be helpful to deliver those messages in all endpoints (etc MQTT integration) including the metadata. This would allow application owners to investigate the data or draw other types of useful information from that data.

In our case, we switched to Helium, due to its vast coverage, in order to run trilateration algorithms and locate the devices but this behavior is a deal breaker for the time being. If the router can send Late Packets to an integration endpoint then the problem is minimized. I do think that this will be a vast improvement in the functionality and reliability of the network. Late or not, these packets do carry a lot more information than just the data :-)

macpie commented 2 years ago

@macpie first things first the image is from a test device which does send a lot of messages. The normal operation of the rest of the devices is to send a single message every 6 hours. The message is indeed a confirmed uplink with a LinkCheck request. If the device has adequate coverage it will drop it's TX power, this is an extra mechanism on top of the ADR.

No matter what, asking for a downlink is a normal process in LoRaWAN. As I understand, latency issues delay this functionality. This is something we can live with, knowing that the light hotspots will eventually make things better.

However, as I aforementioned, since the router delivers the late messages on the console, it would be helpful to deliver those messages in all endpoints (etc MQTT integration) including the metadata. This would allow application owners to investigate the data or draw other types of useful information from that data.

In our case, we switched to Helium, due to its vast coverage, in order to run trilateration algorithms and locate the devices but this behavior is a deal breaker for the time being. If the router can send Late Packets to an integration endpoint then the problem is minimized. I do think that this will be a vast improvement in the functionality and reliability of the network. Late or not, these packets do carry a lot more information than just the data :-)

It is an interesting feature, would you mind create a new ticket for the feature? (Deliver late packets)

dmamalis commented 2 years ago

It is an interesting feature, would you mind create a new ticket for the feature? (Deliver late packets)

I do not mind at all @macpie but I do mind for the data asap as this commercial project is actually crumbling as we speak