Assistance Needed: Incorrect Calculation of Data Rewards for Sensor-Based Data Traffic

[plainsimpleIoT]

Dear Engineering Team,

We are facing an issue with the calculation of data rewards for sensor-based data traffic on our network, which has led to poor data rewards for Community Hotspot Operators. We seek your assistance in resolving this matter promptly.

Background: Since the migration to Solana, our Community Hotspot Operators have observed that data rewards for sensors attached to a Helium Console prior to the Solana transition are not being credited with the new IoT token amounts as expected. This discrepancy in data rewards has caused consternation among the operators and has raised concerns within the community.

Observations from Fault Finding: During the fault finding process conducted by the Community Hotspot Owners, an interesting discovery was made. It was noticed that when a sensor attached to the console prior to the Solana migration is removed and then re-entered into the console, it starts receiving significantly higher IoT token rewards compared to its previous reward levels. It seems that only the sensors that undergo this process are being affected positively, while others remain unaffected.

Challenges Faced by Community Hotspot Owners: Unfortunately, the Community Hotspot Owners lack the necessary knowledge and access to accurately determine the appropriate data rewards for sensor-based data traffic. They are unfamiliar with the data credit reward accounting model, and they do not have the required access to data from the Helium Network oracles and the Solana blockchain.

Request for Assistance: In light of the above situation, we kindly request your assistance in investigating and resolving this issue. Our objective is to ensure fair and accurate data rewards for all sensor-based data traffic within our network. We believe that your expertise and access to relevant systems and data will enable us to identify and rectify the problem effectively.

Please consider the following points during your investigation:

1. Examine the data reward calculation process for sensor-based data traffic after the Solana migration.
2. Identify any discrepancies or anomalies that may be causing the lower-than-expected data rewards.
3. Analyze the impact of removing and re-entering sensors on the console and its effect on data rewards.
4. Determine the root cause of the issue and develop a solution to ensure accurate and fair data rewards for all sensors.

We greatly appreciate your prompt attention to this matter and your commitment to resolving it. If you require any additional information or have any questions, please do not hesitate to reach out to us. We are eager to work closely with you to rectify this problem and restore confidence within our community.

Thank you for your support and cooperation.

[plainsimpleIoT]

As an addendum, please note that the above issue has been found in the AUS region. I have no reports of this issue being felt in other regions of the Helium Network (I also haven't asked). It is also possible that the issue may also be an effect of the AU915 regional channel plan change that happened a couple of days prior to the Solana migration.

[Australian-Hnt-collectors]

as a back up. 3 sensors and a glamos unit were used all at the location of Wild Chambray octopus. LDS01, LDS02 and LWL02 LoRaWAN Wireless Water Leak Sensor Mike V can confirm and all lora compliant and the un install and re install of the sensors they were re installed yesterday and we can provide screen shots of earnings before and after as have those as well if needed. one of the beaconing only units Steep Watermelon Tarantula before factory reset in less than an hour of factory resetting it was witnessing as per the screen shots.

So clearly a problem and issue, Nebra denies any issue with their hotspots but this is now the sixth we have done some i own some others own the others but that proves a simple factory reset will make the units witness and as per what the screen shots show. taken less then four hours apart from actually preforming a factory reset

[Australian-Hnt-collectors]

the above screen shots show the earnings on the unit wild chambray octopus for average spend of the month details below. (https://github.com/helium/wallet-app/assets/84670450/6aae1618-0834-4f97-b46c-37e057300d55) /assets/84670450/86987b1d-169d-431c-a4c8-2519a4c5f168)

after the re install of sensors the day before yesterday so the night of the 21st now we have proof positive of rewards being off. Iot Earned for data transfer for the last 2 days are the following screen shots.

both days were for a spend of 500 DC

so logically the problem was on console and the sensor not rewarding correctly or feeding the information even though they were on boarded over a year ago on the console thus causing the console to not provide correct data of what information and data had been sent.

[Australian-Hnt-collectors]

as per requested all data supplied in screen shots has come from helium tracker

[Talitrus]

Is this related to the Helium Wallet app? This GitHub repo is for the Helium Wallet mobile app for Android and iPhone.

[plainsimpleIoT]

Would it not be both or is this repository for wallet app UX only. My push is that I only ever use the wallet App so this seems to be the most logical place to raise the Issue from my perspective. I was advised by a Foundation member to post it and if it needed to be moved then the repository owner would perform that task.

[ChewingGlass]

Can you guys move this discussion to https://github.com/helium/oracles

That's the repo controlling rewards, and that team will know how to debug here.

[Australian-Hnt-collectors]

Can you guys move this discussion to https://github.com/helium/oracles

That's the repo controlling rewards, and that team will know how to debug here.

No actually because MIKE VIERLING wanted us to put this here and has said it is the correct place for this to be ALSO read Plain and Simple above you have no right to close this as we was told to place it here and I was advised by a Foundation member to post it and if it needed to be moved then the repository owner would perform that task.

[ChewingGlass]

Looks like github allows transferring of an issue while keeping the same owner. I was worried about copy-pasting it myself, as then you would be removed from notifications and that dev team may need more context.

[macpie]

After investigation, it looks like this is due to the fact that HPR did not write a packet report (not triggering reward / payment) for the Join Request (JR).

This only happens when the JR is sent to multiple OUIs, at this point HPR cannot make a proper decision on which OUI to charge, as a result it does not write a packet report (note that data is still transferred to LNSs). This is an intended behavior of HPR to avoid charging the wrong OUI in those edge cases (code here).

When the device got reseted, I am guessing that, the join credentials changed and only pointed to 1 OUI triggering the report to be written properly again.

Thank you for the report and I hope this help clarify the behavior that you saw.

Side note: It was brought to my attention that staging server (Console/Router) might have been used for testing, please note that staging is not a stable environment as it is used for release testing only and is subject to "data wipe out" at any time (devices / orgs ...). Furthermore the multi-buy setting for Staging's OUI has also been lowered to 1 (as it is only a testing server).

[Australian-Hnt-collectors]

After investigation, it looks like this is due to the fact that HPR did not write a packet report (not triggering reward / payment) for the Join Request (JR).

This only happens when the JR is sent to multiple OUIs, at this point HPR cannot make a proper decision on which OUI to charge, as a result it does not write a packet report (note that data is still transferred to LNSs). This is an intended behavior of HPR to avoid charging the wrong OUI in those edge cases (code here).

When the device got reseted, I am guessing that, the join credentials changed and only pointed to 1 OUI triggering the report to be written properly again.

Thank you for the report and I hope this help clarify the behavior that you saw.

Side note: It was brought to my attention that staging server (Console/Router) might have been used for testing, please note that staging is not a stable environment as it is used for release testing only and is subject to "data wipe out" at any time (devices / orgs ...). Furthermore the multi-buy setting for Staging's OUI has also been lowered to 1 (as it is only a testing server).

Does not help the people missing out earning as once again things not working correctly

[madninja]

After investigation, it looks like this is due to the fact that HPR did not write a packet report (not triggering reward / payment) for the Join Request (JR). This only happens when the JR is sent to multiple OUIs, at this point HPR cannot make a proper decision on which OUI to charge, as a result it does not write a packet report (note that data is still transferred to LNSs). This is an intended behavior of HPR to avoid charging the wrong OUI in those edge cases (code here). When the device got reseted, I am guessing that, the join credentials changed and only pointed to 1 OUI triggering the report to be written properly again. Thank you for the report and I hope this help clarify the behavior that you saw. Side note: It was brought to my attention that staging server (Console/Router) might have been used for testing, please note that staging is not a stable environment as it is used for release testing only and is subject to "data wipe out" at any time (devices / orgs ...). Furthermore the multi-buy setting for Staging's OUI has also been lowered to 1 (as it is only a testing server).

Does not help the people missing out earning as once again things not working correctly

That's not what the answer implies, nor is the tone helpful.. it's working as specced right now.. If the device would stop trying to join multiple OUIs (by the user registering it with only one vs multiple) it would work as you would expect.

[Australian-Hnt-collectors]

After investigation, it looks like this is due to the fact that HPR did not write a packet report (not triggering reward / payment) for the Join Request (JR). This only happens when the JR is sent to multiple OUIs, at this point HPR cannot make a proper decision on which OUI to charge, as a result it does not write a packet report (note that data is still transferred to LNSs). This is an intended behavior of HPR to avoid charging the wrong OUI in those edge cases (code here). When the device got reseted, I am guessing that, the join credentials changed and only pointed to 1 OUI triggering the report to be written properly again. Thank you for the report and I hope this help clarify the behavior that you saw. Side note: It was brought to my attention that staging server (Console/Router) might have been used for testing, please note that staging is not a stable environment as it is used for release testing only and is subject to "data wipe out" at any time (devices / orgs ...). Furthermore the multi-buy setting for Staging's OUI has also been lowered to 1 (as it is only a testing server).

Does not help the people missing out earning as once again things not working correctly

That's not what the answer implies, nor is the tone helpful.. it's working as specced right now.. If the device would stop trying to join multiple OUIs (by the user registering it with only one vs multiple) it would work as you would expect.

If you knew this community you would understand why they are annoyed about things not working correctly its everyone else's fault but whos for weeks being asked to supply data to be told its working as specced right now does not once say.. sorry it was broken for this reason and you were right there was an issue.. suppose next it will be Genesis was not paid out to many hotspots because of multiple attempts to send data. Bud honest i have been as respectful as i intend to be with you Have a good day Oh and in this case, and all the cases of missing rewards AU915 is the sensor unit with four different sensors and a glamos so they were all trying multiple were they

[madninja]

After investigation, it looks like this is due to the fact that HPR did not write a packet report (not triggering reward / payment) for the Join Request (JR). This only happens when the JR is sent to multiple OUIs, at this point HPR cannot make a proper decision on which OUI to charge, as a result it does not write a packet report (note that data is still transferred to LNSs). This is an intended behavior of HPR to avoid charging the wrong OUI in those edge cases (code here). When the device got reseted, I am guessing that, the join credentials changed and only pointed to 1 OUI triggering the report to be written properly again. Thank you for the report and I hope this help clarify the behavior that you saw. Side note: It was brought to my attention that staging server (Console/Router) might have been used for testing, please note that staging is not a stable environment as it is used for release testing only and is subject to "data wipe out" at any time (devices / orgs ...). Furthermore the multi-buy setting for Staging's OUI has also been lowered to 1 (as it is only a testing server).

Does not help the people missing out earning as once again things not working correctly

That's not what the answer implies, nor is the tone helpful.. it's working as specced right now.. If the device would stop trying to join multiple OUIs (by the user registering it with only one vs multiple) it would work as you would expect.

If you knew this community you would understand why they are annoyed about things not working correctly its everyone else's fault but whos for weeks being asked to supply data to be told its working as specced right now does not once say.. sorry it was broken for this reason and you were right there was an issue.. suppose next it will be Genesis was not paid out to many hotspots because of multiple attempts to send data. Bud honest i have been as respectful as i intend to be with you Have a good day Oh and in this case, and all the cases of missing rewards AU915 is the sensor unit with four different sensors and a glamos so they were all trying multiple were they

I do know this community and why they're annoyed. I really feel for you all.. This, however was not a "this is not working as specified" moment and assuming it is appears to be a trademark of this community.

If you'd like to have a productive, positive discussion about what can be improved, get into Discord and let's have at it. I think much positive progress has been made and, while definitely annoying that it's been taking this long, how do these negative comments here (which are not even accurate) actually help? Let's move forward.. get into Discord and have a normal, non confrontational, conversation

[Australian-Hnt-collectors]

Not being negative just saying when people been trying for weeks and months it kicks the stuffing out of them hope you have a good rest of your day

[plainsimpleIoT]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

[Australian-Hnt-collectors]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

Not sure how a community sensor operator would have done this if following the videos of how to use or using the set up guide. i would also like to know as well

[madninja]

Not being negative just saying when people been trying for weeks and months it kicks the stuffing out of them hope you have a good rest of your day

hard to interpret "as once again things not working correctly" any other way.. try to make the best out of your day too!

[madninja]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

you'd add a sensor on one console (like vip, production, staging or dev or any of the other console operators out there)and then also add it on another..

[madninja]

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

There were bugs filed against staging and dev consoles for "operational issues" which is definitely against expectations of either of those.. just don't use them unless you understand the implications of losing data, rewards, connectivity

[Australian-Hnt-collectors]

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

There were bugs filed against staging and dev consoles for "operational issues" which is definitely against using either of those.. just don't use them unless you understand the implications of losing data, rewards, connectivity

Ok been wanting to ask then, if i had Mike V, Travis and Victor, Slaven from glamos all on my console and all could not find the issue as they were connected correctly please answer how as a community member i could solve the issue.? maybe your get why we are frustrated

[plainsimpleIoT]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

you'd add a sensor on one console (like vip, production, staging or dev or any of the other console operators out there)and then also add it on another..

Ok that makes sense to a degree. If I am understanding this correctly when you delete the sensor from one console there is a possibility that the Sensor doesn't correctly adjust the change or perhaps console doesn't correctly purge the relationship or the gateway doesn't do its bit? Is there more to it though when you factor in the AU Freq Band Change? I still have 2 sensors that refuse to connect to any console which I am working through with the hoster. These were previously configured to AU915sb6 and have been reconfigured to AU915sb2 since the Freq change.

[madninja]

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

There were bugs filed against staging and dev consoles for "operational issues" which is definitely against using either of those.. just don't use them unless you understand the implications of losing data, rewards, connectivity

Ok been wanting to ask then, if i had Mike V, Travis and Victor, Slaven from glamos all on my console and all could not find the issue as they were connected correctly please answer how as a community member i could solve the issue.? maybe your get why we are frustrated

no need to rehash this over and over.. I certainly get why you're frustrated.. but repeating it isn't helping getting to resolution

[madninja]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

you'd add a sensor on one console (like vip, production, staging or dev or any of the other console operators out there)and then also add it on another..

Ok that makes sense to a degree. If I am understanding this correctly when you delete the sensor from one console there is a possibility that the Sensor doesn't correctly adjust the change or perhaps console doesn't correctly purge the relationship or the gateway doesn't do its bit? Is there more to it though when you factor in the AU Freq Band Change? I still have 2 sensors that refuse to connect to any console which I am working through with the hoster. These were previously configured to AU915sb6 and have been reconfigured to AU915sb2 since the Freq change.

The most common mistake is that people do not remove it from one console before adding it to another. The gateway just forwards packets.. the HPR sees those devices registered with more than one OUI so it can't report which one should get rewarded.

[Australian-Hnt-collectors]

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

There were bugs filed against staging and dev consoles for "operational issues" which is definitely against using either of those.. just don't use them unless you understand the implications of losing data, rewards, connectivity

Ok been wanting to ask then, if i had Mike V, Travis and Victor, Slaven from glamos all on my console and all could not find the issue as they were connected correctly please answer how as a community member i could solve the issue.? maybe your get why we are frustrated

no need to rehash this over and over.. I certainly get why you're frustrated.. but repeating it isn't helping getting to resolution

Does not answer the question just side steps it HOW AS A COMMUNITY MEMBER AM I SUPPOSED TO SOLVE IT. Only one console ever used only installed the sensors on a single console. Had your own team Checking this and still they could not find the issue. So please explain if experts from helium couldnt find the problem your explanation does not make sense was working before the duel without an issue since duel now does not correctly. rewards are not being given correctly and a pattern is showing which is being watched by certain interested parties. how comes the best option as you said above just don't use them unless you understand the implications of losing data, rewards, connectivity.. thought you wanted Data right now you do not. you cannot be half in or out. But hey guess ill guide my community members to "JUST DONT USE IT" would that be better.. only taken seven weeks till this point and still you dance around the questions like my mum ball room dances Honest and being totally respectful it feels like the community does more then a lot of your own employees do on problem solving and that is what Helium and foundation should respect over all not keep side stepping and dodging direct questions

[plainsimpleIoT]

Can you please clarify the process that a Community member sensor operator could have inadvertently placed their sensors into this edge case (the multiple OUI's)? Understanding this would enable a Community Sensor Operator to better be able to make deliberate decisions covering this point.

Continuing the side note - there are Helium Console organisations on both the Dev and Prod networks in use by Community sensor operators. The dev organisations understand the purpose and operations of the Dev environment.

you'd add a sensor on one console (like vip, production, staging or dev or any of the other console operators out there)and then also add it on another..

Ok that makes sense to a degree. If I am understanding this correctly when you delete the sensor from one console there is a possibility that the Sensor doesn't correctly adjust the change or perhaps console doesn't correctly purge the relationship or the gateway doesn't do its bit? Is there more to it though when you factor in the AU Freq Band Change? I still have 2 sensors that refuse to connect to any console which I am working through with the hoster. These were previously configured to AU915sb6 and have been reconfigured to AU915sb2 since the Freq change.

The most common mistake is that people do not remove it from one console before adding it to another. The gateway just forwards packets.. the HPR sees those devices registered with more than one OUI so it can't report which one should get rewarded.

What I am struggling with regarding this particular identified problem/rectification to the issue is that this could be an explanation for 1 or 2 instances but we are talking dozens of affected users. It is unlikely that this problem resolution is a root cause. Taking into account that a number of users did not alter their Sensor configs for the entire AS9123-1 period, electing to have them remain on AU915sb2 then when the switch back to AU915sb2 occurred they started processing data again but the rewards were reduced. Only after deleting and re-adding to the console did the DC rewards return to a level that appears inline with the data processed. Is it possible that there is a logic problem in the HPR code?

[macpie]

they started processing data again but the rewards were reduced

I cannot talk about the frequency changes (not my area of expertise) but I will look deeper into this ^.

[macpie]

I have not found any evidence of miss-calculation.

NOTE: We have posted a new update recently to Session Key Filters (SKF) to better handle packet copies.