Significant reward drop. Affecting mostly DIY miners. Seems peerbook related

[apppie123]

Since the Dec 29th update a growing group of users is reporting a significant reward drop of >80% from one day to the other. Hotspots that normally witness beacons several times per hour. Now only get rewarded several times per day.

What these users have in common:

1. Mostly DIY packet forwarder users with HIP19 approved ECC key and OG helium miners with alpha key.
2. Problem is tied to the P2P address or complete wallet containing multiple hotspots.
3. Are relatively high HNT earners.
4. All the hotspots have stopped beaconing for days(mine stopped beaconing for 10 days and counting).

1 -> This could be a logical consequence since DIY users can see the miner logs and learn what is happening.

2 -> The issue is not tied to one brand. Even when reverting to original manufacturer image does not solve the issue. The local network is OK as tested with helium status and other networking tools. The issue is both with relayed and non-relayed hotspots. Mostly DIY setups but there are reports of original manufacturer hotspots as well.

The affected hotspots do see a high amount of witnesses throughout the day. This is confirmed using the miner logs. It is not a classical case of p2p issue for the hotspot not being able to dial the challenger. The logs clearly show a successful submission of PoC witness to the challenger.

There is currently a lively discussion ongoing on #diy-packet-forwarder and #poc-discussion on the Helium Discord. Unfortunately no one from the Helium team has chimed in.

Affected hotspots still do send out challenges. However the result is usually a low number(0-6 witnesses) indicating that challengees cannot reach the challenger.

"Add ability to control peer gossip for debugging purposes" This new piece of code could be related. When the gossip is disabled, rewards will slowly die down as the miner cannot be found in the network. Are hotspots selectively grey listed? Like a HIP40 deny list?

https://github.com/helium/erlang-libp2p/commit/853cada117daf887f492102d29a8aaa7f8175772

This is how the reward drop looks like:

Miner log, most of these PoC submissions were not rewarded:

[ricopt5]

This is not new and there's already a github issue for this, but it has been closed: https://github.com/helium/miner/issues/1238

[apppie123]

This is not new and there's already a github issue for this, but it has been closed: #1238

This is not just a small fraction. This a complete drop in rewards as if these miners were shut off the network for a seemingly specific group users. When this was the same issue, you would expect that the drop in rewards wouldn't last for over 10 days now with consistent 0 rewards.

It is not the case of "failed to send witness, max retry". These hotspots send PoC witness receipts with "Successfully sent witness to challenger", so they can actually dial the challenger but from there the PoC witnesses disappear. See the picture in OP of one of the logs.

[ricopt5]

This is not new and there's already a github issue for this, but it has been closed: #1238

This is not just a small fraction. This a complete drop in rewards as if these miners were shut off the network for a seemingly specific group users. When this was the same issue, you would expect that the drop in rewards wouldn't last for over 10 days now with consistent 0 rewards.

It is not the case of "failed to send witness, max retry". These hotspots send PoC witness receipts with "Successfully sent witness to challenger", so they can actually dial the challenger but from there the PoC witnesses disappear. See the picture in OP of one of the logs.

Don't give too much attention to the title, check the content. It's the same issue where witness receipts are delivered but there's no poc receipt in the blockchain. There are people who confuse stuff and started posting the "max retry" logs, but that's not what the issue is about.

50% of my miner delivered witnesses (around 46%-47% to be exact) end up with no poc receipt in the blockchain. And this has been happening since the start, even when my sensecap was stock. It can happen for others for that percentage to vary, which is your case in which maybe 90% of witnesses end up with no poc receipt.

[FezzFest]

I can confirm @ledmaniak's findings. Multiple hotspots whose earnings went down to ~10% of what they were before January 1st.

None of the hotspots above have sent a single beacon since January 1st, while they sent multiple before. Configuration of these hotspots remained unchanged during this time. Every single attempt to make them work again so far has been unsuccesful.

[apppie123]

Don't give too much attention to the title, check the content. It's the same issue where witness receipts are delivered but there's no poc receipt in the blockchain. There are people who confuse stuff and started posting the "max retry" logs, but that's not what the issue is about. OK, I see you're right. It is good that it is brought under attention again as it might have become worse?

50% of my miner delivered witnesses (around 46%-47% to be exact) end up with no poc receipt in the blockchain. And this has been happening since the start, even when my sensecap was stock. It can happen for others for that percentage to vary, which is your case in which maybe 90% of witnesses end up with no poc receipt. I can understand that these things happen. But this 80-90% drop(well, it's almost at 100% now) happened from one day to the other for 15 of my units and it's happening since Dec 29th and not just me. No 50% drop, but close to 100% for a lot of days.

I have onboarded 4 new units several days ago. They worked for two days and are now completely flatlined as well.

So far the only response I have seen is one of the team members saying that they don't see this in their data and the network is running(and thus the issue doesn't exist?). It's disturbing.

[ricopt5]

So far the only response I have seen is one of the team members saying that they don't see this in their data and the network is running(and thus the issue doesn't exist?). It's disturbing.

Yeah, no... This data shows that roughly 50% of challenges, end up with no poc receipt. Of course, this can be caused by p2p issues where the challenger isn't able to dial the challengee, but as we can see in our miner logs, that's not the only issue. Unfortunately, there's no way to get statistics of the "delivered witness but no poc receipt" issue.

[PaulVMo]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

[ash-vd]

I have the same issue, with a complete wallet. These miners are professionally installed in radio towers, and were all earning 1HNT+ per 24 hours. From the 1st of January this has been decreasing, with 0,25HNT on 1-1 and now 0.08HNT per miner.

This is a screenshot of the complete wallet, with 67 miners in it:

30-12: 47HNT 31-12: 23HNT 1-1: 12HNT

The drop on the 26th and 27th is expected as there were issues with the internet connection. I've checked, and the connection is stable now.

There is 1 miner in this wallet that has "custom" firmware: the packet forwarder and miner run as a docker container on a linux arm64 OS. These containers are the open-sourced containers from Helium without any changes. When I check the logs for this miner I see many messages that the witness has successfully been sent to the challenger, but almost none of these gets rewarded. Please see the following screenshots. In the miner logs there are clearly multiple witnesses sent in just an hour. However, I only got rewarded for 1(!) witness for the whole day.

Here are some screenshots of other hotspots in that same wallet:

[davidwang1030]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

I have the same problems，can we talk about more details，can you give me ur emails？

[skylign]

https://github.com/helium/miner/issues/1339#issuecomment-1006685421 agree with you I think your answer is other question's solution , my problem is more similar with your answer . I don't receive challenge for more 4 days , not confirmed in blockchain , but don't receive , I doubt my address don't be recorded by other HS in their peerbook and what can I do ?

[zxgangandy]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

Same like this issue, please help to anlyze.

[JJsilvera1]

I can confirm @ledmaniak's findings. Multiple hotspots whose earnings went down to ~10% of what they were before January 1st.

None of the hotspots above have sent a single beacon since January 1st, while they sent multiple before. Configuration of these hotspots remained unchanged during this time. Every single attempt to make them work again so far has been unsuccesful.

Same issue with a few OG helium miners that I have and friends have. I am unsure how to debug them or to dive down deeper to see whats wrong. But I assume this is the issue. After the newest update... just flatlines and no activity.

[timetraveler90]

@JJsilvera1 this is exactly the same issue i am experiencing right now with my sensecap. From nearly 0.5hnt a day dropped to 0.02

[SrxDev]

Same issue here. No beacons in 10 days and lost ~80% of witnesses. None of the other miners in the nearby hex have been impacted.

Miner is only issuing challenges.

[apppie123]

It was noticed by several people that the few witnesses that are registered on the blockchain come from challengers from makers FreedomFi, Syncrobit and PantherX.

Are they late with updating the latest image? What mechanism has changed that other brands like sensecap and bobcat never seem to be able to complete a PoC submission from the witness?

[tomasz-grabowski-netbulls]

I'm experiencing the same issue on my diy miner.

[Woutch]

Same issues: Since 5 day's only challenges. Networking wise all tested and ok. And since the latest network var update last day I also see data packets again. Went from 65 witnesses to 0...

Mine is a SenseCap M1: Eager Watermelon Wombat. Strange thing is also the low beaconer witnesses, if I check most hotspots, they are in populated hexes and should see many more than 0 or 2....

[FezzFest]

It was noticed by several people that the few witnesses that are registered on the blockchain come from challengers from makers FreedomFi, Syncrobit and PantherX.

I can confirm this as well. Looking at one of the affected hotspots, I see:

Looking at an unaffected hotspot closeby, I see:

[semihsmg]

All of my hotspots also flatlined, 6 of them at the same time. Decrease in rewards started in January 3rd but flatlined in 5th. All hotspots lost %90 earnings. All was above average.

[semihsmg]

Same issue here. No beacons in 10 days and lost ~80% of witnesses. None of the other miners in the nearby hex have been impacted.

Miner is only issuing challenges.

my hotspots also only get reward from challenges

[zxgangandy]

The truth is that the helium technical team only five people, they have no processing ability!

If this is true. Why not the helium technical team transfers the project to another team who can hold all the situations. Instead of always putting us to wait and wait.

[velasquez87]

I have the same issue on my bobcat since 29 Dec. Around me everybody is earning 0.5hnt a day while mine is at 0.05...almost no witness and beacon 0 0

B

[serbyxp]

Ditto. I notice the logs are sending the receipts, sometimes it needs to try 2-4 times . Is this an issue on the Erlang and MAC address? Is MAC or HARDWARE address some how used in the the Erlang to obtain a connection / accept receipts with peers? Could p2p peers with a ports with 2154 or any other random port not opened by the container be an issue? Nebra miners have a upnp container, Should one be implemented by all hotspots, or any further ports be open either outbound inbound on the container, or on the network environment?

The receipt gets accepted, but I have not seen all the witnessing rewards on receipts. I understand not all witnesses get put into the PoC rewards if its over X amount of witnesses I think 10 now .. and that is " randomly selected" are the hotspots that dont "win" the PoC reward if total witnesses are over X amount not get their " receipts reported"? and/or are " invalid witness sent and then discarded by the end peer due to a " flagged" invalid status of some sort?

[JJsilvera1]

One of my OG miners is only able to see beacons from other OG miners. It no longer is able to get rewards if it sees bobcat or rak or any other kind of manufacturer.

[serbyxp]

One of my OG miners is only able to see beacons from other OG miners. It no longer is able to get rewards if it sees bobcat or rak or any other kind of manufacturer.

Could this be a lora_hal issue? incompatible? Is the sx130x_hal Packet Forwarder still the defacto? or has it changed to a newer packet_forwarding service that isnt the Semtech flavor or something different? The packet_forwarder from lora-net sx1302_hal is what im running which is the Semtech UDP packet forwarder. Is it possible that something in the msg header that is conflicting with the recipient of our PoC receipt, like if the msg string in the MAC or euid could be formatted differently causing this?

** also I notice in the logs whenever a client connects to the miner to send a message, it isnt using port 1680, its using the second port number in the sys.config ..

      {radio_device, { {0,0,0,0}, 1680,
        {0,0,0,0}, 31341} }
    ]}

I noticed that the 31341 port is the one being used. Should our container reflect this or our networks? if our packet forwarder is in another container or running in shell would that cause a beacon not to be sent? I have seen no error, on this. I have just noticed it. The radio indicator light runs on that it sent a packet. Im not familure with what happens after a packet is sent over the radio?? like their a an ACK reply or packet containing metrics of the concentrator in a msg or string format that could be conflicting with the port, or the p2p network?

[FuraoEvil]

This looks quite similar to my wallet But no DIY miners here, 7 Bobcats and 2 Heltecs I sold one Bobcat but it doesn't look better in the new owners wallet. Almost as if they were blacklisted.

[zxgangandy]

I have the same issue:

[biohazd]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

[Woutch]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

Did you do something? I already power cycled, reset my blocks and start from new snapshot + turbosync (sensecap). But still no beacons for 7 day's now.

I'm not relayed, and sending challenges, but these never go over 2 responses. mostly 0.

Thanks!

[biohazd]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

Did you do something? I already power cycled, reset my blocks and start from new snapshot + turbosync (sensecap). But still no beacons for 7 day's now.

I'm not relayed, and sending challenges, but these never go over 2 responses. mostly 0.

Thanks!

I did a reboot about 4 hours ago.

I did also reboot yesterday and the day before and those did nothing. My miner is always fully synched and has good internet connectivity.

[semihsmg]

Had one hotspot that I registered to helium network in 28/12/21 but couldn't deploy it so it was offline. Yesterday, I deployed it and this one also effected. It seems, it doesn't matter the hotspot is online or not before flatlining, all hotspots that are in that wallet affected...

[FuraoEvil]

This looks quite similar to my wallet But no DIY miners here, 7 Bobcats and 2 Heltecs I sold one Bobcat but it doesn't look better in the new owners wallet. Almost as if they were blacklisted.

As most of mine are Bobcat I opened a case with them, here their first reply

Daphne (BOBCAT) Jan 10, 2022, 18:29 GMT+8 Hi Christophe，

Thank you for contacting Bobcat Miner.

After reviewing your email, we have internal upgraded your issue, and our engineers will deal with it as soon as possible. Please observe your miner together. If the miner has PoC activity again, it means that we have solved it for you.

Thank you for your support.

Best, Daphne

I'll keep you posted

[dogweather]

A vast proportion of Kerlink miners had this same drop - near flatlining, beginning Dec. 30. After 4 or 5 days of this there was some kind of undocumented action taken by Helium - ? - and earning resumed afterwards.

[FezzFest]

@dogweather Can you tell me more about the Kerlink issue? Is this documented/discussed somewhere or is this something you found out yourself? Did Kerlink roll out an update after this issue or did 'resolve itself'? I'd like to make sure we're dealing with the same issue.

[BenoitDuffez]

Kerlink dev/support here.

I was told by the core dev that an update of the seed nodes would help, and it did. All earnings went back to normal after the update. We didn't change anything.

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

[zxgangandy]

Kerlink dev/support here.

I was told by the core dev that an update of the seed nodes would help, and it did. All earnings went back to normal after the update. We didn't change anything.

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

What are the seed nodes u've changed，dude？

[Woutch]

Kerlink dev/support here.

I was told by the core dev that an update of the seed nodes would help, and it did. All earnings went back to normal after the update. We didn't change anything.

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

Nice, care to share what you have changed?

Many thanks!

[semihsmg]

I got this reply from sensecap tech support

Hi the seed node changes were on the helium end not the kerlink end, these issues relate to the p2p side of the helium network and not a specific hotspot or maker I'm afraid. Some things which we have seen that have helped (in some but not all cases) are;

1) Try a new internet connection for 24 hours to see if the problem goes away, you can use your phone as a hotspot to try or 2) remove all port forwarding, this forces your hotspot to be relayed and has helped some people for number 1, you could try the mobile hotspot

Now removed forwarded ports and waiting it to become relayed. I'll post an update if anything happens.

[FezzFest]

@zxgangandy @Woutch Please read @BenoitDuffez's comment carefully. Kerlink did not change anything. Helium pushed an update to the seed nodes after which the issue was resolved. I'm wondering if a helium dev could enlighten us to what was changed on the seed nodes?

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

This is what we see as well. I checked a random Kerlink hotspot and notice that during the period between January 1st and January 5th, most challengers were Synchrobit, FreedomFi and PantherX. This also seems symptomatic for the issue we currently still experience.

[Woutch]

@zxgangandy @Woutch Please read @BenoitDuffez's comment carefully. Kerlink did not change anything. Helium pushed an update to the seed nodes after which the issue was resolved. I'm wondering if a helium dev could enlighten us to what was changed on the seed nodes?

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

This is what we see as well. I checked a random Kerlink hotspot and notice that during the period between January 1st and January 5th, most challengers were Synchrobit, FreedomFi and PantherX. This also seems symptomatic for the issue we currently still experience.

Thanks to point this out. Going to try the next workaround, put miner in relay status by closing port. Seems to be counter productive. But anything is better than only creating challenges... (and maybe wait on an official statement from the Helium team).

[apppie123]

@zxgangandy @Woutch Please read @BenoitDuffez's comment carefully. Kerlink did not change anything. Helium pushed an update to the seed nodes after which the issue was resolved. I'm wondering if a helium dev could enlighten us to what was changed on the seed nodes?

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

This is what we see as well. I checked a random Kerlink hotspot and notice that during the period between January 1st and January 5th, most challengers were Synchrobit, FreedomFi and PantherX. This also seems symptomatic for the issue we currently still experience.

Thanks to point this out. Going to try the next workaround, put miner in relay status by closing port. Seems to be counter productive. But anything is better than only creating challenges... (and maybe wait on an official statement from the Helium team).

Already tried that. It doesn't help either.

[dogweather]

Does anyone know where these seed node changes would show up? Is there a relevant repo? Or blockchain transaction type?

We should be able to find what was changed, and who was responsible.

[dogweather]

This is a graph posted to the Kerlink Discord. It supports the theory that Kerlink was disproportionately affected.

So it'd be very interesting to know what the Helium devs changed that resolved this.

[FezzFest]

@dogweather As far as I know, the code running on the seed nodes is built from this repository. See tag 'seed0.0.45'.

[FezzFest]

After extensive debugging and reverse engineering, the issue was found. The Helium developers included an unofficial modification in the docker container pushed on quay.io. This unofficial modification blocks people from participating in Proof-of-Coverage, thus reducing their hotspots earnings to zero.

If you want to know whether you are affected, I created a github repository with a tool that returns true (if you're banned) or false (if you're fine): https://github.com/FezzFest/check-helium-bannedlist

[SrxDev]

This might help solve the issue. https://github.com/FezzFest/check-helium-bannedlist .created byFezzFest

[Woutch]

YES! Found my hotspot in the list... (eager-watermelon-wombat)

I do not know why, I have a regular Sensecap M1 with an 5.8dbi omni on my roof. Normally earning only a small amount, 0.1 to 0.2 HNT per day. No modification whatsoever.

At least I don't need to keep debugging all things I can think off, just waiting on the new 2022.01.11 firwmare.

I hope if the list comes back, I'm not on it anymore...

[FuraoEvil]

Ok so let me get this straight: "The peoples network" banned thousand of miners in an act without vote or mandate by "The People" following non existent rules. And instead of apologies they are now pulling a quick poll to have it validated afterwards by the community?

So let me line things up: Consider a wallet with 10 miners in it, all legit and above average got blacklisted entirely since 3 weeks This counts for lets say 2 HNT per day/40 HNT over the ban period for that wallet alone. These 10 Hotspots didn't send any Beacons which results in a loss of 200 beacons sent. Which means potentially 3600 witnessed beacons gone missing for the surrounding Hotspots Oh and there are the challenges that never reached their destinations and therefor couldn't be beaconed and witnesses That's another 1800 PoC events not taken into account by completely innocent Hotspots around the world. Do I get this right? And the best you can do is to say you acted in the best interest of "the peoples network"?

Heads up to the hard work of the community members who did the work over several days to find and prove this as Helium obviously decided to not tell anyone till they were forced to break their silence today.

For me that's a very weak stance but I let you all judge for yourself...

Check below if yours on the list or not

The discussion on Reddit: https://www.reddit.com/r/HeliumNetwork/comments/rywhtm/3000_hotspots_flatlined_since_dec_27th/ The inoffical list: https://gist.githubusercontent.com/ricopt5/d7a8fd85de6dc18cbb72ab74b92d0223/raw/640f84a05c6bdbef2133e4638682646006175478/helium_blacklist.csv The declaration of Helium in their Discord > Announcements

[serbyxp]

Ughh this seems all sorts of messed up. Cus if they are black listing people based on “statistics” if I’m 1 wallet with 1 hotspot on it. And I get black listed, and the person with 20 hotspots on 1 account doesn’t … “statistically” speaking the person with 20 hot spots on 1 account is more then likely the person “gaming” if they are . So the statistics of the 20 hotspots vs my 1 makes me the “gamer” ?? If all the RSSI data and all the SNR data is being quantified from “false data” then the person with accurate data is looked as the “gamer” ..?

I'm not on that list... So That doesn't explain very much for me. Unless the people around me are on that list... or the peers / challangers are on that list, that could indirectly affect hotspots not on that list.

[dogweather]

@FezzFest Can you show us the source of your blocklist data?

I looked through your commits and source code but couldn't find info about how you gathered the data.

Your work looks excellent, but I'd like to verify it.