Absolutely no mining activity since mandatory helium hotspot firmware 2021.10.27.0 (0 beacon/0 witness, only challenges)

[ElYoM]

Hello, Since the deployment of the mandatory firmware 2021.10.27.0, my 3 hotspots have simply stopped mining. It's particularly obvious when you look at the curve over 30 days where you can see that all mining activity ceases following the application of this firmware (0 beacons/0 witness, only challenges), there was just one witness on November 05 from one of the 3 hotspots (a last moan before dying certainly). Please note that my 3 Bobcat hotspots are perfectly synchronized, configured and not relayed. They have an Ethernet connection on a 10 Gbps fiber optic connection and are not behind a firewall or VPN. They were also perfectly functional until that fateful date.

You can see my 3 hotspots here : https://explorer.helium.com/accounts/13DNY2QEgfEuQHxAfJmU96KoEq8MVyKt1N8ujyABMSASwW26Eyw Or here : https://app.heliumstatus.io/accounts/13DNY2QEgfEuQHxAfJmU96KoEq8MVyKt1N8ujyABMSASwW26Eyw

Please note that this problem also affects other people and in my case it is very problematic because I am all alone in my city with my 3 hotspots for a few more months. A resolution of this problem with the necessary investigation would be welcome, 15 days of complete unavailability being already an unacceptable downtime. Thanking you in advance for your involvement in solving this problem that puts many people in trouble and I am of course available if I can provide additional elements to help solve the problem.

[leyuansugar]

same here with 16 hotspots

[b-effect]

same here for 8 days

[ElYoM]

Thank you for your confirmation that in fact many hotspots are affected by this problem. It starts to confirm that the problem seems to have appeared with the deployment of this mandatory firmware.

[Mickatchoux]

Same here

[andy210000]

I can’t understand why they don’t revert it back to the firmware before the27th October

this would stop everyone going on about the miners flatlining and then fix the issues before releasing the next firmware

[ElYoM]

Still, they should recognize that there is a problem (Helium and the manufacturers) because considering how I was received on some Discord, it is not won... Either I was told that not many people had this problem (without ever bringing any proof of this statement and even though testimonies were multiplying everywhere), or they didn't care about me at all... In short, as long as it works, you're fine but as soon as you have the slightest problem you're all alone (and this even if the problem is not your fault)...

[gusosborne]

It does seem like maybe firmware & variables (everything) should be rolled back to pre-November 5th, just to get back to normal working baseline, before pushing forward. I will say however, I have seen a ~10% improvement over the past two days. 1-2 beacons per day is much better than none.

[ElYoM]

Arf, knowing that many of us had the problem from October 27, if they rollback that from November 05, we will remain long at 0 rewards ... :/ I would be happy to have 1 reward per day but not even that, currently it's 0 rewards since October 28 and for 3 hotspots !

[gusosborne]

I did not realize the zero beacons issue (still thinking it’s this: https://github.com/helium/miner/issues/1174) had started that early on. From most of the reports I’ve read, it seemed to be centered around the variable application and rollback on the 5th.

That raises a question though: was it a firmware bug in the 10.27 release, which was fully activated by the variable / rollback?

Firmware can take several days to be fully distributed and applied, so the timing on that October 27 firmware release could line up pretty closely to the variable / rollback on the 5th as well.

I fear that I have speculated quite enough, without sufficiently digging into the code..:/

[ElYoM]

Yes, we won't have fun speculating on the exact reasons of the problem (0 beacon and 0 witness, only challenges), that's the developers' job ;)

It was just to insist on the fact that, for some people, this total unavailability became simply unacceptable because of its duration and the fact that we were a bit mistreated on the communities by not taking us seriously.

[ProTechEx]

The same here with 3 hotspots.

I cannot understand the amateurish behavior of helium.

The API servers are often overloaded or completely inaccessible. It is always said that the problem is related to the quick growth of the helium network. I agree with this, but this is not an excuse in my mind. The Helium team should have predicted this rapidity growth.

According to this situation, they had expand the infrastructure and network and as well as their security staff.

Or another solution would have been to stop accessing new devices in to the helium network for a certain time, at least until expanding the required infrastructure and staff team.

It is also incomprehensible to me that new firmware versions are released that contain bugs, which negatively affect the rewards of the hotspot owners.

The big question for me is will helium compensate us for those failures?

[gusosborne]

I would be happy just to have things back to normal, which I assume would be about 25-30% less earnings than October (because of the ~100K new hotspots added since then). The current and instant / unannounced reduction in rewards/beacons since November is unsustainable for me. One of my miners did just send a beacon, for the first time in 4 days though (the other is going on 10 days now). Little bits of hope…

We know something is still obviously very wrong when an outage/overload warning banner is displayed pretty much anytime you open the app. I’m sure the devs are working extremely hard to resolve this- and are probably much more stressed about this than we are. I sure hope we can all get some relief and a workable resolution soon..

[bitlite]

The same here with 3 hotspots. I cannot understand the amateurish behavior of helium.

I can't believe the amateurish behavior of OP. No MINER.JSON to back him up, No RESET Miners performed, No basic Troubleshooting before making an issue ticket. Have you even contacted your Vendor? If it was Bobcat, they would have looked at it within 48 business hours.

The API servers are often overloaded or completely inaccessible. It is always said that the problem is related to the quick growth of the helium network. I agree with this, but this is not an excuse in my mind. The Helium team should have predicted this rapidity growth.

Agreed and have the hardware and code to support this growth.

Or another solution would have been to stop accessing new devices in to the helium network for a certain time, at least until expanding the required infrastructure and staff team.

Not an option as people who bought their miners wants to get connected. This is still a project and this project is still in its youth. there will be hiccups and that is why we have a Helium team that is working on resolving these issues.

It is also incomprehensible to me that new firmware versions are released that contain bugs, which negatively affect the rewards of the hotspot owners.

Unfortunately, that is the nature of software running on multiple hardware platforms. Bugs will appear in one form or another. It is given and be able to engage it. Use the tools they provided to you (Bobcats' just log into miner IP Address and check MINER, Do a Fast Resync or RESET Miner if you are having problems. If problems not resolved after 2-3 times, contact vendor for assistance and open PORT 22 as they can remote in and fix the issue for you. Learn to use your Vendor first before engaging here.

The big question for me is will helium compensate us for those failures?

Never. No one ever will.

[ElYoM]

I can't believe the amateurish behavior of OP. No MINER.JSON to back him up, No RESET Miners performed, No basic Troubleshooting before making an issue ticket. Have you even contacted your Vendor? If it was Bobcat, they would have looked at it within 48 business hours.

A support ticket was opened on November 02 with Bobcat without any response from them since. The troubleshooting has been done. The hotspots are perfectly functional, they just stopped all mining activity since October 27 and the deployment of the mandatory firmware imposed on manufacturers by Helium. Also, it is statistically impossible that 3 hotspots physically fail simultaneously. Moreover, I am not the only one concerned by this problem. Finally, if the developers want more details such as the MINER.JSON, I am at their disposal as indicated before but at some point it will have to be helped by someone, either by Helium or by Bobcat.

[madninja]

Please keep this repo clean from maker chat. This conversation is completely unrelated to miner development or the original issue, and should be taken elsewhere please.

[Harv2021]

Same issue here with Sensecap. I have an outdoor 5.8 dBi antenna on rooftop, LMR400 cable, not relayed, in major city, fast internet connection....and ever since the late October - Early November firmware updates my hotspot beacons has crashed significantly.

I've triple checked my wires and antenna, replaced/upgraded SD card....What can possibly explain the massive drop off in witnessing and being witnessed? I am well aware that earnings fluctuate...but a 53% drop in 30 days? Sensecap support has told me this is an issue with the Helium network...

Docile Clay Wolverine

[bitlite]

Bobcats were having multiple firmware updates (Since Oct 7, we went from 1.0.2.29 to 1.0.2.47 for 19 updates!) and some of those updates actually reset and resynced the blockchain. Therefore, as you can see you have 110k+ Bobcats out there vs your 36k Sensecaps. If 50+% of all Helium hardware is offline/resyncing, obviously you will have less rewards due to less beacon witnessing. Makes sense?

[ElYoM]

What information can be provided to help solve the problem? It's been almost a month now...

[gusosborne]

We know the blockchain was halted, with many issues affecting all services. They are just getting it stabilized now.

I believe we have to wait (at least a few days) for everything to stabilize completely, before any meaningful conclusions can be drawn. Everyone has been affected at this point.

I am glad to see good progress is being made.

[ElYoM]

The total stop of the activity on the graph dates from October 27th (and for my 3 hotspots simultaneously), so it has nothing to do with the problem of the last few days... Still zero PoC activity since.

[ElYoM]

Please keep this repo clean from maker chat. This conversation is completely unrelated to miner development or the original issue, and should be taken elsewhere please.

Many of us have noticed that those concerned by this zero PoC activity have mostly static full stack IPv4 addresses. As these addresses are often geolocated at a different location than the hotspots, could it be that anti-cheating measures wrongly consider our hotspots as cheating? This would be really problematic as it would be an extremely detrimental regression for those like me who need a static IP address because they host other services at home.

[ElYoM]

Fun Punch Ram : https://explorer.helium.com/hotspots/1123CoUsmffo1qTYEB8h1KsC7JrZ8dYk8Xz5EqQMDx5Jav7i3yTJ Last beacon sent : November 6, 2021

Small Charcoal Nightingale : https://explorer.helium.com/hotspots/11xXV4V8marSnCoVRbFWtZZgXstsXXY84P68Hn8boxVFUEoJDj9 Last beacon sent : November 6, 2021

Wonderful Lime Lynx : https://explorer.helium.com/hotspots/11ANbw714zMX4UgkwUVBkohdNMcAeZHBdBxYdqYqYHhtviXfYRW Last beacon sent : Novembeer 7, 2021

Obviously still 0 witness since October 27th when my 3 miners suddenly stopped mining for no reason.

Also, if I force my hotspots to be relayed to try to solve the problem as some people did on the official Helium Discord, my hotspots continue to do only challenges (always 0 beacon and 0 witness) but the gains for these challenges are only worth 0.003 HNT while when they are well configured and therefore not relayed, the challenges are worth 0.01 HNT. Frankly, nothing is right. Please help me, I'm going crazy. There is absolutely no logic to the observed behavior.

[ProTechEx]

I have the same issue that i have 0 witness and the hotspot doesn't beaconing. Before 1st Nov everything has work fine. I had around 50 witnesses, before 1st Nov, currently I should have a more because there a lot of new hotspots nearly.

The Hotspot is mounted on roof of building, arround 50m up from the ground with 360 degrees free sight.

I have check out again for wetness penetration and and whether the antenna is no longer strongly in place. But everything is okey without any issues.

I have read that many others had this problem after the firmware update, so it can only be related to a buggy firmware.

Nebra support doesn't reply my ticket since 14 days.

for me one thing is certain I will never buy Nebra hotspots again!

[ElYoM]

Indeed, we are not the only ones to have this problem.

I have seen many similar testimonies in the last weeks on the different official Discord of the manufacturers. These testimonies have been more or less well received on some channels (even insults have come from some moderators/developers!). The only thing I'm sure of today is that we won't have any support from the manufacturers who will just keep telling us to check our installations as if it wasn't already the first thing we did (and I repeat, my 3 hotspots were perfectly functional and configured, they just stopped all mining activity suddenly and simultaneously following the mandatory firmware update 2021.10.27.0)...

Anyway, the mining software part is under the responsibility of Helium and the appropriate firmware. Our miners don't mine from one day to the next after an update, so it's up to Helium to identify and correct this. We are of course at their disposal to provide them with as much information as possible to help them. I invite people who have the same problem (total stop of mining for weeks) to open an issue on this github to take the measure of the phenomenon.

So if you know or see other people having this problem, invite them to do so. ;)

[Mikael1993]

same here with 10 hotspots

[ElYoM]

Thnak you for your testimony @Mikael1993, we can see that you also your hotspots have simply stopped sending beacons like this one for example : https://explorer.helium.com/hotspots/11esEWwsHx7nPJnTrQ3kEcRG8JFpU2UPVP5uvL2atdzqw4j6gt3/activity

[Mikael1993]

@ElYoM Yes every 10 of my devices have been stopped since October 29th

[jsloane]

3 of my SenseCAPs have basically been flatlined since early November. Connected via 4G with Wireguard VPN for port forwarding. PeerPing on the SenseCAP dashboard indicates a ping in the 200-250ms range for the 3 hotspots.

[Mimik14]

https://github.com/helium/miner/issues/1208 same problem here

[J-Bot87]

Same problem !

able-porcelain-fly

[Mimik14]

[Mimik14]

i had two comebacks but only for one day...

[jakub2907]

Same here. 5 Hot Spot stopped beaconing around three weeks ago. Sporadic witnessing. One every two days of so. Perfect setup. Were working fine before. Static and Public IP. Port opened. Disco mode was showing that I can be witnessed.

[cryptowinteam]

Same issue here with 3 Flat Line Linxdot units.

I have sent the following to Linxdot support but offered here as support to the validity of the issue.

Can I ask you to please look for any reason that the following Linxdot Hotspots are all Zero activity. I understand that the Blockchain halt caused all activity to cease but all other Hotspots in my area are now back to normal activity. I am not sure if there is an OTA going out but according to WingWing in the Discord his unit is back to normal so this makes me think the issue may be limited to AU915 units.

https://explorer.helium.com/hotspots/11QYcTQBLmH9uv1RtquFKGD9aANC1GoubkyfHbBw2hG8dLp4Y9d

https://explorer.helium.com/hotspots/112eg93ty6KxvkjiA7hzS7Yt2UaeSM8RJUFJCpLPC7Ud8tQbwqUj

https://explorer.helium.com/hotspots/11np6gTbkrdsb4pNjsve8P1nbeWExyyvVit1BP1bJ6q6d42GKiv

[Thomas-LEB]

same problem, 3 Sensecap flatlined. Everything started after the firmware 2021.10.27.0 updated sent by helium. they became bricks after the firmware 2021.11.02.0, like 0 witness 0 beacons

[leyuansugar]

I have a feeling that the problem encountered with the hotspot after 10.27 is very similar to this previous problem #1041 , but at the time this was improved after it was discovered in September, so I don't know why this is happening again after 10.27. maybe not the same problem, but a very similar phenomenon.

[likeroman]

Same here. but really 0 Activity, api outage, errors, ip changes etc. histeric behaviour. resynch, reboot, fastsynch, all not helped till now. I try now asserting Location again,maby that is a solution. Wouldnt be the first time!

[Mikael1993]

@likeroman I did all this and it was fruitless

[ElYoM]

@likeroman Thank you for your testimony. Your hotspot has not sent beacons for several days (last sent on 09 november), so you are in the same case as many of us (for some, like me, the problem lasts even longer). Your hotspot has simply stopped mining for no reason.

[Mikael1993]

)

[likeroman]

@ElYoM thats why I was expecting get a new miner from bobcat. tec Support from bobcat tell its a helium issue. Helium blames the vendors. I askes bobcat foe get a new one, for what in this case else we got the guarantee! wrote many mails and texts. As I see they are not willing send us new miners.

[techygeekshome]

My miner has completely stopped working as well. I'm on the very latest firmware which was supposed to fix everything and nothing. I've got a Sensecap M1 and firmware is firmware_2021.11.21.2.

[ElYoM]

@ElYoM thats why I was expecting get a new miner from bobcat. tec Support from bobcat tell its a helium issue. Helium blames the vendors. I askes bobcat foe get a new one, for what in this case else we got the guarantee! wrote many mails and texts. As I see they are not willing send us new miners.

It is not a material problem. If you look at all the testimonials here, it shows that several manufacturers are affected, that the problems started simultaneously around October 28 for some and around November 06 for others. We also see that several countries are affected and that therefore the ISP are also different.

[Mikael1993]

Is port 44158 closed / timeout for you too?

22 | "closed/timeout" 44158 | "closed/timeout"

[licentia88]

Is port 44158 closed / timeout for you too?

22 | "closed/timeout" 44158 | "closed/timeout"

Yes, My 3 bobcats are suffering from this problem since november 6, are there not any fixes yet? how long will this problem continue? I'm very disappointed with helium support...

[Jamovich]

Hi, I too am having the same issues with my miners. Have 5 which have been displaying similar issues as described by everyone here. These are RAK miners.

[Mikael1993]

Therefore, the reason for not sending bacon is the close / timeout as port 44158 It is easier to say that we are relayed, but relayed are not written Of course, because of the helium network problem and its failure #1245

[aemla]

I have 20 hotspots that are on fast LTE connections with static IPs, none of them are relayed. Three of those hotspots have almost 0 activity (all of them are SenseCAPs) - they are only creating challenges, not beaconing and sometimes manage to get 1 witness per day. The first one does almost nothing since Nov 7-8, the second one since Nov 10-11 and the third one since Nov 17-18. All of them had a lot of activity before that and my other hotspots are still performing well while having similar setups. Discovery mode showed a lot of responses even while having all of these problems.

[tonypizza]

My good raks both began to flatline on Nov 7th or 8th. I’ve seen the 11/7 date quoted a lot (in addition to 10/27). Seems very odd/even suspicious that my good performers (frequently 1+ helium per day) totally flatlined whereas my weaker ones are still business as usual

[techygeekshome]

My port 44158 is open, everything on my config side looks perfect.