2021.12.14 Reports from Hotspot Owners

[abhay]

The core team is going to try a new way of tracking issues related to specific miner releases.

We will create release specific threads and see if that helps with miner support so that we don't loose track of issues from deployed Hotspots. Please verify that your Hotspot is on the current release version and if you're still having issues, please let us know on this thread.

RAK and OG Hotspots must be on 2021.12.14.1 and all other Hotspots could be on 2021.12.14.0.

Manufacturer specific issues should be directed to individual manufacturer support.

[bgdn021]

2021-12-15 05:42:27.116 [error] <0.24998.0>@blockchain_txndialer:dial:142 libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/11TAZindz9HqrPEKp6QtEgL6PeuPnXqc5HyM7WHub6v8YpYWLLm", TxnHash: <<145,139,100,163,127,55,124,251,3,182,120,179,149,158,93,173,161,85,91,30,81,235,251,188,161,24,16,116,184,254,183,246>>

Nothing changed

Pisces miner

Huge bamboo fox Miner Version: miner-arm64_2021.12.14.0_GA

[abhay]

2021-12-15 05:42:27.116 [error] <0.24998.0>@blockchain_txndialer:dial:142 libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/11TAZindz9HqrPEKp6QtEgL6PeuPnXqc5HyM7WHub6v8YpYWLLm", TxnHash: <<145,139,100,163,127,55,124,251,3,182,120,179,149,158,93,173,161,85,91,30,81,235,251,188,161,24,16,116,184,254,183,246>

This appears to be a dial to a Validator, likely to submit a poc_request_v1 or poc_receipt_v1 transaction as a challenger. Typically a miner will submit to multiple validators in the consensus group specifically to get around any issues that one Validator may not be listening or may drop the transaction.

This transaction ended up on chain: https://explorer.helium.com/txns/kYtko383fPsDtnizlZ5draFVWx5R6_u8oRgQdLj-t_Y

(val_miner@127.0.0.1)1> base64url:encode(<<145,139,100,163,127,55,124,251,3,182,120,179,149,158,93,173,161,85,91,30,81,235,251,188,161,24,16,116,184,254,183,246>>).
<<"kYtko383fPsDtnizlZ5draFVWx5R6_u8oRgQdLj-t_Y">>

tl;dr you can ignore not_found errors to validators in most cases as long as you're still seeing challenges on chain from your Hotspot.

[bgdn021]

witness log.txt error log.txt

Here are the logs.

[abhay]

witness log.txt error log.txt

Here are the logs.

These logs appear to be from a few days ago based on the timestamps.

[bgdn021]

Woops, sorry. Latest

witness log.txt error log.txt

[abhay]

Woops, sorry. Latest

witness log.txt error log.txt

Is there something in particular you're looking for me to inspect here? The supplied witness log is mostly empty and the error log mostly shows failed dials to validators (which is explained above), and one failed dial to a challengee which could be due to a variety of reasons including things out of your control (the Hotspot you were targeting was not connected at the time).

[bgdn021]

Witness logs don't get much lines. Usually just few times this line.

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

[abhay]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

[abhay]

2021-12-15 04:30:38.513 [error] <0.26129.0>@libp2p_stream_relay:handle_server_data:193 fail to pass request {libp2p_relay_bridge_cr_pb,"/p2p/112MSaVFdqmd8jKFBtFCfqWTmSp5pgq3eDbf7M5fHxqGaureJteC","/ip4/94.60.206.207/tcp/44158"} Server not found

It appears that the Challenger was not connected or was behind a relay that failed at the time of this witness submission.

[Mikael1993]

its crazy My 2 month chart Can we go back to previous updates? 😞😞😞😞 Because everything was fine

[abhay]

its crazy My 2 month chart Can we go back to previous updates? 😞😞😞😞 Because everything was fine

70% Hotspot growth in 2 months and significant networking improvements to witness receipt delivery. There is no "going back" unless you intend to turn off those 165K+ Hotspots as well.

[Mikael1993]

its crazy My 2 month chart Can we go back to previous updates? 😞😞😞😞 Because everything was fine

70% Hotspot growth in 2 months and significant networking improvements to witness receipt delivery. There is no "going back" unless you intend to turn off those 165K+ Hotspots as well.

Well, isn't it my right to witness once a day or send beacon? When I see a lot of people send bacon more than 4 or 5 times a day And are witness 10 times Our situation and theirs are exactly the same In terms of number of witness and network and configuration

[bgdn021]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

I've read it five times.

Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

[abhay]

Well, isn't it my right to witness once a day or send beacon?

No. The network doesn't blindly make guarantees or confer "rights". If you are targeted by another Hotspot, and you are online, and you are synced at that time, and you are dialable by the challenger, you will participate in that PoC activity. If your beacon's receipt is sent back to the Challenger and the Challenger is able to submit it to the Consensus Group, it will show up on the chain. If your challenge beacon is witnessed, that witnessing Hotspot is synced, and it is able to deliver a receipt of that witness event to the Challenger, then that witness receipt will also be on chain. These are the only guarantees.

[abhay]

Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

Assuming they are actually beaconing, and you're synced at the time of their beacons, I would wait a few days after the most recent release to figure out if any of the improvements that have been made to networking have improved your setup. Outside of networking, I'd also ensure that your antenna is connected properly and you have a good line of sight to other potential Hotspots.

[bgdn021]

Top kek. 👍

[Mikael1993]

Well, isn't it my right to witness once a day or send beacon?

No. The network doesn't blindly make guarantees or confer "rights". If you are targeted by another Hotspot, and you are online, and you are synced at that time, and you are dialable by the challenger, you will participate in that PoC activity. If your beacon's receipt is sent back to the Challenger and the Challenger is able to submit it to the Consensus Group, it will show up on the chain. If your challenge beacon is witnessed, that witnessing Hotspot is synced, and it is able to deliver a receipt of that witness event to the Challenger, then that witness receipt will also be on chain. These are the only guarantees.

Maybe I'm not as experienced as you But I am an IT and networking specialist And I know that: My port 44158 is open My hotspots are fully synchronized with the network I am not a relayed My internet speed, stability and ping are good Is there anything else that makes us unable to work like before? I guess that's a network problem

I hope to wait again for a few days after yesterday's update and see what happens

[burakozhan]

MAybe you have a hardware problem? Maybe the LoRa receiver in your miner is dead and can't receive anything? Maybe because the antenna port was open or shorted for a moment and transmission of LoRa Beacon completely reflected into antenna port and destroyed Receiver.

There are so many possibilities. Without further diagnostic data it does not matter if your ports are open and your internet is fast. It is impossible to tell what is wrong and why you are not witnessing other stations. Maybe your antenna is full with rain water and not working anymore?!? Who can tell?

Search youtube with "Squirrel fills Antenna with Acorns" also a possibility why RF system can fail.

[Mikael1993]

MAybe you have a hardware problem? Maybe the LoRa receiver in your miner is dead and can't receive anything? Maybe because the antenna port was open or shorted for a moment and transmission of LoRa Beacon completely reflected into antenna port and destroyed Receiver.

There are so many possibilities. Without further diagnostic data it does not matter if your ports are open and your internet is fast. It is impossible to tell what is wrong and why you are not witnessing other stations. Maybe your antenna is full with rain water and not working anymore?!? Who can tell?

Did 10 devices crash in an instant? Rainwater reached 10 antennas? Did 10 devices have an antenna problem at the same time? Mashallah

[CallableSwap]

Is it normal to have many witnesses but no witnessed at all? (Or just a few) https://explorer.helium.com/hotspots/1124XUwAHFzcfvfMDRAwndNyKZpXYSytifCov1m6bENSJRdxePm6/activity

[burakozhan]

Rainwater reached 10 antennas? Did 10 devices have an antenna problem at the same time?

Why not? Bad quality installation, no insulation on SMA Connectors, no proper water proofing; one day of really bad weather can take down many stations at once. It is possible. Seen it this summer in Germany. Couple hours of heavy rain, fiber optics torn out of ground, mobile base stations without power. Some stations even drowned completely.

Is it normal to have many witnesses but no witnessed at all? (Or just a few)

Check for GSM / 3G / LTE stations close by your miner. (You are in a large city, I bet there are many GSM stations nearby) Very probable their signal is so strong it blocks your miner from receiving other stations. So you can only transmit, but not hear. Probable course of action, relocate antenna of miner, so there is a thick concrete wall between it and the GSM station, and a SAW filter between antenna and miner. Do both, not only one. That might help a tiny bit.

[bgdn021]

Rainwater reached 10 antennas? Did 10 devices have an antenna problem at the same time?

Why not? Bad quality installation, no insulation on SMA Connectors, no proper water proofing; one day of really bad weather can take down many stations at once. It is possible. Seen it this summer in Germany. Couple hours of heavy rain, fiber optics torn out of ground, mobile base stations without power. Some stations even drowned completely.

Is it normal to have many witnesses but no witnessed at all? (Or just a few)

Check for GSM / 3G / LTE stations close by your miner. (You are in a large city, I bet there are many GSM stations nearby) Very probable their signal is so strong it blocks your miner from receiving other stations. So you can only transmit, but not hear. Probable course of action, relocate antenna of miner, so there is a thick concrete wall between it and the GSM station, and a SAW filter between antenna and miner. Do both, not only one. That might help a tiny bit.

Not true.

Would elaborate but don't have time.

Miner set up in the center if cell station working fine. Also cell station operating ~840 MHz

Need more?

Just admit you have problem with libp2p. You took code from someone else, now have a bug and don't know how to debug it because you didn't wrote it.

[CallableSwap]

Rainwater reached 10 antennas? Did 10 devices have an antenna problem at the same time?

Why not? Bad quality installation, no insulation on SMA Connectors, no proper water proofing; one day of really bad weather can take down many stations at once. It is possible. Seen it this summer in Germany. Couple hours of heavy rain, fiber optics torn out of ground, mobile base stations without power. Some stations even drowned completely.

Is it normal to have many witnesses but no witnessed at all? (Or just a few)

Check for GSM / 3G / LTE stations close by your miner. (You are in a large city, I bet there are many GSM stations nearby) Very probable their signal is so strong it blocks your miner from receiving other stations. So you can only transmit, but not hear. Probable course of action, relocate antenna of miner, so there is a thick concrete wall between it and the GSM station, and a SAW filter between antenna and miner. Do both, not only one. That might help a tiny bit.

Not true.

Would elaborate but don't have time.

Miner set up in the center if cell station working fine. Also cell station operating ~840 MHz

Need more?

Just admit you have problem with libp2p. You took code from someone else, now have a bug and don't know how to debug it because you didn't wrote it.

What’s your miner name?

[bgdn021]

Check recumbent sapphire snake and huge bamboo fox. This on the upper image is recumbent sapphire snake. Both of them set up in the middle of cell station with 10 or more panel antennas around them. Both were working fine. After the update and me having to move huge bamboo fox due to low transmit scale it stopped witnessing others. This is huge bamboo fox.

You can not tell me that this is not a good set up.

Also check

[ElYoM]

Hello,

Personally, my 3 hotspots had finally resumed normal activity with the application of firmware 2021.12.09 after more than a month of total stop. And since the application of firmware 2021.12.14.0, again a nice and big flatline. I'll wait a bit more but I hope it won't be a month of zero activity again...

Wonderful Lime Lynx (0 beacon sent since December, 12th) Small Charcoal Nightingale (0 beacon sent since December, 13th) Fun Punch Ram (0 beacon sent since December, 13th)

[amirhaleem]

Just admit you have problem with libp2p. You took code from someone else, now have a bug and don't know how to debug it because you didn't wrote it.

we’ve written every single line of code here, so that’s not it, also not helpful at all when trying to debug your setup.

[billgeoz]

Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

Assuming they are actually beaconing, and you're synced at the time of their beacons, I would wait a few days after the most recent release to figure out if any of the improvements that have been made to networking have improved your setup. Outside of networking, I'd also ensure that your antenna is connected properly and you have a good line of sight to other potential Hotspots.

Hotspot name: Bright Chrome Deer (so all statements bellow can be cross-checked)

1. No witnessed beacons in the last 13 days, and counting!!!
2. Sending out beacons work fine, and are being witnessed by others (max 18 per beacon).
3. Hotspot is situated in a very densely populated area of a major metropolitan area.
4. Antenna connections triple-checked and are OK.
5. No change during the last 2 Helium version update rollout.

p.s. As far as I can understand the Semtech SX1308 RF module has intergraded RX/TX stages, so if TX beacons are sent out and being witnessed by other, there is no reason to believe the RX side is damaged or degraded somehow.

[ykuntsevych]

Hi @abhay

Miner name: Faint Denim Parrot (Kyiv, Kyiv city, Ukraine)

• Static IP: 62.216.54.8
• Ports 44158 + wider range forwarded. All mentioned in various docs , upnp - off, internet on LAN
• Not relayed
• Dashboard Version: v0.2.8 Miner Version: miner-arm64_2021.12.14.0_GA
• My roof is higher than neighbours in same hex and is approx 80 meters. Neighbors are underneath my hotspot and getting 15-20$/day .
• Errors, Errors, errors (logs attached).

log_errors.txt log_miner.txt log_witness.txt

Please take a look, maybe question?

[CallableSwap]

Hello All, I still have this issue many times a day. Isn’t it supposed to have been fixed?

@miner_onion_server:send_witness:243 failed to dial challenger

and eventuality leading to

@miner_onion_server:send_witness:207 failed to send witness, max retry

[ykuntsevych]

Hello All, I still have this issue many times a day. Isn’t it supposed to have been fixed?

@miner_onion_server:send_witness:243 failed to dial challenger

and eventuality leading to

@miner_onion_server:send_witness:207 failed to send witness, max retry

Looks like we are getting same issues. Hopefully the team will come up with a fix

[amirhaleem]

Hello All, I still have this issue many times a day. Isn’t it supposed to have been fixed?

@miner_onion_server:send_witness:243 failed to dial challenger

and eventuality leading to

@miner_onion_server:send_witness:207 failed to send witness, max retry

You will always have some of these, as you won’t always be able to dial the challenger. It’s happening about 11% of the time network-wide and probably won’t change. Generally they can be ignored.

[ykuntsevych]

Hi @amirhaleem what do you think of my case (above) ? ))

[Spotje1981]

Joyous Cider Sheep - Pisces P100 miner

Witness Logs: 2021-12-15 19:45:18.453 6 [warning] <0.9691.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/11nmKDEFQ99DuMPc4H4T4idvHpxm9ap5DTVYqoD8136fTdxZAby": not_found 2021-12-15 18:59:19.634 6 [warning] <0.7325.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/1128dK8Q3aqakHwy5HH4JxmdovtEGiKW9pQFfRDKdooCc7yJR6gt": not_found 2021-12-15 18:51:50.019 6 [warning] <0.6895.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/112LYjukg5t1Hh6YcrzDonn9h1MoMfCE7EEY35T75CQBHseC4aBi": not_found 2021-12-15 18:34:42.299 6 [error] <0.5746.1>@miner_onion_server:send_witness:207 failed to send witness, max retry

Still wating on Create a Challenge step...

[Crypto517]

Hi there, I am hoping someone can help me. My MNTD Rak wireless miner has produced almost no HNT over the past 5 days. Everything looks good on my end ; diagnostics, open port, etc. i have attached info on everything. Hopefully someone can help me figure out what is going on.

Thank you in advance!

[billgeoz]

Hi there, I am hoping someone can help me. My MNTD Rak wireless miner has produced almost no HNT over the past 5 days. Everything looks good on my end ; diagnostics, open port, etc. i have attached info on everything. Hopefully someone can help me figure out what is going on.

Thank you in advance!

Just wait 2-3 days more. It's too soon

[abhay]

Joyous Cider Sheep - Pisces P100 miner

Witness Logs: 2021-12-15 19:45:18.453 6 [warning] <0.9691.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/11nmKDEFQ99DuMPc4H4T4idvHpxm9ap5DTVYqoD8136fTdxZAby": not_found 2021-12-15 18:59:19.634 6 [warning] <0.7325.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/1128dK8Q3aqakHwy5HH4JxmdovtEGiKW9pQFfRDKdooCc7yJR6gt": not_found 2021-12-15 18:51:50.019 6 [warning] <0.6895.1>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/112LYjukg5t1Hh6YcrzDonn9h1MoMfCE7EEY35T75CQBHseC4aBi": not_found 2021-12-15 18:34:42.299 6 [error] <0.5746.1>@miner_onion_server:send_witness:207 failed to send witness, max retry

Still wating on Create a Challenge step...

This looks like the challenger is not dialable for whatever reason when you are sending witnesses. That said, I see a ton of witness and beacon activity on the sheep so I don't think there's anything wrong here and you created a challenge 21 minutes ago (as of this message).

[ykuntsevych]

Hi @abhay what do you think of Miner name: Faint Denim Parrot, logs attached above, previously?

[Spotje1981]

But this is for more than 4 days now and still not created a challenge. Logs have the same issue as ykuntsevych.

[jnspsn]

I suspect that this one can't keep up with the blockchain again https://explorer.helium.com/hotspots/112pEwmpuJrn7trHf1xvzz6j3nSKEhwNjqNaL7XwhP7V138PdFA1

[bkauii]

Ever since the last update I have several hitspots that are only doing data packet transfer

[amirhaleem]

We’re aware of an issue with some Bobcats not transmitting since PoC v11. They are releasing an OTA soon to fix this.

[bkauii]

I do not have any bobcats

We’re aware of an issue with some Bobcats not transmitting since PoC v11. They are releasing an OTA soon to fix this.

[ykuntsevych]

So Рш @abhay @amirhaleem

log2_errors.txt log2_witness.txt log2_miner.txt

What would you say about Miner name: Faint Denim Parrot, logs attached ? Pisces P100, LAN, PORTS forwarded, not relayed, checked antenna everything else inside, Nice placement, above other buildings and other locations I can see on the map getting 10 times more in my hex. Latest miner 14.12 GA Only one wall behind so I can say 300-320 C visibility around out of 360. There is a way to put it a bit higher for 360C view but does not make sense if it's not working in this setup :-(

PS: other miners in my hex get 10-20$/day and I assume have never updated software as I did not notice and flat in their statistics. Please help.

[ElYoM]

Hello Helium team,

I was really happy to find a normal functioning of my 3 hotspots with the firmware 2021.12.06.0 after a month of total inactivity because of the problem with DNS and SeedNodes but I think maybe I was too optimistic and yet I would have been really happy to be able to come here to announce you that it was finally solved. Unfortunately, this is not the case anymore after only a few days of normal activity, the joy is now replaced by fatigue...

The 3 hotspots concerned are Bobcat hotspots, with updated Bobcat firmware (1.0.2.64, the firmware that was supposed to solve the new problem of Bobcat not transmitting), with updated Helium firmware (2021.12.14.0), are not relayed and are each connected in Ethernet on a minimum 1Gbps fiber connection.

Wonderful Lime Lynx (last beacon sent : 12 Dec 2021, 9:29am) Small Charcoal Nightingale (last beacon sent : 13 Dec 2021, 5:23am) Fun Punch Ram (last beacon sent: 13 Dec 2021, 5:38pm)

What specific information can we provide you with to solve the problem permanently or at least to help you diagnose it, so that we don't end up with another month without any activity on our hotspots (I'm obviously not the only one affected by this problem and the previous ones)?

Thanks in advance for solving the problem. Come on little Helium Santa, please bring us hotspots that finally do what they are also supposed to do besides providing network coverage: mine.

[cingscy]

We’re aware of an issue with some Bobcats not transmitting since PoC v11. They are releasing an OTA soon to fix this.

Unfortunately we have the same issue with Pisces. And we are waiting a movement at every update, but it has been getting worse since 2021.10.27.0 firmware release.

[bgdn021]

Pisces problem +1

[amirhaleem]

@ElYoM I’ve asked the Bobcat team if they can share any logs to understand what’s happening with your hotspots. It’s not something we can diagnose on our own at this point

[CallableSwap]

Hi, Look @ my miner in the explorer: some beacons, lots of witnesses, but no witnessed for 6d, expect one invalid… I checked and there is no 3G/4G/5G antenna near the hotspot. I don’t understand what’s going wrong… please help!

https://explorer.helium.com/hotspots/1124XUwAHFzcfvfMDRAwndNyKZpXYSytifCov1m6bENSJRdxePm6/activity

[ElYoM]

@ElYoM I’ve asked the Bobcat team if they can share any logs to understand what’s happening with your hotspots. It’s not something we can diagnose on our own at this point

Thank you @amirhaleem. Do not hesitate if I can also provide you with any information useful to the diagnosis, I am available. For your information, the Bobcat 1.0.2.65 firmware update seems to have allowed two of my hotspots to emit beacons again (Fun Punch Ram and Small Charcoal Nightingale). Unfortunately, the beacons are all coming back with 0 witness when before it was totally OK. On the other hand, for the 3rd hotspot (Wonderful Lime Lynx), even the Bobcat 1.0.2.65 update didn't solve anything: complete flat line and no beacon emitted since Dec 12, 2021, 9:29am.

The Bobcat firmware update v1.0.2.65 was supposed to fix the following problem in particular if it can help you: "LoRa Packet Forwarder Update Backport Function - This is expected to resolve witness/beacon issues some users have been experiencing in the past few days."

EDIT : Thank you @amirhaleem. Finally, it took a few days but the activity has resumed on my 3 hotspots. I hope it will be the same for the other people who had a similar problem. Anyway, it's a good idea to open a GitHub thread for each firmware released by Helium to eventually allow a quick feedback of any problem encountered. However, I am watching to see if the situation is stable (obviously taking into account the usual fluctuations in the rewards and their continued decline).

[xystren]

I have had all my witnesses drop off, and am down to zero now. Searched through my activity, and the last "sent beacon" was on the 13th of December at 7:50am US Central Time. Since then, my witnesses have been falling off. The Miner (Brilliant Marigold Elephant) continues to witness beacons, challenge beacon, and receive payments.

Seems to have occurred after the PoCv11 deployment. I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational.

Hope this can be researched and a solution found.