2021.12.14 Reports from Hotspot Owners

[abhay]

The core team is going to try a new way of tracking issues related to specific miner releases.

We will create release specific threads and see if that helps with miner support so that we don't loose track of issues from deployed Hotspots. Please verify that your Hotspot is on the current release version and if you're still having issues, please let us know on this thread.

RAK and OG Hotspots must be on 2021.12.14.1 and all other Hotspots could be on 2021.12.14.0.

Manufacturer specific issues should be directed to individual manufacturer support.

[runningonEmpte]

I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational and I have no beacons or witnesses (not witnessing either) only sending challenges. Stopped working after the PoCv11 update.. will nebra get us a fix for this.. I see bobcat has already released a fixed update.

[xystren]

I have had all my witnesses drop off, and am down to zero now. Searched through my activity, and the last "sent beacon" was on the 13th of December at 7:50am US Central Time. Since then, my witnesses have been falling off. The Miner (Brilliant Marigold Elephant) continues to witness beacons, challenge beacon, and receive payments.

Seems to have occurred after the PoCv11 deployment. I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational.

Hope this can be researched and a solution found.

Also now it seems my RAK v2 is doing a similar thing. The last beacon sent was December 15th (Short Cerulean Mandrill)

Having a background in IT, it is extraordinary frustrating not being able to see "miner logs" to see where issues are occurring. It would be nice to see that "such and such" activity was sent to the network, and if it isn't showing in explorer or other then we know the network is lagging or behind. The diagnostic pages are extremely limited in usefulness beyond the most basic troubleshooting. My diagnostics says every thing is functioning perfectly.

From the discords, and others, this appears to not be an isolated issue - not affecting everyone, but when it takes 5 days for your witnesses to fall off the table, from ~300 and you drop to 0 after the 5th day after a firmware deployment and network upgrade (PoCv11). Earning were noted to be lower since the changes, and time was given to allow the network/infrastructure to stabilize. That stabilization period has reveled greater issues. And it is concerning that there seems to be no acknowledgement of such an issue does in fact exist.

[amirhaleem]

I have had all my witnesses drop off, and am down to zero now. Searched through my activity, and the last "sent beacon" was on the 13th of December at 7:50am US Central Time. Since then, my witnesses have been falling off. The Miner (Brilliant Marigold Elephant) continues to witness beacons, challenge beacon, and receive payments.

Seems to have occurred after the PoCv11 deployment. I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational.

Hope this can be researched and a solution found.

looks like this hotspot is functioning normally and just beaconed a few hours ago

[amirhaleem]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

I've read it five times.

Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

also appears to be working well now?

https://explorer.helium.com/hotspots/112okzK1NiDyxpyLoBZadCCsEjjveRkXJavLCQcBjHsxwH8BatLk/activity

[runningonEmpte]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

I've read it five times. Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

also appears to be working well now?

https://explorer.helium.com/hotspots/112okzK1NiDyxpyLoBZadCCsEjjveRkXJavLCQcBjHsxwH8BatLk/activity

Any ideas why mine is still not witnessing or receiving any challenges? Everything checks out and I have line of sight. I’m just creating challenges

https://explorer.helium.com/hotspots/11oHpRPvYSNfErkAN7UMNhPQh86ndRSbAiSiJGXFdtZGhSqzRTQ/activity

[thardie]

I think this may be related to the issue (and workaround) in #1307

I had dropped to zero, and after troubleshooting and using the workaround, I'm back up to levels above even before my fix.

[bgdn021]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

I've read it five times. Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

also appears to be working well now?

https://explorer.helium.com/hotspots/112okzK1NiDyxpyLoBZadCCsEjjveRkXJavLCQcBjHsxwH8BatLk/activity

2 times witnessing in 17 days doesn't seem good to me.

[error log.txt] (https://github.com/helium/miner/files/7747542/error.log.txt)

[shawaj]

Error logs get full of not_found error and I didn't witness anyone in two weeks. check my miners all activity on the explorer.

Sounds like you're still not reading what I said about not_found errors. Please re-read my comment above about not_found errors and Validators.

I've read it five times. Please answer me just one thing. Why haven't I witnessed someone's beacon in two weeks?

also appears to be working well now? https://explorer.helium.com/hotspots/112okzK1NiDyxpyLoBZadCCsEjjveRkXJavLCQcBjHsxwH8BatLk/activity

Any ideas why mine is still not witnessing or receiving any challenges? Everything checks out and I have line of sight. I’m just creating challenges

https://explorer.helium.com/hotspots/11oHpRPvYSNfErkAN7UMNhPQh86ndRSbAiSiJGXFdtZGhSqzRTQ/activity

Seems to have witnessed and beaconed now 👍

[Erekle2]

Sorry for newbie question, can someone advice how I can run error/miner/witness logs from COTX miner? I will be eternally thankful <3

[shawaj]

Sorry for newbie question, can someone advice how I can run error/miner/witness logs from COTX miner? I will be eternally thankful <3

You would probably need to SSH into the unit to get them.

You probably need to ask Cotx if it's possible for you to do that

[vanjulio]

I have had all my witnesses drop off, and am down to zero now. Searched through my activity, and the last "sent beacon" was on the 13th of December at 7:50am US Central Time. Since then, my witnesses have been falling off. The Miner (Brilliant Marigold Elephant) continues to witness beacons, challenge beacon, and receive payments. Seems to have occurred after the PoCv11 deployment. I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational. Hope this can be researched and a solution found.

Also now it seems my RAK v2 is doing a similar thing. The last beacon sent was December 15th (Short Cerulean Mandrill)

Having a background in IT, it is extraordinary frustrating not being able to see "miner logs" to see where issues are occurring. It would be nice to see that "such and such" activity was sent to the network, and if it isn't showing in explorer or other then we know the network is lagging or behind. The diagnostic pages are extremely limited in usefulness beyond the most basic troubleshooting. My diagnostics says every thing is functioning perfectly.

From the discords, and others, this appears to not be an isolated issue - not affecting everyone, but when it takes 5 days for your witnesses to fall off the table, from ~300 and you drop to 0 after the 5th day after a firmware deployment and network upgrade (PoCv11). Earning were noted to be lower since the changes, and time was given to allow the network/infrastructure to stabilize. That stabilization period has reveled greater issues. And it is concerning that there seems to be no acknowledgement of such an issue does in fact exist.

Is it possible we are getting same issue as #1305 ? (https://github.com/helium/miner/issues/1305) I was joking on reddit about this but seriously the Cayman Islands miners look exactly like my hotspot since November.

[bkauii]

Wobbly Sangria Jellyfish finally started witnessing but it won't send beacons

[tathatvamasi]

My hotspot Sparkley Felt Giraffe isn't sending beacons or witnessing

[xystren]

I have had all my witnesses drop off, and am down to zero now. Searched through my activity, and the last "sent beacon" was on the 13th of December at 7:50am US Central Time. Since then, my witnesses have been falling off. The Miner (Brilliant Marigold Elephant) continues to witness beacons, challenge beacon, and receive payments.

Seems to have occurred after the PoCv11 deployment. I am on a Nebra Indoor Hotspot Gen 1. Diagnostics report 100% synced. Firmware 2021.12.14.0, LoRa is operational.

Hope this can be researched and a solution found.

UPDATE: Well, a updated firmware dash-one (2021.12.14.0-1) suddenly appeared and I started beaconing again. My witnesses are returning, but the earning have increased slightly from the update, but still nowhere near what I was earning a month ago.. I will give it some more time to see if it returns/stabilizes some.

[ProTechEx]

Hi, my Nebra Outdoor, doesn't has Witnesses any more and it doesn't sent Beacons anymore: Witty Flint Platypus https://explorer.helium.com/hotspots/11GE32qw7wXo3isB9NTThi851UMiZUF2DhVvoUAwpgQCYvwMH5H

However, this may not be possible because there are many hotspots arround me. The antenna is tight and also Hotspot just witnessing beacons from other hotspots.

[hcl-83]

Hello, My rak v2 stop beaconing and drop from 0.5hnt/24h to 0.01 i dont change anything. Gsm connection public static ip nat none. Soft version 14.1 Breezy Bamboo Ape

[abhay]

Hello,

My rak v2 stop beaconing and drop from 0.5hnt/24h to 0.01 i dont change anything.

Gsm connection public static ip nat none.

Soft version 14.1

Breezy Bamboo Ape

It appears to be just fine, it is being challenged, being witnessed, and is witnessing other challenges. https://explorer.helium.com/hotspots/112eWJoRTVngRKzZoeJ7uZNHS7dPMXe9k9HPxw8NfUHv2DL2aHBr/activity

[abhay]

Hi, my Nebra Outdoor, doesn't has Witnesses any more and it doesn't sent Beacons anymore:

Witty Flint Platypus

https://explorer.helium.com/hotspots/11GE32qw7wXo3isB9NTThi851UMiZUF2DhVvoUAwpgQCYvwMH5H

Within the last 24 hours, it appears to be issuing challenges, receiving them, and witnessing other challenges.

[abhay]

UPDATE:

Well, a updated firmware dash-one (2021.12.14.0-1) suddenly appeared and I started beaconing again. My witnesses are returning, but the earning have increased slightly from the update, but still nowhere near what I was earning a month ago.. I will give it some more time to see if it returns/stabilizes some.

Great to hear!

[abhay]

Sparkley Felt Giraffe

It appears to be completely offline. Have you been able to do locally diagnose it over Bluetooth? Is it running the latest firmware and is it synced to the chain?

[bkauii]

Sparkley Felt Giraffe

It appears to be completely offline. Have you been able to do locally diagnose it over Bluetooth? Is it running the latest firmware and is it synced to the chain?

It wasnt offline this morning. Ill check and get back to you.

[bkauii]

Sparkley Felt Giraffe

It appears to be completely offline. Have you been able to do locally diagnose it over Bluetooth? Is it running the latest firmware and is it synced to the chain?

It wasnt offline this morning. Ill check and get back to you.

It's back online

[CallableSwap]

Hello All,

@abhay Could you tell me why I don’t see the following witnessed in Helium Explorer All Activity ?

https://explorer.helium.com/hotspots/1124XUwAHFzcfvfMDRAwndNyKZpXYSytifCov1m6bENSJRdxePm6/activity

2021-12-23 09:19:25.326 6 [info] <0.1634.0>@miner_onion_server:decrypt:373 decrypted a layer: <<227,57>> received via p2p 2021-12-23 07:42:42.637 6 [info] <0.17257.22>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 07:42:41.198 6 [info] <0.17197.22>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:11.198 6 [warning] <0.17197.22>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/11yHetZYUaicGi1fGHMmMVU5XvVeu1QnBYRSWns4HdRodk7XtW4": not_found 2021-12-23 07:42:10.763 6 [info] <0.17197.22>@miner_onion_server:send_witness:188 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 05:46:44.579 6 [info] <0.9051.22>@miner_onion_handler:handle_info:78 client sending data: <<…74>> 2021-12-23 02:39:19.291 6 [info] <0.31765.21>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 02:39:17.751 6 [info] <0.31750.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:00:16.167 6 [info] <0.1634.0>@miner_onion_server:decrypt:414 sending receipt with observed power: 16 with radio power 12

Or this one :

2021-12-23 01:26:58.324 6 [info] <0.27868.21>@miner_poc_handler:handle_info:74 client sending data: <<18,178,1,10,33,0,139,179,134,224,125,33,175,115,133,178,57,240,13,217,112,6,54,214,248,229,140,142,134,63,205,237,221,193,81,174,149,93,16,199,207,213,180,227,189,207,225,22,24,145,1,34,32,243,242,142,172,41,141,152,166,241,248,97,249,116,140,88,214,154,55,232,29,120,132,33,46,66,175,222,27,53,26,87,91,42,71,48,69,2,33,0,224,184,57,228,129,142,23,29,241,121,147,99,241,211,254,220,217,16,229,135,47,54,193,66,183,236,139,172,195,168,131,48,2,32,54,113,223,26,198,160,25,172,176,29,46,187,213,217,241,47,83,193,30,111,247,162,211,128,167,102,101,207,120,249,120,65,53,102,102,134,193,61,102,6,89,68,64,5,74,9,83,70,49,50,66,87,49,50,53>> 2021-12-23 01:26:52.740 6 [info] <0.27681.21>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.739 6 [warning] <0.27681.21>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/112RpuacdsTZ3K6LkWZQdSeAMJHE25cMhVspcihgKmCCAf3vS2Hh": not_found 2021-12-23 01:26:22.620 6 [info] <0.27681.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8

[heliumminermh]

@abhay @amirhaleem Could you tell me anyone can do like this?? Because earning is increasing.

I think poc 11 not working with these hotspots these are 23 miner in Gujarat India, and located UP, India You can check there IP It's Local GTPL(means from Gujarat)

https://explorer.helium.com/hotspots/11JM1rYTBwaRQgfdB6buZ7FxbcDajC8HGM72LaHvUcQN6RHx3ys

[thardie]

Hello All,

@abhay Could you tell me why I don’t see the following witnessed in Helium Explorer All Activity ?

https://explorer.helium.com/hotspots/1124XUwAHFzcfvfMDRAwndNyKZpXYSytifCov1m6bENSJRdxePm6/activity

2021-12-23 09:19:25.326 6 [info] <0.1634.0>@miner_onion_server:decrypt:373 decrypted a layer: <<227,57>> received via p2p 2021-12-23 07:42:42.637 6 [info] <0.17257.22>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 07:42:41.198 6 [info] <0.17197.22>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:11.198 6 [warning] <0.17197.22>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/11yHetZYUaicGi1fGHMmMVU5XvVeu1QnBYRSWns4HdRodk7XtW4": not_found 2021-12-23 07:42:10.763 6 [info] <0.17197.22>@miner_onion_server:send_witness:188 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 05:46:44.579 6 [info] <0.9051.22>@miner_onion_handler:handle_info:78 client sending data: <<…74>> 2021-12-23 02:39:19.291 6 [info] <0.31765.21>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 02:39:17.751 6 [info] <0.31750.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:00:16.167 6 [info] <0.1634.0>@miner_onion_server:decrypt:414 sending receipt with observed power: 16 with radio power 12

Or this one :

2021-12-23 01:26:58.324 6 [info] <0.27868.21>@miner_poc_handler:handle_info:74 client sending data: <<18,178,1,10,33,0,139,179,134,224,125,33,175,115,133,178,57,240,13,217,112,6,54,214,248,229,140,142,134,63,205,237,221,193,81,174,149,93,16,199,207,213,180,227,189,207,225,22,24,145,1,34,32,243,242,142,172,41,141,152,166,241,248,97,249,116,140,88,214,154,55,232,29,120,132,33,46,66,175,222,27,53,26,87,91,42,71,48,69,2,33,0,224,184,57,228,129,142,23,29,241,121,147,99,241,211,254,220,217,16,229,135,47,54,193,66,183,236,139,172,195,168,131,48,2,32,54,113,223,26,198,160,25,172,176,29,46,187,213,217,241,47,83,193,30,111,247,162,211,128,167,102,101,207,120,249,120,65,53,102,102,134,193,61,102,6,89,68,64,5,74,9,83,70,49,50,66,87,49,50,53>> 2021-12-23 01:26:52.740 6 [info] <0.27681.21>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.739 6 [warning] <0.27681.21>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/112RpuacdsTZ3K6LkWZQdSeAMJHE25cMhVspcihgKmCCAf3vS2Hh": not_found 2021-12-23 01:26:22.620 6 [info] <0.27681.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8

You are suffering from the peer not found issue. See the issue I linked above for a workaround.

[CallableSwap]

Hello All, @abhay Could you tell me why I don’t see the following witnessed in Helium Explorer All Activity ? https://explorer.helium.com/hotspots/1124XUwAHFzcfvfMDRAwndNyKZpXYSytifCov1m6bENSJRdxePm6/activity 2021-12-23 09:19:25.326 6 [info] <0.1634.0>@miner_onion_server:decrypt:373 decrypted a layer: <<227,57>> received via p2p 2021-12-23 07:42:42.637 6 [info] <0.17257.22>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 07:42:41.198 6 [info] <0.17197.22>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:11.198 6 [warning] <0.17197.22>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/11yHetZYUaicGi1fGHMmMVU5XvVeu1QnBYRSWns4HdRodk7XtW4": not_found 2021-12-23 07:42:10.763 6 [info] <0.17197.22>@miner_onion_server:send_witness:188 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 07:42:10.762 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -69, Frequency: 867.5, SNR: -10.2 2021-12-23 05:46:44.579 6 [info] <0.9051.22>@miner_onion_handler:handle_info:78 client sending data: <<…74>> 2021-12-23 02:39:19.291 6 [info] <0.31765.21>@miner_poc_handler:handle_info:74 client sending data: <<…0,53>> 2021-12-23 02:39:17.751 6 [info] <0.31750.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 02:39:17.751 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -74, Frequency: 868.1, SNR: -16.8 2021-12-23 02:00:16.167 6 [info] <0.1634.0>@miner_onion_server:decrypt:414 sending receipt with observed power: 16 with radio power 12 Or this one : 2021-12-23 01:26:58.324 6 [info] <0.27868.21>@miner_poc_handler:handle_info:74 client sending data: <<18,178,1,10,33,0,139,179,134,224,125,33,175,115,133,178,57,240,13,217,112,6,54,214,248,229,140,142,134,63,205,237,221,193,81,174,149,93,16,199,207,213,180,227,189,207,225,22,24,145,1,34,32,243,242,142,172,41,141,152,166,241,248,97,249,116,140,88,214,154,55,232,29,120,132,33,46,66,175,222,27,53,26,87,91,42,71,48,69,2,33,0,224,184,57,228,129,142,23,29,241,121,147,99,241,211,254,220,217,16,229,135,47,54,193,66,183,236,139,172,195,168,131,48,2,32,54,113,223,26,198,160,25,172,176,29,46,187,213,217,241,47,83,193,30,111,247,162,211,128,167,102,101,207,120,249,120,65,53,102,102,134,193,61,102,6,89,68,64,5,74,9,83,70,49,50,66,87,49,50,53>> 2021-12-23 01:26:52.740 6 [info] <0.27681.21>@miner_onion_server:send_witness:246 re-sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.739 6 [warning] <0.27681.21>@miner_onion_server:send_witness:243 failed to dial challenger "/p2p/112RpuacdsTZ3K6LkWZQdSeAMJHE25cMhVspcihgKmCCAf3vS2Hh": not_found 2021-12-23 01:26:22.620 6 [info] <0.27681.21>@miner_onion_server:send_witness:188 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:370 could not decrypt packet received via radio: treating as a witness 2021-12-23 01:26:22.620 6 [info] <0.1634.0>@miner_onion_server:decrypt:360 sending witness at RSSI: -73, Frequency: 868.1, SNR: -16.8

You are suffering from the peer not found issue. See the issue I linked above for a workaround.

Thanks @thardie!

Isn’t it possible to completely refresh the cache (from time to time)?

Cheers

[unzael]

Hello team,

Seems like I am also having the same issues some have with my bobcat. There hasn't been any Witnessed Beacon activity for the last couple of days. I do send beacons however no witnessed beacon activity. Is there anything I can do on my side to troubleshoot this? Apoligies if I am missing something obvious.

Blunt Bronze Giraffe - https://explorer.helium.com/hotspots/11L4TbQfKuPw7pBigD6Mh94FC6N3rYTJh8vU1TCdPfEgoYVdHn3

[thardie]

Thanks @thardie!

Isn’t it possible to completely refresh the cache (from time to time)?

Cheers

The closest I know of, which I've never seen help, is to reset the relay. You can try docker exec miner miner peer relay_reset

I suspect this problem will just get worse and worse since it looks to me that Helium's Erlang p2p or libp2p itself can't handle the size of their network. They are working on a replacement (gateway-rs) which will remove the need for p2p on hotspots, but it'll take a while to get that finished, and then get things like explorer to use the new architecture. They're ALL based on libp2p right now.

[Erekle2]

Hello all - I created separate miner issue post already, did not realise I could post my hotspot issue here.

Long story short my two miners can not be seen on the network. They are able to send challenges and witness , but all of the challenges come back 0 and it never beacons. It has been confirmed with 2 miner images that two Hotspots are not available to the network, Peer connect via ip works, connecting peer book peer ping and peer connect all fail with the hotspots /p2p/ address.

Any help is much appreciated.

https://github.com/helium/miner/issues/1310

[yooitsgreg]

Hi all - Looking for help troubleshooting why my hotspot has hasn't sent a beacon in over 5 days. In addition, I also see in Explorer under All Activity many "Created Challenge" entries but rarely a corresponding "Challenged Beaconer" entry.

Looking for information on how to troubleshoot the lack of Challenges / Sent Beacons so I can test out a new antenna at my location (not worried about no Witness activity atm)

Bluetooth Diagnostics, Hotspotty.net, Heliumstatus.io all show everything is setup correctly/no issues.

Atomic Mocha Cheetah https://explorer.helium.com/hotspots/112eWyTrhGz6qYM7YSRGvmC1aeW7QUdfVeXALDiGMwCX9sn8c1gp/activity

[bkauii]

I am still having troubles with Wobbly Sangria Jellyfish not sending beacons. I'm having am issue with Sparkly Felt Giraffe doing nothing

[hcl-83]

Hello I have 10 rak v2. 3 days ago 4 of them: High Amber Chimpanzee Sneaky Cotton Scorpion Dancing Lipstick Puppy Gorgeous Dijon Wasp Stop mining :/. Explorer says thet they sinking and online. What to do to work again ?? Regards Kamil :)

[bysoclose]

Herkese merhaba - Erişim noktamın 5 günden fazla bir süredir neden bir işaret göndermediğini gidermek için yardım arıyorum. Ek olarak, Explorer'da Tüm Faaliyetler altında birçok "Created Challenge" girişi, ancak nadiren buna karşılık gelen bir "Challenged Beaconer" girişi görüyorum.

Bulunduğum yerde yeni bir anteni test edebilmem için Zorluklar / Gönderilen İşaretlerin eksikliğini nasıl giderebileceğim hakkında bilgi arıyorum (Tanık etkinliği ATM'sinden endişe etmiyorum)

Bluetooth Diagnostics, Hotspotty.net, Heliumstatus.io'nun tümü, her şeyin doğru şekilde kurulduğunu/sorun olmadığını gösterir.

Atomik Mocha Cheetah https://explorer.helium.com/hotspots/112eWyTrhGz6qYM7YSRGvmC1aeW7QUdfVeXALDiGMwCX9sn8c1gp/activity

I'm following you but it's too bad you still can't get a result, I'm having the same problem and I'm following yours.

https://explorer.helium.com/hotspots/112jKgdfZnHKs1wXe1toKw7YyAkYK88DKWvtk6soLZPEyAcvmZWk

[amanro]

Hi,

I'm not getting any witnesses on my hotspot 'Upbeat Golden Seal" It's 6 kms away from my other hotspot "Gorgeous Inky Squid" both using Spectrum and port forwarding is working fine. I'm the only one in both hexes. Upbeat Golden Seals did not receive any witnesses inside with stock antenna, I moved it out and same thing. I upgraded to a 5.8 Dbi Antenna and mounted it to my roof, which is on the highest spot possible with clear view of all the hotspots behind me that are getting hundreds of witnesses. I have a ticket open with Bobcat support and they asked me to post here as well. Please let me know if I need to provide anything else?

Thanks

[o1298098]

Hi,

I'm not getting any witnesses on my hotspot 'Upbeat Golden Seal" It's 6 kms away from my other hotspot "Gorgeous Inky Squid" both using Spectrum and port forwarding is working fine. I'm the only one in both hexes. Upbeat Golden Seals did not receive any witnesses inside with stock antenna, I moved it out and same thing. I upgraded to a 5.8 Dbi Antenna and mounted it to my roof, which is on the highest spot possible with clear view of all the hotspots behind me that are getting hundreds of witnesses. I have a ticket open with Bobcat support and they asked me to post here as well. Please let me know if I need to provide anything else?

Thanks

same

[bysoclose]

Existing problem Continuing. sudo nano /home/pi/hnt/paket/paket/packet_forwarder/global_conf.json I applied the steps https://explorer.helium.com/hotspots/112jKgdfZnHKs1wXe1toKw7YyAkYK88DKWvtk6soLZPEyAcvmZWk

[yooitsgreg]

Hi all - Looking for help troubleshooting why my hotspot has hasn't sent a beacon in over 5 days. In addition, I also see in Explorer under All Activity many "Created Challenge" entries but rarely a corresponding "Challenged Beaconer" entry.

Looking for information on how to troubleshoot the lack of Challenges / Sent Beacons so I can test out a new antenna at my location (not worried about no Witness activity atm)

Bluetooth Diagnostics, Hotspotty.net, Heliumstatus.io all show everything is setup correctly/no issues.

Atomic Mocha Cheetah https://explorer.helium.com/hotspots/112eWyTrhGz6qYM7YSRGvmC1aeW7QUdfVeXALDiGMwCX9sn8c1gp/activity

I swapped out the SD card last night and now PoC activity has resumed after almost a week of nothing. FYI bluetooth diagnostics said the old SD card was ok but....

Hope this helps for others having the same issue.

[serbyxp]

not sure this is related. but my miner is "fully synced" but showing syncing, yet the block is latest or even 2-3 blocks ahead of the explorer and about 600 more then hotspotty, I have been getting a relayed on explorer but on hotspotty and the heliumstatusio it shows everything is good except that its still syncing. I am getting this message similar to the above this is the 12/30/21 latest or _GA.

: 2022-01-01 23:15:56.000 7 [error] <0.16668.0>@blockchain_txndialer:dial:{142,21} libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/13JowaFgp9pwoLH8VVygiMBz7X1nJ5rtLA67DAFAjA2DY1o9ZAt", TxnHash: <<156,166,16,248,47,124,47,240,229,6,242,184,228,15,4,117,45,209,155,127,205,122,62,156,96,147,7,204,247,10,103,89>> 2022-01-01 23:15:55.999 7 [error] <0.16667.0>@blockchain_txndialer:dial:{142,21} libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/11RCnXLeZpA74auiN1BCu9iKiMm6ojeXrRQZ4qmK3J9S8S85zbv", TxnHash: <<156,166,16,248,47,124,47,240,229,6,242,184,228,15,4,117,45,209,155,127,205,122,62,156,96,147,7,204,247,10,103,89>> 2022-01-01 23:15:55.996 7 [error] <0.16663.0>@blockchain_txndialer:dial:{142,21} libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/13vV3kxXGepnFjhj4A8LPtpXpiEgtwcQUSwhaXHQrD1PNkpeC9E", TxnHash: <<156,166,16,248,47,124,47,240,229,6,242,184,228,15,4,117,45,209,155,127,205,122,62,156,96,147,7,204,247,10,103,89>> 2022-01-01 23:15:55.989 7 [error] <0.16662.0>@blockchain_txndialer:dial:{142,21} libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/13RowdgDYCQnhJCHWNGZmzvgo52Z5TVYehn7b8mqD3oCdhMwHFb", TxnHash: <<156,166,16,248,47,124,47,240,229,6,242,184,228,15,4,117,45,209,155,127,205,122,62,156,96,147,7,204,247,10,103,89>> 2022-01-01 23:15:55.988 7 [error] <0.16661.0>@blockchain_txndialer:dial:{142,21} libp2p_framed_stream dial failed. Reason: not_found, To: "/p2p/112pauM9BFDkuUPfkW1qptAiDH3JEBFoRwmbucEPXGcemZ2rUR9z", TxnHash: <<156,166,16,248,47,124,47,240,229,6,242,184,228,15,4,117,45,209,155,127,205,122,62,156,96,147,7,204,247,10,103,89>>

• my ulimit nofile = 64000 and on the os its ulimit -Sn is 65535 NAT = none dialable yes, connected yes, and latest block everytime I do a miner info summary... *

* dont know if the helium "gods" did something after I posted this but I just sent a packet out throught the concentrator, still not showing up in the app but it litterally happened 2 mins ago.

[bkauii]

Wobbly sangria Jellyfish is still not sending beacons? What gives guys? Any ideas?

[serbyxp]

My miner finally updated to fully synced status, it’s been challenging and it’s beaconed 2 -3 times but with 0 witnesses and I’m surrounded by hundreds of miners in my zip code alone. First beacon I thought it might be because I had it on my work table , I put it back to its original spot, it was challenged 2 more times and sent beacon, I saw it in the concentrator terminal , I don’t believe it was logging but it mentioned something like … “ beacon tx (33dbi) transmit power invalid tx power using 27dbi tx power “… did that the two times I was watching it. Not sure what the deal is with that… I even burned some dc to re assert the antena value and elevation ( which hasn’t changed ) and I have been using the helium GitHub global conf settings which has the “pa” values all set to 1 and the updated values acc ending from like 4-20 skipping like 18 ( whatever stock is ) it’s 915 sx1302 from seeed studio version WM1302

• also I’m not sure but there is no jitqueue peers in the concentrator terminal but it sends like 3 data grams in push pull ack and gets 100% received … if that means anything sorry for lack of log files, working on that now.
• Hotspot name noisy pine seal , north Miami

[SAVAMforWIN]

hi guys, I recently bought a P100, after having mounted it I found problems with invalid witnes despite everything being declared correctly, I read on github that it was necessary to change 2 "rssi offset" parameters. so I did and actually the invalid witnesses have become valid, however the p100 often crashes and loses sync with the blockchain, a few hours ago I carried out a portforwarding on port 44158 (tcp and udp) but in any case I notice that during the resync process there are things that crash, below are the logs

errorlog.txt minerlog.txt witnesslog.txt

[heliumminermh]

This is PoCv11??

any one can do like this??

Why any one not taking action on these 23 spoofed hotspot

https://explorer.helium.com/accounts/14L2z92r4QAgeYFogRYL3JUfLWmubxTbuhZYcmVvaeQw88cjgEx

[blackoutstudiostv]

uptime since nov 17th , 0 witnesses till today firmware always updated in the last week witness in explorer decreased from 156 to 50 beacons decreased from 56 to 39 nothing changed with the setup it self stock antenna , correct gps location and elevation hight api reported adress didnt change for 6 days now 0 earnings it's a pisces

if you check those p2p addressess, most of them alive and pingable... so it is helium issue??

in the mean time I ve setup a SCPrime Miner -.- minerlogs.txt errorlogs.txt

[bkauii]

Wobbly Sangria Jellyfish says offline even though when I pair with it, it says online and connected. Sparkly Felt Giraffe is offline but when I pair with it, says online and connected Plain Plastic Bull is online but has done nothing for days Creamy Dijon Seahorse is online but has done nothing for days... All my other hotspots are functioning properly. Idk why these arent

[serbyxp]

well I updated the miner to the 1/12/2022.1 and all the errors have gone away. Ive realized more witnesses just in the past 3-4 hours since.. the blockchain _handler error 112 has disappeared. I still am not fully synced even tho the miner is up to the latest block, I'll give it a day or 2 since its fresh update. No beacon yet but to soon to say..And the snapshot saved-snap error went away as well...

1/12/22

• update still failing to dial challenger
• Failed to send witness, max retry

9:15am 1/13/22 ( EST)

• • still not fully synced last update 1 week ago
• • no beacons only challenger and witness

[scottsweb]

Hoping this is the best place to report a potential issue. I used to transmit data packets once or twice per day, after the big outage I have not seen any data packets on my device (so around two months now). All firmwares up to 2022.01.04.0-2 on a EU Nebra miner have been live during that time. I am hoping data is still flowing over the network from time to time, its just not being logged on the explorer or rewarded.

[serbyxp]

Error log update of 1/12/2022.1 on Noisy pine seal block height supersedes the explorer.helium.com api block height, still not fully synced says last update over a week ago. Only receiving challenger rewards and some witness rewards last beacons / full sync was on pre 2022 firmware. Logs below.

• Its almost like everyone I try to dial has a firewall or not accepting inbound messages. I don't have any of those fancy graphs and what not. But if someone that has that framework already built wants to plug those addresses into their database and see if those addys match up with any of the supposed " black list" that might shine some light on the issue..

error_log_1.13.22_fw_1.12.2022.1.txt

[bkauii]

It's been over 30 days since my hotspot has done anything. SPARKLY FELT GIRAFFE has done nothing but is fully synced.... Wobbly Sangria Jellyfish is also not beaconing and rarely witnessing

[serbyxp]

I'm witnessing consistently, not the same as before. But at least that is working. Challenging is working, I was told that beacons are random, but I still yet to beacon on the new firmware. Prior firmware on hotspotty showed my syncing state at -10,000+ but now shows synced, but the helium API does not show an update in over 2 weeks. I looked at the status.helium.com website and it shows everything is good except for the scheduled maintenance, and a couple issues that seemed to have been resolved by the helium team. I don't even want to dig into the logs again, I'm sure I'm going to find myself frustrated by that, and I don't get paid to get frustrated. How exactly is the status.helium.com quantifying that the API is in the "green" if its not taking into account hotspot addresses that haven't been updated in weeks or months?

I'm just proposing some sort of mechanism that goes through all the hotspots, and does some sort of checksum on erroneous factors, IE. hotspots that are online and reachable that are showing lack of updates to the API in weeks or months, in an attempt to update them, and the API itself. I completely understand that the network is moving towards a different system or M.O., in coming updates.

I'm also proposing for the Dial errors, something similar to how the miner software " keep alive" function with the LoRa concentrator works, ACK up ACK dwn. But for the connecting peers, Like a Ping , Pong. when a connection is established to a peer that the handshake is 2 way, to insure an in and out bound connection, that can be maintained through the course or "window" of the witness block receipt. Failure to maintain that connection from the validator, could result in some sort of penalty.

Again this are suggestions and proposals for this issue. Im sure the people getting paid for this have better Ideas, solutions, and actual knowledge of the situation at hand. As it is not public knowledge, I'm not going to rack my brain over what ifs.

Thank you.

• note- When I have been " synced" as per the helium API, Is the only time I have noticed, That I get challenged to beacon. Im not sure if thats part of the algo or mechanism for being " available" to beacon or be challenged.

[serbyxp]

Well I decided to finish a beautiful day with error logs, Frustration settles in... Just a couple hundred failed to dial I missed out on in 3 days I have no Idea whats going on with this my internet hasn't gone down I have the advanced security off the port forwarded, I do get some witnesses registered. I notice some bad TX frequency issues but those are few and far between. I notice a in gen_server:call/2 line 239 Not sure what that is. Erlang version on AMD is totally different then the ARM version on the call lines, if your on arm you talk to arm, if your on AMD you talk to AMD but not cross platform, Im 90% sure of this and I don't know enough about it to fix it, I am giving it a go. so If I run the AMD version which all the validators on the cloud are probably running, I will get all those dials to go through. BUT I lose all the ARM dials, so its a trade off. Which I believe would be better to be dialing AMD since its validators. vs ARM which is just miners, and not all miners.

The only Idea for a quick patch / workaround, I dont think your going to like, and I dont want to get flagged for gaming but Im going to do it, Im going to run 2 miners off one EEC on AMD and on ARM. with 1 packet forwarder. DONT FLAG ME! I am doing this as a TEST to see if IT works or not.

ARM64 latest 1/12/2022 error_logs_ARM64Latest_2022_01_12.txt