Rak - No inbound/outbound/block-height 1, after 12.29.2 update

[smeakles79]

Have reports of several Rak hotspots that, after the 2021.12.29.2, are connecting to wifi/ethernet and obtaining an IP and connecting to the local router, no longer have an inbound/outbound connection. Block height on these Raks are either unknown or at 1.

Some unasserted, but staying synced hotspots are. Narrow Umber Stork Exotic Satin Anteater Alert Lead Tapir Restless Pewter Anteater Hidden Brick Wasp

Asserted Units Muscular Turquoise Donkey

[Ramsin]

Add Jumpy Berry Sheep to the list... Had it fully synced a few days ago, unplugged it... Turned it on a few days later to fully sync it before taking it to the location it was going to be installed at and I get no in/out, 0% synced. Running 2021.12.29.2

(I have a fully working hotspot at the same location so internet is good)

[smeakles79]

Powerful Crepe Cat is another members hotspot. Same issue. 2021.12.29.2 firmware.

[MrBlue995]

Gorgeous Banana Shrimp

[Swoesh33]

I am the owner of Powerful Crepe Cat, I received the hotspot on 31dec and set it up (without setting location) and it started syncing without a problem the 1st I unplugged it and drove to my location and put the location in the app. from that point it has not done anything .... even back at home where it worked before. now I have no inbound/outbound so it's just doing nothing. Hope this will get fixed so I can drive again for 3 hours but now it will work.

[hcl-83]

I have 3rak the same issues: Quaint Champagne Bee Plain Sable Cod Clumsy Rusty Monkey

[Peanudt]

Same issue: Cool Taffy Alpaca

[pajche]

Tart Tiger Mammoth, worked fine for more then month. Now since last update no activity.

[m1xmstr]

Can someone who is having this issue try changing their IP address and see if their hotspot starts working.

If you have a separate router and modem: First check what your IP is by Googling "What's my IP"

-Power off both router and modem. -Power on router -Change mac address (usually under clone mac just change last octet) and save/apply -Power on Modem -Verify IP is different ~ Google "What's my IP" -Restart RAK Hotspot

See if this resolves your issue.

[smeakles79]

I’ve got a Static IP from my ISP, so I can’t make a change. Powerful Crepe Cat is at a business, so can’t make a change there.

[hcl-83]

I try on fiber static ip and gsm lte connection no change no connection to p2p network

[stuck2007]

Hi m1xmstr, I just tried changing the public IP address on my miner but it made no difference. I can provide packet traces from the router if needed...

[Swoesh33]

i plugged it in at 2 different places both don't work anymore, where the first one did work

[m1xmstr]

Hi m1xmstr, I just tried changing the public IP address on my miner but it made no difference. I can provide packet traces from the router if needed...

Okay thank you for the update. Someone mentioned this on a discord channel and thought it was worth a try. I'll keep digging and post if I find anything.

[gdanskgrom]

Hi m1xmstr, I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

[m1xmstr]

Hi m1xmstr, I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

Okay thank you for the update. It was worth a try.

[kibria88]

Attractive Daisy Shetland Inbound/Outbound Ok Block height on rak either 0% or at 1 Firmware Update: 2021.12.29.2

[Swoesh33]

Hi m1xmstr, I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

mine is on height 0 and when I "send diagnostic" I see this

Connected to Blockchain: Dialable: Height: Unknown NAT Type: (most are just empty)

[Ramsin]

It's 100% related to the firmware, luckily I had made backups of my SD Cards before even powering up the hotspots. When I restored my problematic one (Jumpy Berry Sheep), it had In/Outbound, and was at 84.x% synced. It updated the firmware automatically and went back to being problematic. (Original FW that came with mine was 2021.08.something.something)

[m1xmstr]

This feels a lot like the issue that occurred here back on early 2021. There wasn't a clear fix other than OTA

https://engineering.helium.com/2021/03/17/rak-firmware-issue-2.html

[pajche]

Something is wrong with newest firmware I think. But my other RAK works fine. I just came back from location. Diagnostics are okay. Tho before I changed my IP only Outbound was OK, after changing the IP both Inbound/Outbound are OK. Blockchain is 100%, and with block number it was synced. Disk is OK And last challenge was 6 days ago, and that's when it stopped doing any activity.

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

[Ramsin]

But my other RAK works fine.

Are you sure? My Goldspot drastically dropped in earnings once this firmware kicked in, but at least it’s working….my other GS is totally dead in the water…

[beervoley]

After the firmware upgrade my RAK MINTD stopped working too. Inbound/Outbound are shown as working but it's offline and sync is stuck at 99%~

[serbyxp]

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time.

I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

[pajche]

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

I can confirm now, in my case changing IP fixed Inbound Status and its now showing activity, started working again. At least it works for now.

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time.

I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

Also this is a problem as well, I noticed very low activity on some other miners where I have log access and I could see a lot of failed to send witness errors, couldn't dial challenger, after I checked most of my peers were RAK devices and most of them offline, after rebooting hotspot and refreshing peer list, activity was much better and less errors.

[MrBlue995]

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

I can confirm now, in my case changing IP fixed Inbound Status and its now showing activity, started working again. At least it works for now.

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time. I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

Also this is a problem as well, I noticed very low activity on some other miners where I have log access and I could see a lot of failed to send witness errors, couldn't dial challenger, after I checked most of my peers were RAK devices and most of them offline, after rebooting hotspot and refreshing peer list, activity was much better and less errors.

how did you exactly changed your ip? did you just connect the miner to another network? could it work?

[pajche]

how did you exactly changed your ip? did you just connect the miner to another network? could it work?

I rebooted my internet modem, at that location I don't use static IP, so that was a solution. But yes, connecting to another network could also work. Or probably trying to connect it over VPN.

If you can try and check diagnostics and share results with us it would be great to know. I see for some people it doesn't work. But in my case it worked.

[Peanudt]

Initially trying to setup a unit without location asserted. After assertion latest firmware did not work. Flashed an older firmware version and then the hotspot was operating as normal. All connections green and block height increasing. During the night, the Hotspot received over the air update to latest firmware and peer2peer connections and block height are grey again. So for my situation it all points towards an issue with my hotspot + latest fw version:

• issue not related to location assertion
• sd card is not corrupted
• with old fw connections are good (using dmz, because other unit in network is using port forwarding). One static IP from ISP.
• Similar behavior wired and wifi. Hotspot gets IP and diagnostics accessible via BT.

[jonasjv]

Rapid Vanilla Kookaburra Same problem here Inbound Outbound grey - O% blockchain sync

In my case i tried to sync before assert location

[gdanskgrom]

Initially trying to setup a unit without location asserted. After assertion latest firmware did not work. Flashed an older firmware version and then the hotspot was operating as normal. All connections green and block height increasing. During the night, the Hotspot received over the air update to latest firmware and peer2peer connections and block height are grey again. So for my situation it all points towards an issue with my hotspot + latest fw version:

• issue not related to location assertion
• sd card is not corrupted
• with old fw connections are good (using dmz, because other unit in network is using port forwarding). One static IP from ISP.
• Similar behavior wired and wifi. Hotspot gets IP and diagnostics accessible via BT.

Exactly the same situation for 2 of my units as well. Flash back to 12.14.1 and everything is operational until the OTA is consumed.

[jonasjv]

Do you know where i can get fw 12.14.1?

[serbyxp]

well does that work?! rolling back...? Im going to test it now. The issue is we dont have controllo f our hardware. only the manufacturers have control of our hardware. Which isnt de centralized what so ever. You cant just plug in a cable and get things working how you want. Its like Xfinity or any of those types of companies that control your hardware remotely. I had an issue with my hardware that's the only reason I had to get into making my own firmware. But then you lose all support from your manufacturer. see Xfinity ATT all them monopolized companies, you rent there modems etc... but we own this hardware, And you cant even log into it to change anything. I don't mean to sound against it... but the manufacturers that make all this hardware are Chinese ... so ... if you know how China works then you know how this works.. Also 5G dont expect that anytime soon. The FAA alreaady is pushing back ATT from setting up 5G networks in big cities with the Airports pushing back on them here in the US. so dont expect that freedomfi modem anytime soon

[smeakles79]

Do you know where i can get fw 12.14.1?

https://docs.helium.com/troubleshooting/replace-sd-card/

[serbyxp]

Do you know where i can get fw 12.14.1?

https://docs.helium.com/troubleshooting/replace-sd-card/

I got a 5 TB hard drive connected to mine... so... not an SD card issue

[serbyxp]

Also. The new PoC and the new HIP ... means we dont get rewarded for mining anymore... so we are being forced to continue mining and following the blockchain, for no reason. They should have rolled out gateway-rs at the same time they took away the rewards for mining. The manufacturers are still selling out of 500$ machines that are hoard bought by people who sell them for 1000$ + when you dont need that level of hardware. It was a bad momemnt for them to roll out that HIP, without the supporting software.

[smeakles79]

What are you going off on? You are 100% incorrect that “we don‘ t get rewarded for mining anymore” please stop the FUD. Unless you have something to help, stop the FUD and let the Devs pick up the issue vs filling this specific item with garbage.

[serbyxp]

we dont only validators do. what do we get? .003 every 6 hours for crerating challenges? im sorry if I sound like im going off, im just frustrated at this.

[smeakles79]

Validators are rewarded for being in Consensus group. That’s been off the miners since July. Validators do not do proof of coverage, hotspots do. Leave this post to the specific Rak issue. If you want to argue on the above, go to discord.

[Ramsin]

We get that, but how does it contribute to this specific issue? You're just flooding this with nonsense at this point, and this issue will be ignored...as it apparently already is.

[serbyxp]

Your absolutely right. let me not mess your attempt at resolving this, I will not speak up to try and motivate any sort of call to action, to fix the current bugs and issues. IF anyone finds a fix, please do @ me to it.

[smeakles79]

From Raks Discord Update on firmware 2021.12.29.2 issues and SD image availability

While firmware 2021.12.29.2 brought huge stability improvements, bringing more than 8000 hotspots back to life, we've been monitoring a small cluster of hotspots which stop showing inbound and outbound connections when updating to this release (but appeared fine in the 2021.12.24). Right now we have seen about 50 reports of these and it seems to affect particularly freshly flashed or brand new hotspots, especially those without location assert.

These findings have been communicated both by the community and ourselves to Helium but we're still waiting on a solution from them. We believe this should be coming soon via another update.

Until the core issue is identified and confirmed by Helium we are also not offering our SD card image to reflash miners as we're not sure what the problem is yet . We strongly recommend not re-flashing hotspot microSD cards right now until there is an all clear.

[Tylerlhess]

I have this issuse on my Bobcat 300. It synced at my home (without an asserted location) fine but when I moved it to its location it stopped working. I thought it might have been the business internet and moved it to another location. No dice.

Cheery Blush Zebra no inbound outbound connections Stopped working when I asserted it while on the 2021.12.29.2 firmware.

[serbyxp]

From Raks Discord Update on firmware 2021.12.29.2 issues and SD image availability

While firmware 2021.12.29.2 brought huge stability improvements, bringing more than 8000 hotspots back to life, we've been monitoring a small cluster of hotspots which stop showing inbound and outbound connections when updating to this release (but appeared fine in the 2021.12.24). Right now we have seen about 50 reports of these and it seems to affect particularly freshly flashed or brand new hotspots, especially those without location assert.

These findings have been communicated both by the community and ourselves to Helium but we're still waiting on a solution from them. We believe this should be coming soon via another update.

Until the core issue is identified and confirmed by Helium we are also not offering our SD card image to reflash miners as we're not sure what the problem is yet . We strongly recommend not re-flashing hotspot microSD cards right now until there is an all clear.

You need to find someone that is fluent in Erlang code. The libp2p is throwing errors that have been posted in another thread I will post them here maybe it will shine some light on the issue. (https://github.com/helium/erlang-libp2p/issues/417#issuecomment-1003880885)

[kovicak]

my MNTD has had 0 earnings since Dec 17. I've reset, many times and says 'syncing'. It's been syncing for the past 6 days. Please add:

Sticky Cerulean Nightingale

[herculanocosta]

Firmware 2021.12.29.2 update The Helium core team has identified the issue with firmware 2021.12.29.2 where the miners may show issues connecting to the blockchain and showing no inbound or outbound.

This issue only affects some miners that have had a location asserted after the 2021.12.29.0 update, Hotspots asserted before the 2021.12.29.0 will not experience this issue (but may show similar symptoms due to other problems like network connection)

Helium is fast tracking version 2022.01.04.0 to deal with this issue.

Follow Rakstars discord for more updates. Thanks for your patience.

[Tylerlhess]

the new firmware image fixed my hotspot yesterday

[herculanocosta]

This issue has been cleared. This thread may be closed

[kovicak]

Could you point me to the instructions for updating the firmware?

On Thu, Jan 6, 2022 at 11:15 AM herculanocosta @.***> wrote:

This issue has been cleared. This thread may be closed

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1325#issuecomment-1006716268, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXFAQGAEJE4PZ2KEHDUGO4TUUW53DANCNFSM5LD4MWPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

-- Thanks, Tony Kovicak

[smeakles79]

Could you point me to the instructions for updating the firmware?

-- Thanks, Tony Kovicak

If it’s a Rak, use this https://support.getmntd.com/hc/en-us/articles/4418137820951-I-need-to-replace-the-microSD-card-on-my-hotspot

[velasquez87]

I have same issue with my bobcat300

Please add Basic onyx snake

[redleg105]

Having the same problem. Special Strawberry Platypus