search

helium / erlang-libp2p

An Erlang implementation of libp2p swarms
https://helium.github.io/erlang-libp2p
Apache License 2.0
120 stars 33 forks source link

IP change not detected #373

Open madninja opened 3 years ago

madninja commented 3 years ago

Many EU ISPs change consumer IP addresses every 24-48 hours.

There have been a number of reports where the IP address changes, and while the API has a recent peerbook record the IP address was not updated to reflect the change either never, or it took a very long time which put it past the time for the next IP update.

Restarting the hotspot seems to fix it every time this was observed.

The theory is that IP change detection is failing, not being done often enough or not being done at all?

maco2035 commented 3 years ago

This occurs in the helium OG, and rak's. This happens on occasion. With about 50+ miners this has only occurred a few times with me. I have use a wifi outlet that I can remotely turn off and on. It is a minor inconvenience for me, but for others this might be worse as the hotspots aren't able to be adjusted properly.

leyuansugar commented 3 years ago

This problem also happens with bobcat.

wghewitt commented 3 years ago

NO BEACONS

I've recently experienced the exact problem with 2 of my hosts and miners where they did not send any beacons for over 9 days which was concerning, when we rebooted their routers beacons started to happen again in the next few hours.

This problem has now become more of a recent report of events by several people in the South African Helium Community, as our ISP also changes IPs every 24-48 hours.

And there are now several hotspots where you would see they have not sent beacons at all in the last week or more and they are not even aware of this problem or how to resolve it.

This is becoming a problem now for us as we are starting to witness less as beacons are not being sent due to this problem.

And we have seen this with Bobcat and Sensecap miners.

Hope this will get as much attention to be resolved as soon as possible, please.

Vagabond commented 3 years ago

Can anyone affected help me troubleshoot this? I need access to an affected hotspot, or at least the logs from it.

chillbreak commented 3 years ago

what exactly is needed, i can provide it because i am affected by this problem.

abhay commented 3 years ago

@chillbreak a hotspot name would be helpful. It’s easiest for the core team if it’s an OG or Rak

chillbreak commented 3 years ago

Prehistoric pearl dinosaur

radicale commented 3 years ago

I've seen the issue and reproduced it with the following test: 1) Hotspot behind CG NAT (shared IP) and thus relayed 2) Route hotspot through a VPN to a public IP 3) Forward port 44158 4) Peerbook "listen ip" is updated from the relay address to the CG NAT shared IP, and not the VPN public IP that it's actually using. It is as if the hotspot caches the IP it initially discovered and never bothers to check it again.

Vagabond commented 3 years ago

I know what the problem is, I just don't know why it happens. Prehistoric pearl dinosaur is not a hotspot I have support access too, unfortunately.

westerwalder commented 3 years ago

A simple reboot of the miner/hotspots resolves the issue, as during reboot it checks for an public ip change. This has to be implemented periodically, as it is nothing high sophisticated... But this issue is quite critical, as in some cases this would result in an unreachable miner for several days/months.

radicale commented 3 years ago

I know what the problem is, I just don't know why it happens. Prehistoric pearl dinosaur is not a hotspot I have support access too, unfortunately.

Perhaps try to reproduce it by changing the outward facing IP of a hotspot or test setup? Are you having trouble reproducing the issue?

shawaj commented 3 years ago

@Vagabond if you DM me and we have a willing volunteer from https://github.com/NebraLtd/helium-miner-software/issues/108 I can give you logs / info about the HS. I know @dakky221 has this issue

dakky221 commented 3 years ago

Yeah, I'm having the same issue, and I "fixed" it by plugging the miner into a smart-plug and making it reboot the miner on IP change, but that will only decrease the lifespan of the hotspot and must be other way around, like @westerwalder said - add periodically check for public ip, for example every 5 minutes, it would fix the issue entirely.

dakky221 commented 3 years ago

btw @shawaj unfortunatley my Nebra is now behind CGNAT on a semi-static IP so my logs won't really help. I can share the Sensecap's logs though if there is a way to get them.

ghost commented 3 years ago

Yep, plugged my hs on a timer as well to reboot every 24h. Currently takes 2-3 hrs to reflect updated IP, but lag seems to increase (week before: ~1hr) due to network growth(?).

westerwalder commented 3 years ago

By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..

ghost commented 3 years ago

I have cloned SD-Cards as backup should that happen. But I'm about to switch to tunneling a static IP instead.

dakky221 commented 3 years ago

By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..

Well, hopefully the libp2p gets updated to periodically check for external IP before the sd-card goes kaboom ;)

Vagabond commented 3 years ago

I should have a fix for this today or tomorrow that we can test.

wghewitt commented 3 years ago

I should have a fix for this today or tomorrow that we can test.

That is fantastic, will this fix be rolled out in an OTA on all miners or how will the update be released?

dakky221 commented 3 years ago

I should have a fix for this today or tomorrow that we can test.

I can renew the IP as many times as you wish on my VDSL2 connection, but the hotspot is Sensecap, so if they can deploy the updated firmware we can test.

ci-work commented 3 years ago

possible related: https://github.com/helium/erlang-libp2p/issues/376

Vagabond commented 3 years ago

Has this improved any now both PRs are merged and deployed?

dakky221 commented 3 years ago

There is no way to be sure if the fix worked or it didn't. Listen addr doesn't change on external IP change, but I guess that's normal because it gets it from the API and API is too slow to be of any use. I also guess manufacturers should add to the dashboard some kind of a log if the hotspots sent the new IP to the peers to be sure...

leyuansugar commented 3 years ago

My isp updated the ip address about 10 hours ago, but looking at the address on the api it still shows the previous one. but when I just checked inside peerbook it shows the correct ip address, of course I had rebooted the hotspot before that so not sure if it was the reboot that made the peerbook change. but as of the time of the comment the api still shows the wrong address.

Vagabond commented 3 years ago

Rebooting the hotspot will find the new address, yes. The API may not have a perfectly up to date view of the system so it may take a while to see the IP change (if the hotspot notices it).

dakky221 commented 3 years ago

OK, just to add: I didn't reboot the hotspot since the IP change. 10 hours already passed and it's not updated on peerbook or API. Listen addr. is still the yesterday's IP. Will reboot hotspot now, I guess 10 hours is enough of waiting to see that fix isn't working.

radicale commented 3 years ago

It pains me to say that my Sensecap on firmware 2021.08.24.4 still shows the same behaviour and has not updated its IP address in several days (API and peerbook). Consequently the last beacon it performed was 5 days ago (compounded by the block halt issue).

Vagabond commented 3 years ago

Ok, I guess I can't fix it blind. I need details on these affected hotspots (manufacturer and hotspot name) so I can try to get some logs from them. You can privately provide me details via email to andrew@helium.com

westerwalder commented 3 years ago

All of my sensecaps (all on newest fw) are still affected. Only a reboot will result in an updated listen-ip information due to a changed ISP public ip change.

dakky221 commented 3 years ago

Same here, also Sensecap M1. We don't have any accessible logs via dashboard @Vagabond .

madninja commented 3 years ago

Same here, also Sensecap M1. We don't have any accessible logs via dashboard @Vagabond .

As he said, please provide hotspot mfr and address/name details via email so he can contact the maker to get the logs

dakky221 commented 3 years ago

Ahh. That. Sent email.

ghost commented 3 years ago

Ok, I guess I can't fix it blind. I need details on these affected hotspots (manufacturer and hotspot name) so I can try to get some logs from them. You can privately provide me details via email to andrew@helium.com

Emailed you the details of my RAKv1 and suspending the 24h reboot cycle. Hope you'll find something useful to find a fix.

dakky221 commented 3 years ago

So just to confirm, there might be some misunderstanding here.

The hotspots sees the IP change - it seen it even before the fix - but it doesn't pushes it to the peers. This might not be related to libp2p at all, I don't know. I only know it didn't push the new IP for 10 hours (other miners don't see the change in their peerbook) but after reboot of the hotspot, the IP is propagated through the network in an hour or so.

So it's not the change of IP here what is the problem - it is publishing it to the other peers!

ghost commented 3 years ago

The logs should tell the story - at least from the hs POV - and hopefully point the way to a fix.

In June/July a new IP was reflected 16-28hrs - a hs reboot seemingly having no/negligible effect. Begin August a hs reboot seemed to have a meaningful effect again - I assumed due to the (libp2p) updates deployed then.

So, could be a 2-part issue: Hs not declaring re-issued IP + p2p propagation - with the latter seemingly somewhat resolved.

Currently, with my hs reboot new IP is reflected within 1-4h. Without reboot new IP was not reflected for 52hrs before I rebooted.

leyuansugar commented 3 years ago

My hotspot lasted up to 6 days with the old ip without a reboot

Without reboot new IP was not reflected for 52hrs before I rebooted.

shawaj commented 3 years ago

Same here, also Sensecap M1. We don't have any accessible logs via dashboard @Vagabond .

@dakky221 we can access logs. can you forward me the email you sent to @Vagabond and then I can pull the logs for a hotspot that has this issue.

(on your Nebra one having the issue, not the sensecap of course 👍 )

dakky221 commented 3 years ago

@shawaj as I said (or didn't but meant) I moved Nebra to other location which isn't affected by the problem (it has static external IP).

Only Sensecap is currently affected and can't move it from dynamic public IP.

haze94 commented 3 years ago

Hi guys I'm new hear I've been browsing everywhere to solve this issue I been having for over a week since I bought my miner my listen up isn't matching my public ip my I have a sensecap with name : Acrobatic Brown Okapi If someone can please help I would really appreciate it thank in advance.

Vagabond commented 3 years ago

The patch will be in the next release after it's reviewed/merged

haze94 commented 3 years ago

The patch as in an update to fix the ip4 listen ip to update to public ip and also any idea when would that release be thank you in advance.

Sent from my iPhone

On Aug 30, 2021, at 6:08 PM, Andrew Thompson @.***> wrote:



The patch will be in the next release after it's reviewed/merged

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/helium/erlang-libp2p/issues/373#issuecomment-908736808, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVNF4RM5OAPZVWHT7HOGQ7LT7P6PDANCNFSM5CIMUQUQ.

haze94 commented 3 years ago

The patch will be in the next release after it's reviewed/merged

The patch as in an update to fix the ip4 listen ip to update to public ip and also any idea when would that release be thank you in advance.

Vagabond commented 3 years ago

I'm hopeful we can get it out this week.

Vagabond commented 3 years ago

We should have it in a release this morning. Other manufacturers will pick it up after we tag a GA.

Everyone can stop emailing me.

Vagabond commented 3 years ago

Ok, it's GA'd. Please wait a few days to see if the fix has helped before reporting issues again.

ghost commented 3 years ago

Ok, it's GA'd. Please wait a few days to see if the fix has helped before reporting issues again.

Fix seems to be working. After 3 days and 3 IP changes (every 24hrs), re-issued IP was reflected within 1-2 hrs in peerbook. As long as p2p propagation time doesn't return to June/July levels, we should be good.

Thanks for the fix! A major headache (for EU residents) removed. Greatly appreciated!

leyuansugar commented 3 years ago

The issue seems to have been resolved, my isp has changed its ip address twice in the last few days and I haven't manually rebooted the bobcat hotspot in that time, but I can see the correct ip displayed on the api and peerbook. Thanks for the fix!

Chris-S44 commented 3 years ago

I still seem to have an issue with the listen ip. It's been 8hrs since my public ip has changed and api has not updated yet. Only peerbook has updated.

westerwalder commented 3 years ago

Yes, seems to be broken again. Experiencing the same and even a hotspot reboot didn't help yet. Before the 14-09 FW update, it worked quite well... SenseCAP with newest 16-09-01 FW.