Open fryorcraken opened 1 year ago
fryorcraken
commented
1 year ago Note that this happened because my laptop (running js-waku in the browser) went to sleep. However, I was not able to reproduce the issue when I disconnect the browser from the internet (libp2p does not seem to notice the disconnection).
Removing the critical label as it seems to be an edge case.
alrevuelta
commented
1 year ago Thanks for reporting!
I am still trying to pin down the exact error but would be keen to know if nwaku logs (from the fleet) can be extracted using the local timestamps.
With this kibana link&_a=(columns:!(message),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:ffcc22b0-0116-11ed-9719-cdd3b483481c,key:fleet,negate:!f,params:!(wakuv2.prod),type:phrases),query:(bool:(minimum_should_match:1,should:!((match_phrase:(fleet:wakuv2.prod)))))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:ffcc22b0-0116-11ed-9719-cdd3b483481c,key:logsource,negate:!f,params:(query:node-01.ac-cn-hongkong-c.wakuv2.prod),type:phrase),query:(match_phrase:(logsource:node-01.ac-cn-hongkong-c.wakuv2.prod)))),index:ffcc22b0-0116-11ed-9719-cdd3b483481c,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))) you should be able get the logs from node-01.ac-cn-hongkong-c.wakuv2.prod. Let me know if the link doesn't work.
Also keen to know what information I need to provide to help extraction of said logs.
Perhaps your peerId can help, since logs are quite verbose. With that it should be easier to search. We can also try to search based on timestamp, I can see it says 09:53:58.126. Is this UTC or your local time?
alrevuelta
commented
1 year ago Summarizing here Tanguy comments:
Looking for "Too many connections for peer" in the logs helps to know when this is happening on nwaku side.
Its expected, since the maximum number of connections per peer is 1 maxConnectionsPerPeer = 1. So when a peer that goes offline (without properly disconnecting) tries to reconnect again, the old connection is still tracked, so the new one is rejected. It can take up to 10 minutes for this connection to be cleared. Here's an example conns=2.
DBG 2023-01-12 09:22:51.231+00:00 Too many connections for peer topics="libp2p connmanager" tid=1 file=connmanager.nim:393 conn=16U*dz5wdf:63bfd16b81b6ce33c2b2d928 conns=2Note sure how easy would be to reproduce this scenario, but lets say that I'm connected to like 50 nodes and suddenly my internet is gone. Since no disconnection was made, that connection is still being kept as my connection by all 50 peers. So if I go online later in 5 minutes, I wont be able to connect to any of these peers?
Could be a problem in a network without many peers (total_peers_in_network < max_peer_connections), so it can happen that the node can't connect to any peer during a max of 10 minutes.
fryorcraken
commented
1 year ago Perhaps your peerId can help, since logs are quite verbose. With that it should be easier to search. We can also try to search based on timestamp, I can see it says 09:53:58.126. Is this UTC or your local time?
Noted. Will provide peerId and clarify time (local) next time.
I'm connected to like 50 nodes and suddenly my internet is gone. Since no disconnection was made, that connection is still being kept as my connection by all 50 peers. So if I go online later in 5 minutes, I wont be able to connect to any of these peers? Could be a problem in a network without many peers (total_peers_in_network < max_peer_connections), so it can happen that the node can't connect to any peer during a max of 10 minutes.
I would not expect a js-waku peer to connection to 50 peers but yes, I agree with you.
@danisharora099 is starting to look into peer management on js-waku side. We will have to investigate browser/libp2p behaviour around disconnection/signal loss.
On nwaku side, does it make sense to timeout a connection after 10min? Also, what would be the impact if the timeout is reduced? Finally, how does the timeout works? ie, if data is sent over the connection, does it delays the timeout closure? However clever is the connection management on this aspect?
danisharora099
commented
1 year ago Update: I can confirm that this does not happen by manually going offline but there's a weird inconsistent bug that is seen: the browser that goes offline can send messages after reconnection but can't send messages or the receiving is random.
Steps I followed:
It was now seen that on spamming messages from w2, w1 seemed to receive one message of 5-6 messages sent while w2 seems to receive almost all messages sent.
I will post a consistently reproducible set of steps once I find it
Update: sending and receiving messages on eth-pm is highly inconsistent for me. Can't seem to put down a trend, but messages are randomly lost.
vpavlin
commented
1 year ago @danisharora099 are you still looking into this?
danisharora099
commented
3 months ago @danisharora099 are you still looking into this?
(thanks @Ivansete-status for the ping)
not particularly this, but we do have some share of issues that track similar work: https://github.com/waku-org/pm/issues/143 (check liveness of a node)
also note that we have done some work on js-waku side (like keep-alive management, filter pings, etc) that also go intro improving reliability of this.
as for the current status of the original issue and the above repro, @adklempner did some investigations around liveness recently that might be more relevant here now
Problem
js-waku (js-libp2p) automatically reconnects to remote nodes after disconnection/network access loss.
I can see the re-connection happening but it fails during the identify protocol.
I am still trying to pin down the exact error but would be keen to know if nwaku logs (from the fleet) can be extracted using the local timestamps.
Also keen to know what information I need to provide to help extraction of said logs.
Impact
I am adding
criticallabel because disconnection has been the main issue platforms have complained about js-waku. Now that it makes to focus on it (refactor of js-waku), I would like this to be resolved.To reproduce
If you go to https://examples.waku.org/eth-pm/ stop your internet access (wifi off) and on. You should be able to see that the Waku node tries to reconnect and successful does a libp2p noise handshake but fail during the identify protocol.
Expected behavior
Reconnection is successful
Screenshots/logs
Here are some logs, feel free to let me know what kind of information would be needed to find the nwaku fleet node logs:
nwaku version/commit hash
Whatever is deployed on
/dns4/node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net/tcp/8000/wss/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrDat the time of the issue creation.