search

waku-org / js-waku

JavaScript implementation of Waku v2
https://js.waku.org
Apache License 2.0
168 stars 42 forks source link

peer-exchange browser feedback #1454

Closed danisharora099 closed 1 year ago

danisharora099 commented 1 year ago

This PR is meant to track the feedback and dial errors encountered via peer exchange on browser using the web-chat example, and is part of the milestone https://github.com/waku-org/js-waku/issues/1429 and https://github.com/waku-org/js-waku/issues/1326

Related:

danisharora099 commented 1 year ago

Connections:

Screenshot 2023-08-03 at 5 43 12 PM

Types of errors encountered:

SionoiS commented 1 year ago

After some time I got.

Peers Discovered: 126 Bootstrap: 2 Peer Exchange: 124 Peers Connected: 18 Bootstrap: 2 Peer Exchange: 16

and only WSS errors.

chaitanyaprem commented 1 year ago

Left for a long time and noticed below.

Screenshot 2023-08-03 at 9 40 59 PM

Only logs were errors like below

The connection to wss://node-02.gc-us-central1-a.status.prod.statusim.net/p2p/16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg was interrupted while the page was loading.

I think there should be a limit set to have active peer connections, as per this looks like there is no limit and increases over time. Otherwise app will eat lot of resources and doesn't help having too many peers as well.

danisharora099 commented 1 year ago

I think there should be a limit set to have active peer connections, as per this looks like there is no limit and increases over time. Otherwise app will eat lot of resources and doesn't help having too many peers as well.

Correct. This is the idea. The idea right now is to just to gauge the % success in the number of connections we can make. Eventually, once we go prod with peer-exchange, we will limit total # of connects on js-waku :)

Thanks for the feedback!

fryorcraken commented 1 year ago

@mfw78 what's the setup on dappnode? I see some ws error, can we ensure that nwaku is configured using wss by default?

fryorcraken commented 1 year ago

I am getting

 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e.
 Firefox can’t establish a connection to the server at wss://node-01.gc-us-central1-a.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA.

Despite node connections being quite low: https://grafana.infra.status.im/d/qrp_ZCTGz/nim-waku-v2?orgId=1&refresh=30s&var-host=node-.%2A&var-fleet=wakuv2.test&var-fleet=wakuv2.prod&var-dc=All&from=now-2d&to=now&viewPanel=2&editPanel=2

And again (my peer id: 12D3KooWQbgESPNa8SnDofj5PxiCha1wUVYi8tchfCBSeQ1Q7PKq)

15:09:39.250 waku:waku Waku node created +0ms 12D3KooWQbgESPNa8SnDofj5PxiCha1wUVYi8tchfCBSeQ1Q7PKq relay: false, store: true, light push: true, filter: true common.js:113:9
15:09:39.252 waku:connection-manager Connection Manager is now running +0ms common.js:113:9
15:09:39.325 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 1 +73ms common.js:113:9
15:09:39.326 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 1 +1ms common.js:113:9
15:09:59.311 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e. client.ts:18:17
15:09:59.317 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - undefined +20s common.js:113:9
15:09:59.320 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 2 +5ms common.js:113:9
15:09:59.447 Firefox can’t establish a connection to the server at wss://node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD. client.ts:18:17
15:09:59.449 waku:connection-manager Error dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD - undefined +129ms common.js:113:9
15:09:59.450 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 2 +0ms common.js:113:9
15:10:19.577 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e. client.ts:18:17
15:10:19.579 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - undefined +20s common.js:113:9
15:10:19.579 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 3 +0ms common.js:113:9
15:10:19.675 Firefox can’t establish a connection to the server at wss://node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD. client.ts:18:17
15:10:19.677 waku:connection-manager Error dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD - undefined +97ms common.js:113:9
15:10:19.677 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 3 +1ms
fryorcraken commented 1 year ago

Same in Brave:

main.10c2788a.js:1 waku:peer-discovery-dns Use following EIP-1459 ENR Tree URLs:  +0ms ['enrtree://AOGECG2SPND25EEFMAJ5WF3KSGJNSGV356DSTL2YVLLZWIV6SAYBM@prod.waku.nodes.status.im']
15:12:55.436 main.10c2788a.js:1 waku:peer-discovery-dns Starting peer discovery via dns +84ms
15:12:55.440 main.10c2788a.js:1 waku:peer-exchange-discovery Starting peer exchange node discovery, discovering peers +0ms
15:12:55.961 main.10c2788a.js:1 waku:discovery:fetch_nodes got new peer candidate from DNS address=40ac1a3894f347e04f7fa105c6e2b1a31af5cf68cdd10df9bb6e5e3e576b855a@188.166.135.145 +0ms
15:12:56.122 main.10c2788a.js:1 waku:discovery:fetch_nodes got new peer candidate from DNS address=5ca27c4d8f796b85475e03f326fd045241668f4f25bc3de0a9bc8a1ae29d7894@8.210.222.231 +161ms
15:12:56.133 main.10c2788a.js:1 waku:waku Waku node created +0ms 12D3KooWRSPam6T7v1WV7pw4BTdzbfN9mcxLekZspnHp4XZeFrnS relay: false, store: true, light push: true, filter: true
15:12:56.134 main.10c2788a.js:1 waku:connection-manager Connection Manager is now running +0ms
15:12:56.212 main.10c2788a.js:1 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 1 +78ms
15:12:56.214 main.10c2788a.js:1 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 1 +1ms
15:13:26.156 main.10c2788a.js:1 WebSocket connection to 'wss://node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD' failed: WebSocket is closed before the connection is established.
(anonymous) @ main.10c2788a.js:1
close @ main.10c2788a.js:1
a @ main.10c2788a.js:1
r @ main.10c2788a.js:1
r @ main.10c2788a.js:1
15:13:26.160 main.10c2788a.js:1 waku:connection-manager Error dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD - The operation was aborted +30s
15:13:26.161 main.10c2788a.js:1 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 2 +1ms
15:13:26.171 main.10c2788a.js:1 WebSocket connection to 'wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e' failed: WebSocket is closed before the connection is established.
(anonymous) @ main.10c2788a.js:1
close @ main.10c2788a.js:1
a @ main.10c2788a.js:1
r @ main.10c2788a.js:1
r @ main.10c2788a.js:1
15:13:26.172 main.10c2788a.js:1 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - The operation was aborted +11ms
15:13:26.172 main.10c2788a.js:1 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 2 +0ms
danisharora099 commented 1 year ago

I am getting

 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e.
 Firefox can’t establish a connection to the server at wss://node-01.gc-us-central1-a.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA.

Despite node connections being quite low: https://grafana.infra.status.im/d/qrp_ZCTGz/nim-waku-v2?orgId=1&refresh=30s&var-host=node-.%2A&var-fleet=wakuv2.test&var-fleet=wakuv2.prod&var-dc=All&from=now-2d&to=now&viewPanel=2&editPanel=2

And again (my peer id: 12D3KooWQbgESPNa8SnDofj5PxiCha1wUVYi8tchfCBSeQ1Q7PKq)

15:09:39.250 waku:waku Waku node created +0ms 12D3KooWQbgESPNa8SnDofj5PxiCha1wUVYi8tchfCBSeQ1Q7PKq relay: false, store: true, light push: true, filter: true common.js:113:9
15:09:39.252 waku:connection-manager Connection Manager is now running +0ms common.js:113:9
15:09:39.325 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 1 +73ms common.js:113:9
15:09:39.326 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 1 +1ms common.js:113:9
15:09:59.311 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e. client.ts:18:17
15:09:59.317 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - undefined +20s common.js:113:9
15:09:59.320 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 2 +5ms common.js:113:9
15:09:59.447 Firefox can’t establish a connection to the server at wss://node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD. client.ts:18:17
15:09:59.449 waku:connection-manager Error dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD - undefined +129ms common.js:113:9
15:09:59.450 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 2 +0ms common.js:113:9
15:10:19.577 Firefox can’t establish a connection to the server at wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e. client.ts:18:17
15:10:19.579 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - undefined +20s common.js:113:9
15:10:19.579 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 3 +0ms common.js:113:9
15:10:19.675 Firefox can’t establish a connection to the server at wss://node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD. client.ts:18:17
15:10:19.677 waku:connection-manager Error dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD - undefined +97ms common.js:113:9
15:10:19.677 waku:connection-manager Dialing peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD on attempt 3 +1ms

Something changed on the fleet/node side.

I was able to connect with the node last week

waku:connection-manager Dialing peer 16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA on attempt 1 +1ms
duplex.js:31 WebSocket connection to 'wss://node-01.do-ams3.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e' failed: WebSocket is closed before the connection is established.
(anonymous) @ duplex.js:31
close @ duplex.js:27
onAbort @ index.js:56
onAbort @ index.js:8
onAbort @ index.js:8
common.js:108 waku:connection-manager Error dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e - The operation was aborted +30s
common.js:108 waku:connection-manager Dialing peer 16Uiu2HAmL5okWopX7NqZWBUKVqW8iUxCEmd5GMHLVPwCgzYzQv3e on attempt 2 +0ms
duplex.js:31 WebSocket connection to 'wss://node-01.gc-us-central1-a.wakuv2.prod.statusim.net:8000/p2p/16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA' failed: WebSocket is closed before the connection is established.
(anonymous) @ duplex.js:31
close @ duplex.js:27
onAbort @ index.js:56
onAbort @ index.js:8
onAbort @ index.js:8
common.js:108 waku:connection-manager Error dialing peer 16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA - The operation was aborted +5ms
common.js:108 waku:connection-manager Dialing peer 16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA on attempt 2 +0ms
mfw78 commented 1 year ago

@mfw78 what's the setup on dappnode? I see some ws error, can we ensure that nwaku is configured using wss by default?

nwaku actually runs as ws, and not wss as TLS is handled by a proxy within dappnode. This proxy is responsible for auto-configuring over the SSL certificates etc.

danisharora099 commented 1 year ago

Something changed on the fleet/node side. I was able to connect with the node last week

Update: this issue is now resolved from nwaku side.

fryorcraken commented 1 year ago

@mfw78 what's the setup on dappnode? I see some ws error, can we ensure that nwaku is configured using wss by default?

nwaku actually runs as ws, and not wss as TLS is handled by a proxy within dappnode. This proxy is responsible for auto-configuring over the SSL certificates etc.

Fair but is nwaku in dappnode configured so that it advertised itself using the wss proxy multiaddr and NOT the ws one? looks like it's not always the case (advertise ws instead of wss).

danisharora099 commented 1 year ago

@fryorcraken i propose to make a separate issue/edit this issue to track work around common errors -- for eg:

this issue imo should not be a blocker, but parallel work to be done, for the milestone: https://github.com/waku-org/js-waku/issues/1429

fryorcraken commented 1 year ago

Regarding dappnode, we can open an orphan issue for now and track with @mfw78 to understand what can be done.

What is the second issue about? I'd suggest we investigate a bit more before considering peer exchange as done.

fryorcraken commented 1 year ago

Some more feedback:

fryorcraken commented 1 year ago 
* `Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Payload identity key does not match expected remote peer`

Interestingly enough this is a fleet peer used as a rendezvous peer:

common.js:113 libp2p:connection-manager:dial-queue:error error during dial of /dns4/node-01.gc-us-central1-a.wakuv2.prod.statusim.net/tcp/8000/wss/p2p/16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA/p2p/16Uiu2HAmEhN1JUTSVau1xf71a8z7web8b7kTMv2xZD6JwJ8UT19q +948ms CodeError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Payload identity key 16Uiu2HAmVkKntsECaYfefR1V2yCR79CegLATuTPE6B9TxgxBiiiA does not match expected remote peer 16Uiu2HAmEhN1JUTSVau1xf71a8z7web8b7kTMv2xZD6JwJ8UT19q

I think. Is that even the correct multiaddr format for rendezvous? cc @richard-ramos

Not sure it makes sense to dial rendezvous peers for filter purposes.

I wonder if it means that the 90+ nodes we are discovering are actually Status Desktop nodes reachable via rendezvous. This would make sense.

fryorcraken commented 1 year ago

Actually, if js-waku can connect to a Status desktop node via a fleet node thanks to ~rendezvous~ p2p-circuit to consume filter and lightpush that would be beneficial.

While the js-waku node would still hog a connection on the fleet (~rendezvous~ p2p-circuit) node and take bandwidth (but only to send messages and receive messages on specific content topic), the usage of memory resources to hold the subscription would be delegated to the Status Desktop node. I assume it would not consume one more connection on the status desktop node either as the connection to the ~rendezvous~ p2p-circuit node is re-used.

In terms of privacy, it would also mean that the fleet would not know the messages being sent by js-waku node, and may not know content topics of interest (except if store is used on fleet node).

@richard-ramos @waku-org/research is my analysis correct?

fryorcraken commented 1 year ago

Note that support for ~rendezvous~ p2p-circuit can be tracked in a browser scalability milestone yet to be created.

Edited.

chaitanyaprem commented 1 year ago

Actually, if js-waku can connect to a Status desktop node via a fleet node thanks to rendezvous to consume filter and lightpush that would be beneficial.

That is a very nice idea @fryorcraken . This way the load of light clients can be distributed to the nodes in the network rather than just the fleets making it more decentralized.

Wrt connection, same libp2p connection will be reused for multiple protocols as it is the same peer.

jm-clius commented 1 year ago

Actually, if js-waku can connect to a Status desktop node via a fleet node thanks to rendezvous to consume filter and lightpush that would be beneficial.

Agree here that connecting to distributed nodes for filter and lightpush is beneficial. Currently Status Desktop nodes do the same. What I'm confused about is why this necessarily has to be discovered via rendezvous? Status Desktop nodes are discoverable in most cases via discv5 and peer exchange. For now, we suggest peer exchange as better discovery mechanism than rendezvous, as it's less prone to neighbourhood-effects. Rendezvous is useful to discover nodes behind a NAT and coordinating a hole punch.

richard-ramos commented 1 year ago

I think. Is that even the correct multiaddr format for rendezvous? cc @richard-ramos

This is not a correct multiaddress format as it is missing the p2p-circuit component. Where is this address from? I did a problem similar with how go-waku was building multiaddresses, and it was fixed in this commit: https://github.com/waku-org/go-waku/commit/dbd94ebb81ba25641d2423f6fc0e8afe9c474196

richard-ramos commented 1 year ago

I assume it would not consume one more connection on the status desktop node either as the connection to the rendezvous node is re-used.

yeah, libp2p does stream multiplexing, so a single connection can be reused for multiple protocols

fryorcraken commented 1 year ago

@jm-clius my bad, wherever I said "rendezvous" I actually meant "p2p-circuit" Thanks @richard-ramos for jumping in.

This is not a correct multiaddress format as it is missing the p2p-circuit component. Where is this address from?

Maybe older Status Desktop versions? Would be good to see if we successfully find and connect to p2p-circuit addresses via peer exchange cc @danisharora099 And if it can't connect or can't find, then let's create an issue to track.

richard-ramos commented 1 year ago

@danisharora099 let me know if i can help. FWIW the go-waku test/prod fleets should have circuit relay enabled so maybe they can be used to force an scenario win which p2p circuit addresses appear in peer exchange.

danisharora099 commented 1 year ago

Thanks everyone for the feedback! Digesting it:

are peers discovered only wss peers?

no. they are all the peers received through the event emitter. do we only want to showcase wss peers on the counter?

fryorcraken commented 1 year ago

do we only want to showcase wss peers on the counter?

I think so. But maybe once we get p2p-circuit (confirmed) working

danisharora099 commented 1 year ago

closing this issue in favour of the following sub-issues derived: