shutter-network / rolling-shutter

Rolling Shutter is an MEV protection system to be plugged into rollups.
https://twitter.com/project_shutter/
26 stars 7 forks source link

Improve Gnosis keyper p2p connections #437

Closed jannikluhn closed 1 month ago

jannikluhn commented 4 months ago

In my testing setup, the Gnosis keypers don't create the gossiping mesh properly. This leads to high propagation times as messages are not gossiped immediately but only forwarded on request.

Logs

On startup, the keyper (12D3KooWRmoxBapB4EBZDK74R4yGwjL8uhgrPiRGRmNnbQuDMt7U) connects to the configured bootstrap node (12D3KooWMyutShWdqYj7fre4Vjuq2QnCTb26Ki1KpyDyVsrmKeki):

2024-03-01T11:38:29.675+0100    DEBUG   swarm2  swarm/swarm_dial.go:240 dialing peer    {"from": "12D3KooWRmoxBapB4EBZDK74R4yGwjL8uhgrPiRGRmNnbQuDMt7U", "to": "12D3KooWMyutShWdqYj7fre4Vjuq2QnCTb26Ki1KpyDyVsrmKeki"}

Soon after, they PRUNE the connection to it:

2024-03-01T11:38:29.776+0100    DEBUG   pubsub  go-libp2p-pubsub@v0.10.0/gossipsub.go:849   PRUNE: Remove mesh link to 12D3KooWMyutShWdqYj7fre4Vjuq2QnCTb26Ki1KpyDyVsrmKeki in decryptionKeys

Later, they try to connect to another peer (12D3KooWLyURWEKUX9uGnx8wMTR7svUneR7GqE7SzRVU5Gs2cdk6) whose contact info they should have received from the bootstrap node during pruning. However, this fails:

2024-03-01T11:38:29.776+0100    DEBUG   pubsub  go-libp2p-pubsub@v0.10.0/gossipsub.go:966   error connecting to 12D3KooWLyURWEKUX9uGnx8wMTR7svUneR7GqE7SzRVU5Gs2cdk6: failed to dial: failed to dial 12D3KooWLyURWEKUX9uGnx8wMTR7svUneR7GqE7SzRVU5Gs2cdk6: no addresses

It seems like the keyper did not receive the peer's address, even though they know the peer identity.

Effect

As a result of missing mesh peers, the propagation and subsequently key generation time is very large:

2024/03/01 12:39:07.095271 DBG [       keyper.go:349] sending decryption trigger block-number=24 num-identities=1 tx-pointer=0 tx-pointer-age=1
...
2024/03/01 12:39:08.152568 INF [messagingmiddleware.go:263] sending keys block=24 num-keys=1 num-signatures=2 tx-pointer=0

The time difference is ~1s, but should be on the order of milliseconds.

jannikluhn commented 4 months ago

It is to be expected that peer exchange only sends the peer id, but not the address. In this case, we are supposed to look up the peer in the DHT: https://github.com/libp2p/go-libp2p-pubsub/blob/048a4d30d0c3c00829181fd81aa697eb60497448/gossipsub.go#L1877-L1879

However, we disable the DHT: https://github.com/shutter-network/rolling-shutter/blob/8414b9e14582546be2fa81a074b9ff76c480050e/rolling-shutter/p2p/messaging.go#L88-L91

Simply setting those values to true does not do the trick unfortunately. Apparently, the DHT stays empty:

2024-03-01T18:38:56.924+0100    WARN    pubsub  go-libp2p-pubsub@v0.10.0/discovery.go:207   bootstrap: error providing rendezvous for decryptionKeys: failed to find any peer in table
jannikluhn commented 1 month ago

Done