Track in-flight IWANT requests

libp2p / go-libp2p-pubsub

The PubSub implementation for go-libp2p

https://github.com/libp2p/specs/tree/master/pubsub

Other

327 stars 186 forks source link

Track in-flight IWANT requests #285

Closed vyzo closed 4 years ago

vyzo commented 4 years ago

This came up in #284.

It is desirable to track in-flight IWANT requests, so that:

we don't repeatedly ask multiple peers for the same message when we are poorly connected to the mesh
we can track spammers who send us bogus message IDs and never follow up on the IWANT request in order to execute a distributed IHAVE flood attack.

Complexities, as enumerated by @raulk:

What if the peer we ask never sends back the message?
Do we queue up other candidates we know we can fall back on?
What is the grace period? And how does it relate to window sliding? (by the time we fall back to the next candidate, the message might have slid away from the cache)
Memory footprint of this new state to maintain.

whyrusleeping commented 4 years ago

This kinda just feels like bitswap

vyzo commented 4 years ago

yeah, but in real-time.

vyzo commented 4 years ago

Maybe one day bitswap can switch to gossipsub for supporting block discovery instead of using the dht. It would be much more efficient and it will avoid taxing the provider records system. One idea would be to find the root CID through the dht, using provider records, and then use a pubsub topic for handling block discovery. cc @Stebalien

vyzo commented 4 years ago

cc @dirkmc as well.

dirkmc commented 4 years ago

In Bitswap nodes remember which blocks their peers want for the lifetime of the connection. That probably wouldn't be necessary for gossipsub so it should be possible to have lower memory requirements.

aschmahmann commented 4 years ago

One idea would be to find the root CID through the dht, using provider records, and then use a pubsub topic for handling block discovery.

This was proposed in January, it's a pretty reasonable idea. However, there's some tricky work to be done in figuring out if/how we'd want IPFS to switch from a random access pattern to one that's slightly different (e.g. if you only provide/find the root then people can't string arbitrary blocks together into a new file and get all the providers of those blocks to help out, but it seriously helps on resource usage). libp2p/go-libp2p-kad-dht#559 is the first step along the path.

vyzo commented 4 years ago

So a solution for tracking IWANTs for non-follow up purposes without using too much memory, is to random sample the set of IWANTs and track only a small subset of them. This will allow us to respond to a distributed IHAVE flood attack without blowing the memory. It will not help with duplicate IWANTs, but that's less of a problem and less pressing to solve.