Handling large number of topic subscriptions

libp2p / go-libp2p-pubsub

The PubSub implementation for go-libp2p

https://github.com/libp2p/specs/tree/master/pubsub

Other

319 stars 185 forks source link

Handling large number of topic subscriptions #402

Open aschmahmann opened 3 years ago

aschmahmann commented 3 years ago

Currently if a node subscribes to enough topics such that the RPC that is used to send topic subscription announcements exceeds the maximum message size (defaults to 1MiB) then we'll silently fail and stop announcing any of our topics to newly connected peers.

We should more gracefully error, or inform the user, when they've exceeded the maximum number of topics they can subscribe to and additionally it'd be nice to have a mechanism in place to allow for a larger number of subscriptions.

[ ] (Beginner) Error when exceeding maximum topic limit
- Easiest: Just check if the (hello packet) message size is too large and log an error
- More correct: When subscribing to a new topic check if the new topic would force the hello packet to exceed the maximum size and if so return an error instead of allowing the topic to be subscribed to.
[ ] (Expert) Allow sending more topics than fit in a single message

Stebalien commented 3 years ago

Ideally, we'd be able to send a HAMT or a bloom filter of topics.

BigLep commented 3 years ago

@iand is working on this

BigLep commented 3 years ago

Beginner issue is to just get a proper error message.
This is back in the backlog.

BigLep commented 3 years ago

@aarshkshah1992 : for your rotation can you please take the error message at the minimum so that if this happens on Gateway we know about it?

samlaf commented 12 months ago

Is there a rough estimate of how many topics can be subscribed to using that 1MB limit?

lthibault commented 12 months ago

@samlaf I'd expect that to be hardware and network dependent.

samlaf commented 12 months ago

@lthibault Oh then I really have a bad understanding of this issue. I was assuming this was just saying that each subscription takes some amount of bytes, which need to be packed inside 1MB?

aschmahmann commented 12 months ago

@samlaf it's been a while since I wrote this issue and poked around in this codebase, but I think should be something like 1MiB/(sum of your topic name byte lengths + some overhead) and as a result it's a function of your topic names. i.e. you're encoding this https://github.com/libp2p/go-libp2p-pubsub/blob/d13e24ddc9f288557732c9c02b143c09b86c6778/pb/rpc.proto#L5-L15 and sending it over the wire and it's got to be under the max size.

lthibault commented 11 months ago

@samlaf As I understand it, the 1MB default limit pertains to the maximum size of messages transported over a pubsub topic. The maximum number of topics is therefore going to minimally depend on (1) network characteristics (2) available memory/cpu/etc of the machines, and (3) message transmission rate.