search

libp2p / go-libp2p-pubsub

The PubSub implementation for go-libp2p
https://github.com/libp2p/specs/tree/master/pubsub
Other
319 stars 185 forks source link

GossipSub Max Message Size #401

Closed iNDicat0r closed 3 years ago

iNDicat0r commented 3 years ago

Publishing a message using the func (p *PubSub) Publish(...) method requires a topic and byte slice. When reading the message using Subscription.Next we get a protobuf message which contains the data received. However the problem is that there is a protobuf message size limit by google, in some cases 64MB is the max size. How does go-libp2p-pubsub handle larger messages ?

vyzo commented 3 years ago

The default max message size is 1MB; you can change this with the WithMaxMessageSize option.

However, I really don't recommend sending 64MB messages over pubsub because of amplifcation; if your messages are that large, it's better to send a content identifier and pull it with a side protocol.

TianshuDing commented 3 years ago

I have the question about Max Message Size. As you say, it's better to send a content idertifier and pull it with a side protocol. So, if you can, please show me some examples about this situation. And if I want to send a message to a single peer in the topic not broadcast to every peers in the topic, what option need to config the gossipPubSub?

vyzo commented 3 years ago

check go-libp2p-pubsub-router, it uses a side protocol for last write persistence.

On Mon, Apr 26, 2021, 09:30 TianshuDing @.***> wrote:

I have the question about Max Message Size. As you say, it's better to send a content idertifier and pull it with a side protocol. So, if you can, please show me some examples about this situation. And if I want to send a message to a single peer in the topic not broadcast to every peers in the topic, what option need to config the gossipPubSub?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/libp2p/go-libp2p-pubsub/issues/401#issuecomment-826544806, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAI4SUSCDDFIYDDZ72MW43TKUCA5ANCNFSM4YNA23GA .

TianshuDing commented 3 years ago

Thank you for your advice, any really project using this? And what about the second question, do not broadcast, just send to one peer.

vyzo commented 3 years ago

yes, ipfs uses it.

On Mon, Apr 26, 2021, 12:20 TianshuDing @.***> wrote:

Thank you for your advice, any really project using this? And what about the second question, do not broadcast, just send to one peer.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/libp2p/go-libp2p-pubsub/issues/401#issuecomment-826665428, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAI4SX3S73JSWSLXZVAVITTKUV5HANCNFSM4YNA23GA .