search

Bitmessage / PyBitmessage

Reference client for Bitmessage: a P2P encrypted decentralised communication protocol:
https://bitmessage.org/wiki/Main_Page
Other
2.82k stars 575 forks source link

invbloom - New command for reduction of bandwidth #1404

Open PeterSurda opened 5 years ago

PeterSurda commented 5 years ago

Reason

After the version/verack handshake, Bitmessage sends a list of all inventory vectors to the other side. This causes a huge amount of bandwidth wastage, in particular for nodes that accept inbound connections. The proposal is to add a new command invbloom which will reduce this step in synchronisation.

This issue will be updated with more specifics as time goes on.

Data

After the command, there will be some metadata, and data. The bloom filter will use Murmurhash v3, just like Bitcoin's bloom filters. It's implemented in pybloomfiltermmap and it's very fast.

When to use

The support for this command will be indicated by a new bitfield in the version command. If both sides support this bitfield, instead of sending the list of all inventory vectors (referred to as "biginv" in PyBitmessage source code), the two nodes will send an invbloom.

Since a bloom filter allows for false positives, the invbloom command should be issued periodically to reduce the probability that an object goes missing. The delay should use a Poisson probability and pick a random outbound connection (similarly to how Dandelion works).

How to handle

Upon receiving invbloom, the node can send a list of inventory vectors that aren't in the bloom filter. This allows to reuse the rest of the protocol with no change.

Further development

Additional improvements can be added in the future with further protocol changes.

Anonymity

A different bloom filter should be used for each connection to prevent tracking of a node. I did tests with pybloomfilter and it looks like it isn't deterministic, so it probably isn't necessary to add random data to it.

hub229ox commented 5 years ago

sounds very good as long as precautions for anonymity are met like you describe :-) :1st_place_medal:

g1itch commented 5 years ago

The original pybloomfiltermmap code seems abandoned, last commit was in 2015. Also it probably contains serious bugs, e.g. https://github.com/axiak/pybloomfiltermmap/issues/84.

g1itch commented 5 years ago

Another option I found is pybloof

g1itch commented 5 years ago

Also I started to write tests for any bloom filter implementation: 9298b61