Compact block filter Blockchain backend

[notmandatory]

This is an overview issue (based on same issue for murmel) for discussion and tracking the addition of a fully functional compact filters Blockchain backend to bdk. Ideally we'll have additional issues and/or PRs for each missing feature to link to this issue and we check off each feature as it's completed.

The below BIP-157/158 protocol features are based on the BIP-157 Client Operation section. Please suggest additions and corrections in the comments.

Missing protocol features for bdk are indicated below with an unchecked box [ ], completed features with a checked box [x].

Terminology

• peer: bitcoin full node peer
• block header: bitcoin standard block header
• filter header: BIP-157 compact block filter header
• block filter: BIP-157/158 compact block filter

Peer Discovery

1. • [x] Allow user to specify peer addresses
2. • [ ] Receive peer addresses via [addr] messages from connected peers
3. • [ ] Store new peer addresses
4. • [ ] If no provided or previously stored peers, find seed peer addresses via DNS

Connect to Peers

1. • [ ] Allow user to specify number of peers to connect to
2. • [ ] Allow user to specify minimum number of BIP-157 peers to connect to
3. • [x] Connect to peers and request signalling bits via [version] message
4. • [ ] Keep track of peers signalling service bit NODE_COMPACT_FILTERS
5. • [ ] Connect to specified number of peers
6. • [ ] Connect to minimum number of BIP-157 peers

Request Block Headers

1. • [x] Request block headers via [getheaders] message to connected peers
2. • [x] Request block headers from genesis to most work block, 1 to 2000 blocks per request
3. • [ ] Optional: Request block headers from stored checkpoint block instead of genesis block
4. • [ ] Optional: Request block headers from DNS (see https://bitcoinheaders.net/)

Receive Block Headers

1. • [x] Receive block headers [headers] from peers
2. • [x] Build chain of block headers from stored and received new block headers
3. • [x] Store new block headers and new block header tip
4. • [ ] Disconnect peers whose longest chain has significantly less work than known tip
5. • [ ] Request block filter headers from all peers for any new block header received

Request Filter Headers

1. • [x] Allow user to specify start block height for block filter sync
2. • [x] Request filter headers via [getcfheaders] message to peers
3. • [x] Request from start block height to end block hash (1-1000 per request)
4. • [ ] Request filter headers for the same block from more than one peer
5. • [ ] Optional: First request filter headers at intervals of 1000 via [getcfcheckpt] message
6. • [ ] Optional: Request filter headers from DNS (see https://bitcoinheaders.net/)

Receive Filter Headers

1. • [x] Receive filter headers [cfheaders] from peers
2. • [ ] Verify filter headers from multiple peers are the same
3. • [ ] If any received filter headers for a block are different,
     1. • [ ] request full block and compute correct block filter and filter header
     2. • [ ] ban peers that provided incorrect filter header
4. • [x] Build chain of filter headers from stored and received new filter headers
5. • [x] Store verified filter headers, at least to depth of 100
6. • [x] Request block filter for any new verified filter header
7. • [ ] Optional: store more than 100 verified filter headers

Request Block Filters

1. • [x] Request block filters after all filter headers are received and verified
2. • [x] Request block filters from start block via [getcfilters] message to peers
3. • [x] Request from start block height to end block hash (1-1000 per request)
4. • [ ] Optional: Check if filter header is for empty block filter if so skip request
5. • [ ] Optional: Request different block filters in parallel from multiple peers

Receive Block Filters

1. • [x] Receive block filter [cfilter] from peers
2. • [x] Verify block filters sequentially, must link to corresponding filter header
3. • [ ] Ban any peer that sends an invalid block filter
4. • [x] Check block filters for relevance
     1. • [x] Previous output scripts for transaction inputs
     2. • [x] ScriptPubKey for transaction outputs

Fetch Full Blocks

1. • [ ] If fetching full blocks, download from random peers
2. • [ ] Fetch full blocks from peers even if they don’t support compact block filters
3. • [x] Fetch full blocks that correspond to relevant block filters

[thomaseizinger]

There is https://github.com/cloudhead/nakamoto which should help implementing this.

[notmandatory]

Thanks having a chat with that team on Monday.

[johncantrell97]

Any updated thoughts on this after talking with Nakamoto folk? I'm interested in working on this whether it's from scratch or implementing Nakamoto as a blockchain backend for bdk.

[notmandatory]

The team is certainly open to PRs to integrate Nakamoto as an optional blockchain backend for bdk. I briefly discussed it with Alexis here: https://bitdevsla.org/socratic-seminar-18/

Or you can build on the existing bdk::blockchain::compact_filters module which currently supports connecting to a single peer (multiple times if you like to improve performance). We'd also love any PRs to improve this module to add multiple peer support and Sybil resistance. For now I've updated the above issue description to reflect what is implemented and integrated with bdk.

[notmandatory]

Current plan is to integrate Kyoto CBF node with BDK.

https://github.com/rustaceanrob/kyoto