caps/wit.md: Ethereum Witness Protocol v0

[carver]

This simple budding protocol has been successfully tested among a couple cooperating clients. It is tremendously useful for Beam Sync. Let me know if there's anything I can do to help get it approved & merged!

Large chunks were shamelessly stolen from the snap protocol PR, so thanks to Peter for that.

[pipermerriam]

Love seeing this move forward. One thought (which I've already brought up privately with carver) is that this protocol might be best modeled similar to how the transaction pool messages work, with both a gossip, and request/response component.

• AdvertiseWitnessHashes -> gossip message to advertise what witness hashes you have available
• GetBlockWitnessHashes
• BlockWitnessHashes

Reasoning here is that a client which is beam syncing will have some amount of lag behind the head of the chain and we don't want to have to do things like polling a connected peer until they have a witness available. By having an explicit advertisement mechanism we can eliminate un-necessary requests that would require empty responses.

[gsalgado]

AdvertiseWitnessHashes -> gossip message to advertise what witness hashes you have available

Its name seems to imply the payload would contain the witnesses themselves, but I guess what we really need to know is the block hashes for which a given peer has witnesses, no? Maybe that's what you meant, but in that case I think a name like AdvertiseWitnessesForBlock would be better

[tkstanczak]

AdvertiseWitnessHashes -> gossip message to advertise what witness hashes you have available

Its name seems to imply the payload would contain the witnesses themselves, but I guess what we really need to know is the block hashes for which a given peer has witnesses, no? Maybe that's what you meant, but in that case I think a name like AdvertiseWitnessesForBlock would be better

I think it should be pull based. GetWitnessedBlocks() -> return which of the last 256 hashes the client has WitnessedBlocks()

[tkstanczak]

AdvertiseWitnessHashes -> gossip message to advertise what witness hashes you have available

Its name seems to imply the payload would contain the witnesses themselves, but I guess what we really need to know is the block hashes for which a given peer has witnesses, no? Maybe that's what you meant, but in that case I think a name like AdvertiseWitnessesForBlock would be better

I think it should be pull based. GetWitnessedBlocks() -> return which of the last 256 hashes the client has WitnessedBlocks()

eth65 changes the very inefficient behaviour of transaction broadcasting -> let us not repeat it also NewBlock NewBlockHint are methods to listen to.

[gsalgado]

GetWitnessedBlocks() -> return which of the last 256 hashes the client has

That seems wasteful to me as nodes will have to send GetWitnessedBlocks() requests quite frequently to avoid sending GetBlockWitnessHashes() to peers that don't have witnesses for a certain block, and each WitnessedBlocks() response will contain mostly the same data sent in a previous WitnessedBlocks(). Whereas with the gossip approach, every msg would (AIUI, but @pipermerriam can confirm?) contain only one block hash

[pipermerriam]

@tkstanczak

I'm not entirely following your two comments. They seem to propose different things.

Re-proposing what I proposed above with (hopefully) clearer language.

1. AdvertiseBlockWitnesses: List of block hashes that the client has witnesses available for. Clients would keep track of what hashes they've already sent and only include hashes that have not already been advertised.
2. GetBlockWitnesses: request witnesses by their block hash. A client can ensure that they only issue requests for witnesses that have been advertised ensuring that the majority of requests can be served.
3. BlockWitnesses: response message, including the requested witnesses.

IIUC this should satisfy both of @tkstanczak concerns. 1) The actual transmission of the witnesses is pull based. 2) Leveraging the efficiency that eth65 applied to transaction and block gossip.

Any objections?

[gsalgado]

@pipermerriam

In discussions with @carver we agreed that we can only store witness hashes for recent blocks, and since AdvertiseBlockWitnesses will only incrementally advertise new block hashes, there's a chance peers will make witness requests that we can no longer fulfill. I can think of two alternatives to avoid that:

• advertise witnesses we no longer have (which could possibly even be on the same message?)
• have the spec state that clients must keep witnesses for at least the last N block hashes advertised via AdvertiseBlockWitnesses, so that peers know there's no guarantee a witness for a given block hash will be available if N block hashes were advertised after it

Thoughts?

[tkstanczak]

I think that there are two roles really: full nodes that always can be expected to have witnesses and beam syncing nodes that have very inconsistent witnesses

[tkstanczak]

FullNode that participtes in wit protocol should have the witnesses for the last FULL_NODE_HISTORY_SIZE blocks (I would say 1024) BeamSync node that participates in wit protocol may have some of the witnesses (for example witnesses for blocks 800, 805, 806, 808) <- with gaps. Also during reorganizations both full nodes and beam sync nodes may have non-canonical witnesses.

I think in Nethermind we can advertise witnesses but we will probably ignore tracking what others advetise and will just keep asking other nodes about latest witnesses that we need (I think this strategy will be better for the synchronizing nodes game-theoretically).

[gsalgado]

@carver It just occurred to me we should probably introduce a Status msg, including the list of block hashes for which the node has witness hashes

[tkstanczak]

@carver It just occurred to me we should probably introduce a Status msg, including the list of block hashes for which the node has witness hashes

Do we expect the beam sync to ever sync historical blocks?

[tkstanczak]

I see it this way (BN, FN -> BeamNode, FullNode)

Scenario with status:

BN->FN Status (empty) FN->BN Status (full 256 blocks) BN->FN requests witness for the latest block only or does not issue any request FN->BN responds with witness

Scenario without status:

BN->FN requests witness for the latest block only FN->BN responds with witness or empty if missing

Second scenario is less verbose.

[tkstanczak]

Also I believe that BN should simply always send the GetBLockWitnessHashes to all the wit/0 nodes that send the NewBlockMessage to it.

[tkstanczak]

I would focus now on trying to get the blocks synced with this protocol and check how it behaves. This should give us some insight into what the problems could be.

[gsalgado]

Do we expect the beam sync to ever sync historical blocks?

I don't think so. Why?

[carver]

I would focus now on trying to get the blocks synced with this protocol and check how it behaves. This should give us some insight into what the problems could be.

I agree. It seems like we can just treat every NewBlock announcement from a peer with wit devp2p protocol as an announcement that they probably have the witness.

I don't expect clients to end up wasting much time polling for witnesses (like I guess can be a problem with transactions). We have so much information about the consensus about which blocks are at the tip, and which blocks our peers claim to have.

[fjl]

Is this ready for merging? It's implemented by several clients at this point.

[carver]

I'll clean up the lint issues, squash it, and mark it as ready for proper review & merge.