feature: subscribe reorg only headers

[jsvisa]

Rationale

Ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newheads we have a newHeades subscribe,

Fires a notification each time a new header is appended to the chain, including chain reorganizations. Users can use the bloom filter to determine if the block contains logs that are interested to them. Note that if geth receives multiple blocks simultaneously, e.g. catching up after being out of sync, only the last block is emitted.

In case of a chain reorganization the subscription will emit the last header in the new chain. Therefore the subscription can emit multiple headers on the same height.

In some cases, we only want to retrieve the blocks that have undergone reorg, such as for ETL programs, which update old data once a reorg occurs. However, the newHeads subscription does not differentiate between whether it is a reorg situation, and this needs to be determined by the subscriber themselves.

Implementation

Do you have ideas regarding the implementation of this feature? // yes Are you willing to implement this feature? // yes

[lightclient]

What would you like this feature to look like?

[jsvisa]

Here is a simple workflow of this RPC.

Subscribe to chain reorg with params:

• reorgAdd: subscribe to newChain added events
• reorgDrop: subsribe to oldChain deleted events
• reorg: combines of reorgAdd + reorgDrop

eg:

{ "id": 1, "method": "eth_subscribe", "params": ["reorg"] }

response:

{
  "jsonrpc": "2.0",
  "method": "eth_subscription",
  "params": {
    "result": {
      "reorgAdd": [
        {
          "difficulty": "0x15d9223a23aa",
          "extraData": "0xd983010305844765746887676f312e342e328777696e646f7773",
          "gasLimit": "0x47e7c4",
          "gasUsed": "0x38658",
          "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
          "miner": "0xf8b483dba2c3b7176a3da549ad41a48bb3121069",
          "nonce": "0x084149998194cc5f",
          "number": "0x1001",
          "parentHash": "0x7736fab79e05dc611604d22470dadad26f56fe494421b5b333de816ce1f25701",
          "receiptRoot": "0x2fab35823ad00c7bb388595cb46652fe7886e00660a01e867824d3dceb1c8d36",
          "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
          "stateRoot": "0xb3346685172db67de536d8765c43c31009d0eb3bd9c501c9be3229203f15f378",
          "timestamp": "0x56ffeff8",
          "hash": "0x0167ffa60e3ebc0b080cdb95f7c0087dd6c0e61413140e39d94d3468d7c968a0"
        }
      ],
      "reorgDrop": [
        {
          "difficulty": "0x15d9223a23aa",
          "extraData": "0xd983010305844765746887676f312e342e328777696e646f7773",
          "gasLimit": "0x47e7c4",
          "gasUsed": "0x38658",
          "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
          "miner": "0xf8b483dba2c3b7176a3da549ad41a48bb3121069",
          "nonce": "0x084149998194cc5f",
          "number": "0x1000",
          "parentHash": "0x7736fab79e05dc611604d22470dadad26f56fe494421b5b333de816ce1f25701",
          "receiptRoot": "0x2fab35823ad00c7bb388595cb46652fe7886e00660a01e867824d3dceb1c8d36",
          "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
          "stateRoot": "0xb3346685172db67de536d8765c43c31009d0eb3bd9c501c9be3229203f15f378",
          "timestamp": "0x56ffeff8",
          "hash": "0x0167ffa60e3ebc0b080cdb95f7c0087dd6c0e61413140e39d94d3468d7c9689f"
        }
      ]
    },
    "subscription": "0x9ce59a13059e417087c02d3236a0b1cc"
  }
}

So the client knows that block 0x1000 with hash 0x0167ffa60e3ebc0b080cdb95f7c0087dd6c0e61413140e39d94d3468d7c9689f was dropped, and block 0x1001 with hash 0x0167ffa60e3ebc0b080cdb95f7c0087dd6c0e61413140e39d94d3468d7c968a0 was added.

[s1na]

I see many people have reacted to this feature request. I'd like to hear more about your use-cases (cc @dkatzan who commented on the related PR).

For now I want to highlight an approach of detecting re-orgs using the existing newHeads subscription:

• The requirement here is you need to store the block header for the new heads that you receive. The more you store, the larger of a re-org you can handle. E.g. 1 day's worth of blocks.
• Let's say the latest block you have locally has number N. Upon receiving a new head you have the following conditions:
  • New head has number < N => reorg happened and the new chain is shorter. Iterate back and check parent hashes to find the fork point.
  • New head has number == N => reorg happened. Same as above.
  • New head has number > N. In this case it could be just that we receive a new block on the existing chain OR it's a reorg that is longer than the previous chain. Iterate back and fetch blocks via (eth_getBlockBy..) and check parent hashes. If parentHash of N+1 doesn't match with what's stored locally it was a reorg situation.