Implement EIP-4844 (proto danksharding)

[benjaminion]

Specification

Everything that builds on top of previous fork should build on top of Capella, so it is strongly suggested to wait Capella classes to be defined before introducing Eip4844 versions.

Decoupling EIP-4844 blobs

• [x] #6822

backlog

• [x] #7167
• [x] #7203
• [x] #6668
• [x] Optimize interface to Kzg library #6520 (we should minimize unnecessary data copy) block fetcher)
• [x] https://github.com/ConsenSys/teku/issues/6969

Nice to have

• [x] #7946
• [x] #7947
• [x] #7369
• [x] Add new blob related structures in JsonProviderPropertyTest
• [x] Package consistency (blobs vs blob and blocks vs block)
• [x] #6395
• [x] Issues under the KZG library (https://github.com/ConsenSys/jc-kzg-4844/issues)

Tasks below are prior to the blobs decoupling spec change, so may not be required

Configuration

• [x] #6377
• [x] #6378
• [x] #6379
• [x] #6382
• [x] #6608

SSZ Containers and schema

• [x] #6383
• [x] #6388
• [x] #6479

Spec

• [x] #6391
• [x] #6401
• [x] #6392
• [x] #6440
  • [x] BlindBlockUtilEip4844
  • [x] SignedBeaconBlockBlinderEip4844
  • [x] SignedBeaconBlockUnblinderEip4844
• [x] #6454
• [x] #6673

Validator coordinator

• [x] #6411

Storage

• [x] #6586
• [x] #6629
• [x] #6633
• [x] #6724

context:

• expected minimum space requirements (with current mainnet params) for the worst case scenario (maximum blob capacity): FIELD_ELEMENTS_PER_BLOB BLSFieldElement MAX_BLOBS_PER_BLOCK MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS SLOTS_PER_EPOCH = 409632 (131KB: max blob size) 16 (2MB max blobs sidecar) 4096 32 = 256GiB

Update: MAX_BLOBS_PER_BLOCK is now 4 so space requirements goes down to 64GiB

• we need to serve blobs:

  • by block root from the last finalized blobs to the most recent (to responds with a coupled SignedBeaconBlockAndBlobsSidecar)

  • by slot range for p2p (still need do be decided if coupled or decoupled from blocks)

Other things to consider:

• check "Additional notes" from here: https://github.com/ConsenSys/teku/pull/6636

Sync

• [x] #6669
• [x] #6691

HistoricalBlockSyncService

• [x] Implement blob sidecar backfill. This becomes critical since we can consider our head valid only if all blobs within MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS have been downloaded (verified against the corresponding block)

We have to bare in mind that we have to run validate_blobs_sidecar against the alleged coupled block to make sure that they actually couple.

Networking

• gossip
  • [x] #6460
  • [x] #6473
• request\response
  • [x] Implement BeaconBlocksByRange v2
  • [x] #6571
  • [x] #6584
  • [x] #6585
  • [x] #6602
  • [x] #6609
  • [x] BeaconBlocksByRootV2 is replaced by BeaconBlockAndBlobsSidecarByRootV1 when 4844 activates. We have to continue using BeaconBlocksByRootV2 until 4844 fork finalizes, to cover the case in which we want to lookup blocks from previous fork. (https://github.com/ethereum/consensus-specs/pull/3089) (covered in #6691)
  • [x] #6638

interesting point from Lion: https://github.com/ethereum/consensus-specs/issues/3113

Beacon API

• [x] #6710

Engine API

• [x] #6387

KZG Crypto library

• [x] #6389
• [x] #6470

backlog

• [x] #6422
• [x] Follow https://github.com/ethereum/consensus-specs/pull/3145 preset changes when merged
• [x] #6634
• [x] Enable 4844 reference tests
• [x] Enable Capella operations when it will hit EIP-4844 reference test (should be v1.3.0). It was disabled in our code and it's enabled finally in spec.
• [x] change tx type from 0x05 to 0x03 https://github.com/ethereum/consensus-specs/pull/3317

[benjaminion]

Implementation notes for consensus clients.

[benjaminion]

4844 Readiness Checklist

[lucassaldanha]

@tbenr Are we happy to close this epic now?

[mehdi-aouadi]

@lucassaldanha there is only #7946 left, will close the epic once it's merged

[mehdi-aouadi]

All done now 🥳