Total Difficulty in Header accumulator or not?

[kdeme]

Currently, the EpochAccumulator holds HeaderRecord which holds the total_difficulty field that gets updated on each BlockHeader update.

This issue is about whether we should restructure this object and/or remove total_difficulty from it.

cons: The inclusion of total_difficulty in a HeaderRecord means that to create an EpochAccumulator for a specific epoch, you need, next to the headers, access to the total difficulty from the last record of the previous EpochAccumulator.

This means no parallelization is possible when creating the MasterAccumulator and all EpochAccumulator. However, that is not really an issue as it is something that needs to be done once to inject into the Portal history network and it doesn't take that long when you already have all the headers downloaded & stored locally.

There might be other uses cases however where this is a an issue. One such case is for the storage of historical data in (era) files per epoch (8192 blocks), a hash_tree_root of the EpochAccumulator can be provided, so that the blocks in the file can be verified. However, with the total_difficulty included this would be more difficult as access to the previous total difficulty would also be required. But perhaps the solution here is just adding the previous total difficulty in the file? @lightclient

pros:

With the total_difficulty per block accessible, the totalDifficulty in the eth_getBlockByHash / eth_getBlockByNumber can be provided. For Portal specific, this does require a rather large object (EpochAccumulator) to be downloaded for just this value. This is not ideal in the first place. There might be other use cases where the value is useful that I'm not aware of.

(Another, minor, benefit is that when building the MasterAccumulator, one can find the merge point, not just by some hard defined merge block number, but by comparing the total difficulty with TTD.)

There could be some hybrid options too, for example where the MasterAccumulator gets split in a list of block hashes and a list of total_difficulties, or similar.

[lightclient]

Hi @kdeme. Yes seems like lot's of potential designs possible for this. Is there something specific you'd like to accomplish? The current format is sufficient for my needs, however it would be marginally simpler for me to remove the total difficulty altogether because it isn't necessary for my use case.

[pipermerriam]

@lightclient can you describe our explain your use case? In an EIP4444 context of geth fetches a historical block, how would it return totaldifficulty for

Sent from Proton Mail mobile

-------- Original Message -------- On Mar 9, 2023, 8:14 AM, lightclient wrote:

Hi @.***(https://github.com/kdeme). Yes seems like lot's of potential designs possible for this. Is there something specific you'd like to accomplish? The current format is sufficient for my needs, however it would be marginally simpler for me to remove the total difficulty altogether because it isn't necessary for my use case.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[lightclient]

My use case is simply focused on exporting, sharing, and importing the full pre-merge chain. Being able to return historical block over JSON-RPC w/o any additional processing is a bonus, but not something I am looking into actively right now. The nimbus team might be more interested in that.

I'm trying to avoid boiling the ocean here and get an initial agreed upon format out there so data providers can start providing historical blocks in bulk (e.g. eras) and users can import them to their client. The import works similarly to having a blocks.rlp file. The client has to process the blocks and ingest them to their own database. Along the way, the client likely has its own mechanism for tracking the total difficulty. Therefore, for my use case today, it isn't necessary.

[pipermerriam]

And you are onboard with using the format we are using that includes total difficulty? We'd very much like to use the same format/schema as you and we also would like to have total difficulty available to us.

@kdeme assuming lightclient says yes, if suggest you make an executive decision here on final

Sent from Proton Mail mobile

-------- Original Message -------- On Mar 9, 2023, 8:30 AM, lightclient wrote:

My use case is simply focused on exporting, sharing, and importing the full pre-merge chain. Being able to return historical block over JSON-RPC w/o any additional processing is a bonus, but not something I am looking into actively right now. The nimbus team might be more interested in that.

I'm trying to avoid boiling the ocean here and get an initial agreed upon format out there so data providers can start providing historical blocks in bulk (e.g. eras) and users can import them to their client. The import works similarly to having a blocks.rlp file. The client has to process the blocks and ingest them to their own database. Along the way, the client likely has its own mechanism for tracking the total difficulty. Therefore, for my use case today, it isn't necessary.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[lightclient]

Yes I am happy with the current format which has total difficulty. Definitely want to use the same format as you if possible!