Open kdeme opened 1 year ago
Hi @kdeme. Yes seems like lot's of potential designs possible for this. Is there something specific you'd like to accomplish? The current format is sufficient for my needs, however it would be marginally simpler for me to remove the total difficulty altogether because it isn't necessary for my use case.
@lightclient can you describe our explain your use case? In an EIP4444 context of geth fetches a historical block, how would it return totaldifficulty for
Sent from Proton Mail mobile
-------- Original Message -------- On Mar 9, 2023, 8:14 AM, lightclient wrote:
Hi @.***(https://github.com/kdeme). Yes seems like lot's of potential designs possible for this. Is there something specific you'd like to accomplish? The current format is sufficient for my needs, however it would be marginally simpler for me to remove the total difficulty altogether because it isn't necessary for my use case.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>
My use case is simply focused on exporting, sharing, and importing the full pre-merge chain. Being able to return historical block over JSON-RPC w/o any additional processing is a bonus, but not something I am looking into actively right now. The nimbus team might be more interested in that.
I'm trying to avoid boiling the ocean here and get an initial agreed upon format out there so data providers can start providing historical blocks in bulk (e.g. eras) and users can import them to their client. The import works similarly to having a blocks.rlp
file. The client has to process the blocks and ingest them to their own database. Along the way, the client likely has its own mechanism for tracking the total difficulty. Therefore, for my use case today, it isn't necessary.
And you are onboard with using the format we are using that includes total difficulty? We'd very much like to use the same format/schema as you and we also would like to have total difficulty available to us.
@kdeme assuming lightclient says yes, if suggest you make an executive decision here on final
Sent from Proton Mail mobile
-------- Original Message -------- On Mar 9, 2023, 8:30 AM, lightclient wrote:
My use case is simply focused on exporting, sharing, and importing the full pre-merge chain. Being able to return historical block over JSON-RPC w/o any additional processing is a bonus, but not something I am looking into actively right now. The nimbus team might be more interested in that.
I'm trying to avoid boiling the ocean here and get an initial agreed upon format out there so data providers can start providing historical blocks in bulk (e.g. eras) and users can import them to their client. The import works similarly to having a blocks.rlp file. The client has to process the blocks and ingest them to their own database. Along the way, the client likely has its own mechanism for tracking the total difficulty. Therefore, for my use case today, it isn't necessary.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Yes I am happy with the current format which has total difficulty. Definitely want to use the same format as you if possible!
Currently, the
EpochAccumulator
holdsHeaderRecord
which holds thetotal_difficulty
field that gets updated on eachBlockHeader
update.This issue is about whether we should restructure this object and/or remove
total_difficulty
from it.cons: The inclusion of
total_difficulty
in aHeaderRecord
means that to create anEpochAccumulator
for a specific epoch, you need, next to the headers, access to the total difficulty from the last record of the previousEpochAccumulator
.This means no parallelization is possible when creating the
MasterAccumulator
and allEpochAccumulator
. However, that is not really an issue as it is something that needs to be done once to inject into the Portal history network and it doesn't take that long when you already have all the headers downloaded & stored locally.There might be other uses cases however where this is a an issue. One such case is for the storage of historical data in (era) files per epoch (8192 blocks), a hash_tree_root of the
EpochAccumulator
can be provided, so that the blocks in the file can be verified. However, with the total_difficulty included this would be more difficult as access to the previous total difficulty would also be required. But perhaps the solution here is just adding the previous total difficulty in the file? @lightclientpros:
With the total_difficulty per block accessible, the
totalDifficulty
in theeth_getBlockByHash
/eth_getBlockByNumber
can be provided. For Portal specific, this does require a rather large object (EpochAccumulator) to be downloaded for just this value. This is not ideal in the first place. There might be other use cases where the value is useful that I'm not aware of.(Another, minor, benefit is that when building the
MasterAccumulator
, one can find the merge point, not just by some hard defined merge block number, but by comparing the total difficulty with TTD.)There could be some hybrid options too, for example where the
MasterAccumulator
gets split in a list of block hashes and a list of total_difficulties, or similar.