filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.84k stars 1.26k forks source link

Add ChainExportRange API method #8641

Open travisperson opened 2 years ago

travisperson commented 2 years ago

Checklist

Lotus component

Improvement Suggestion

The ChainExport command doesn't provide an accurate way to export a segment of the lotus chain.

For archival and testing purposes it would be beneficial to be able to export an exact range of tipsets by adding a new method ChainExportRange that accepted two TipSetKey values as arguments and does not enforce a minimum epoch height between them (eg a call with both arguments the same should export a single TipSet).

The ChainExportRange should also accept a set of options

It will be important to think through the exact inclusiveness of the method to ensure that no data is missed under the following usage:

ChainExportRange(ChainGetTipSetByHeight(a), ChainGetTipSetByHeight(b))
ChainExportRange(ChainGetTipSetByHeight(b), ChainGetTipSetByHeight(c))

I think ChainGetTipSetByHeight is the correct method to use here, I think as long as it's consistent it should be fine

The above should include the same data as

ChainExportRange(ChainGetTipSetByHeight(a), ChainGetTipSetByHeight(c))

Additionally a command should be added to lotus-shed to support the usage of this API method, mostly because chain export enforces a minimum epochs to export.

The export should construct CARv2 files with the index.

travisperson commented 2 years ago

If we want to go crazy we could also add options to make including state and blockheaders optional as well, which would allow this method to be used to extract just the messages. However, ChainGetMessagesInTipset already exists and can be used to grab this data and pack a car file manually.

travisperson commented 2 years ago

With support of message receipts this will close out https://github.com/filecoin-project/lotus/issues/3849

travisperson commented 2 years ago

This export should allow for exporting of CARv2 files with the index.