Redesign Proposal: write snapshots to disk and implement sidecar design

[ognots]

Problem Statement: Using the Lotus API, snapshot production is slow due to the constraints of Lotus's API. The average snapshot job takes 4h on average, post FVM upgrade.

Possible Solution: Reports from other operators shows snapshot production being exponentially faster writing snapshots directly to disk. Many report <1h snapshot production. Draft a design doc and prototype of a "sidecar" design, where filecoin-chain-archiver runs in a process with read-only access to a volume where Lotus can write snapshots directly to.

[ognots]

Design 1:

[ognots]

the first prototype of this is working successfully in theory, in https://github.com/filecoin-project/filecoin-chain-archiver/pull/67 there are however a few relevant blockers:

• https://github.com/filecoin-project/lotus/issues/10672 - by far the largest blocker, the ExportRangeInternal method curently returns an EOF error. this is reproduceable with lotus CLI and using filecoin-chain-archiver create
• https://github.com/filecoin-project/lotus/pull/10602 - needs reviewed and merged for this design to work. current implementation in lotus makes the filename indeterminate from automation, because the method names the file with a future timestamp