Epic: Reducing the data transfer memory footprint on both client and miner

[raulk]

It has been reported that the memory footprint of storing and retrieving data with Lotus scales with the size of the payload, and it stays high during the data transfer phases.

This is a meta-issue that tracks reports, the desired outcome, and related work taking place across the board.

Reports

• https://github.com/filecoin-project/lotus/issues/4877 (Data Transfers are RAM hungry)
• https://github.com/filecoin-project/lotus/issues/5441 (Client memory usage is high when executing retrieving deal)

Desired outcome

• Memory overhead no longer scales linearly with payload size.
• Storing a ~32GiB file (i.e. largest file minus sector overhead) consumes max 1GiB of RAM during transfer.
• Storing a ~64GiB file (i.e. largest file minus sector overhead) consumes max 1.5GiB of RAM during transfer.
• Parallel transfers no longer scale linearly with concurrency level.

Work outline

This work outline will keep getting updated.

Legend

• 🌑 Unstarted
• 🚧 In progress
• 👀 In review
• ✅ Done

Reproducing the problem with a Testground test plan that involves data transfer

• go-graphsync already has a testground test plan: https://github.com/ipfs/go-graphsync/tree/master/testplans/graphsync.
• the testplan has recently been updated to remove red herring OOMs: https://github.com/ipfs/go-graphsync/pull/147.
• ⚠️ current findings show that throughput in ideal conditions is limited to 100Mbit/s -- issue incoming (@nonsense, please edit and add link)

Real-world benchmarks to exercise throughput and memory footprint of go-data-transfer and go-graphsync in isolation

• 3-dimensional benchmarks for go-graphsync: https://github.com/ipfs/go-graphsync/issues/150
• looks like go-data-transfer already has some useful benchmarks, but they probably need to be enhanced: https://github.com/filecoin-project/go-data-transfer/tree/master/benchmarks

Fixes

TBD from discoveries above.

[rjulius23]

What are the current benchmarks for memory usage for PreCommit1,2 and Commit phases ? For 1 sector ?

[raulk]

@rjulius23 this issue is concerned with data transfer. Could you please open a discussion with that question?

[rjulius23]

Sure sorry :)

[kernelogic]

I support this improvement!