Re-tune piece-reader to allow for random parallel access of data

[rjan90]

Re-tune the piece reader code to allow for random, parallel access to unblock retrieval bottlenecks.

Quote from Magik:

from what I’ve observed in booster-bitswap (which most of the time is serving ~150 blocks/sec), the bottleneck comes from the piece-reader code, which was optimized for sequential full-piece retrievals. Given that now we need more random-access reads of data in sectors, that bit of code needs to be re-tuned to allow random, parallel access (lack of which seems to be a major bottleneck in traces I’ve observed).

Just tuning piece-reader should be easy ~10x, for more we may need to educate SPs on how to store unsealed sectors (e.g. when using RAID configure large stripe size, so that only one/fewer drive heads need to move to read each block)

Estimated time effort: 1-2 days

[magik6k]

Some more context:

• PieceReader code lives in https://github.com/filecoin-project/lotus/blob/master/storage/sealer/piece_reader.go

• It is essentially just a simple wrapper around a func(offset int64) io.ReadCloser (which is basically just a getter for http-request-reader for unsealed sector data) into a io.ReadSeeker + a thread-safe io.ReaderAt.

• Today it is entirely optimized for a single, sequential, forward reader, which can occasionally skip forwards - which is roughly the workload we see from graphsync

• The reader setup function it operates on is not the cheapest thing to call - it's starting an http request, which may imply a new tcp stream, so each call is likely a couple ms.

  • So to mitigate that cost, for sequential reads, PieceReader may discard some data in case of a forward seek
  • Again, this only really makes sense in case of a single Graphsync retrieval
  • And the way the logic is implemented means that we can only have a single parallel stream reading from a PieceReader, which is a major bottleneck in some cases - e.g. random retrievals with bitswap

• The ask in this issue is that we need to rewrite this logic to support random, parallel ReadAt calls with better performance (e.g. actually support reading from the sector with multiple streams)

  • Bonus points for not sacrificing sequential retrieval efficiency too much, e.g. by having a small pool of readers, and being smart about which one we restart on a random read / reusing existing readers for sequential reads

[shrenujbansal]

@magik6k @rjan90 This domain/logic/code is completely foreign to me. Could you provide some basic instructions I can perform to witness the core issue locally? So I'm guessing something that performs a retrieval on an unsealed sector and takes long

[davidd8]

Following up here, as this is an important fix that will enable SPs to serve retrievals at scale, since retrieval traffic from Saturn and the wider IP network will request smaller (<< piece size) and "random" data from SPs, not just full Pieces. In practice and through load testing (testing results, http load testing tool and example), we've seen the current limitations, so we currently heavily rate-limit Saturn nodes from requesting from SPs.

A 10x performance improvement would help us increase the rate limit to a level that should enable SPs to handle all of Project Rhea's traffic, agnostic of transport protocol, which is why we should try to get this into Lotus sooner, rather than later. Can we get it into 1.23?

[brendalee]

Here is a link to the load testing tooling that @hannahhoward and team built for load testing: https://github.com/filecoin-project/retrieval-load-testing

Adding a bit more to what @davidd8 mentioned, we saw the impact of this specifically with the range request http load testing in Boost (this has similar access patterns as bitswap retrievals).