rvagg
commented
1 year ago @bajtos the only purpose of lassie needing to keep temporary files is to handle the duplicate blocks case in a traversal, which we can't avoid. Essentially we need a place to put blocks we've fetched in case we discover later in a traversal that we need to look at it again.
There's a secondary use-case for a temporary store in parallelising bitswap requests - we fetch ahead (preload) of the blocks we'll need when we discover them, so as we proceed with a linear traversal we discover that the blocks we want are already in our temporary store. It really speeds things up. Currently we put these blocks in the same per-retrieval temporary pool (CAR) which gets cleaned up after a request.
It's not a straightforward thing to deal with, but here's some thoughts about this topic that maybe we could riff off:
- Firstly, Lassie should clean up its temporary files on a per-request basis, if it's not then something wrong. The only case where Lassie should be leaving temporary files around is in the case of a hard termination where it doesn't have a chance to clean up. Once a request has ended, or been closed, then any temporary files should be cleaned up.
- The
dupsparameter makes this interesting, both more difficult and less difficult depending on how you look at it.- If we are fetching from an http upstream that we know properly supports
dups=ythen we don't need to care about a blockstore, but we don't have clear guarantees at the moment, although the main http upstream implementation (eipfs) only doesdups=yand I think we could get fairly easily to a place where we have good guarantees about this. So potentially for http retrievals if we can force through adups=ythen we wouldn't need a temporary store. - If we had a strictly
dups=nretrieval, where lassie isn't expected to provide duplicates on the way out, then we do have a path to skipping blocks that we've already seen, we just haven't used this yet. The traversals we use are relatively simple and linear, they don't have the kind of scenarios where you need to revisit the same block and make different decisions (there are branching selectors where this happens, but since we don't support arbitrary selectors we don't have to worry about that).
- If we are fetching from an http upstream that we know properly supports
- For bitswap retrievals we can do what we want, including re-fetching blocks if we don't have them. But our preload still needs somewhere to put blocks if we want parallelism—although we have a flag to turn "bitswap-concurrency" down to 0 to stop this behaviour. Also possible is to have the preloader put blocks into a separate place where they can be deleted as used, or even just memory—it's just that it could get fairly large. Since we use CARs for these temporary stores they're append-only so we don't have the ability to "delete" when used; but we could use an alternative block store that does (I'm also not sure what the scale of savings would be to have this facility for the preloader, it's going to be heavily dependent on the type and size of DAG).
- For graphsync we can't really avoid this, yet, it expects to have the ability to look at blocks again. Something we haven't experimented too much with yet but there's potential for us to just say "nope, skip this block" if it does a re-lookup of the same block and we have the confidence it doesn't need it (this kind of thing is supported through traversal code, just not exercised by us yet).
- I have a TODO (mainly in my head tbh) to write an "overflow" blockstore to use as temporary storage. These little temporary files for very small fetches; which are the most common in the Station use-case, mean you have to get disk i/o involved which slows things down a bit. So the idea here would be to have a blockstore that is allowed to use up to X Mb of memory, above which it spills over to disk. Perhaps this is a per-retrieval number or perhaps it's an across-all-retrievals number. Then the user can decide whether to trade off RAM for disk. For large retrievals it's not going to be too much help though.
Having said all of that, I think there's a path to having some kind of --notemp flag, but opting in to that would mean a reduction in functionality. i.e. no dups=y output, potentially turning off one or more retrieval protocols depending on the work needed to make them work properly for this case, and either turning off the bitswap preloader, or doing something more creative with those temporary blocks. There's a fair bit of work here in both doing the plumbing but also making sure we have enough test coverage for cases where it matters—most synthetic DAG tests end up not needing temporary storage, it's always edge cases where you end up with surprising blow-ups.
I can imagine plugging in our custom block store
Since you're not using Lassie as a library, but as a daemon, what might this look like? We'd presumably need an API to talk to a blockstore across some RPC boundary.
distora-w3
bajtos
We are running Lassie in Filecoin Station - an app that runs on desktop computers of (possibly non-technical) users. We want Stations to be unobtrusive to the user and leave plenty of resources (e.g. free disk space) for user workloads.
To achieve that, we would like to limit how much space Lassie can use for temporary files.
A new configuration option for Lassie (the library) or Lassie Daemon (the HTTP server) would be ideal.
How easy or difficult would the implementation be? Are there any other options that would allow us to limit the maximum amount of disk space used? For example, I can imagine plugging in our custom block store if the Lassie library/daemon support that.
/cc @juliangruber