search

holepunchto / corestore

A simple corestore that wraps a random-access-storage module
https://docs.holepunch.to
MIT License
72 stars 31 forks source link

Allow to create in-memory hypercores #16

Open Frando opened 4 years ago

Frando commented 4 years ago

What do you think about adding a { persist: false } option to corestore.get? It would create an in-memory hypercore that behaves just the same as stored hypercores, with regard to both key derivation (from the master key and namespaces) and replication.

The dat-sdk currently has some code to "patch in" in-memory cores into corestore's replication logic, using internal methods. This could be removed then. There's several usecases for replicated in-memory cores I think.

mafintosh commented 4 years ago

Can you describe the use case for this a bit more? Makes things inside the corestore much more complex in terms of lifecycle when storage isn't consistent

RangerMauve commented 4 years ago

I use it in applications where I want to load data temporarily and throw it out after I'm done while using the same corestore to manage persistent data.

RangerMauve commented 4 years ago

Flow could be something like:

mafintosh commented 4 years ago

You can use hypercore(ram) for that with no added complexity currently. Not sure what’s gained by adding that to the corestore.

RangerMauve commented 4 years ago

If you use hypercore(ram) you need to set up the swarming for it separately from the stuff you're already doing for the corestore. 😁 It can be nice to leverage connection reuse / reduce overall network traffic.

andrewosh commented 4 years ago

Adding this to corestore would definitely simplify getting the temporary cores hooked up to networking, but as @mafintosh said it'd come with a complexity cost. I think the need for this motivates one or more of four things:

  1. A corestore method that temporarily inserts an externally-managed core into a corestore's caches + replication streams.
  2. A userland module that wraps a single-corestore networking module (like corestore-swarm-networking) and makes it easy to push other cores into its replication streams.
  3. A corestore deletion API (which I know you're keen on @RangerMauve, but which we've been hesitant to do until a good approach to key backups is available).
  4. A corestore persist: false option.

The persist: false option seems the least desirable because of its complexity -- the corestore API surface/internals should be as small/simple as possible, and we're already pushing it IMO!

I'd personally opt for (2) and/or (3). (2) because it's userland which is always nicer, and (3) because it's something we need to tackle anyway, so we'd get many benefits elsewhere.

RangerMauve commented 4 years ago

For what it's worth (3) could suit the needs of temporary cores, and it doesn't need any special APIs inside corestore. You could do feed.destroyStorage() which'd do the trick. 🤷 I think it'd require developers to be aware that they should clean up cores they don't want stored anymore.