Sandboxed/namespaced datastores

[Stebalien]

Currently, we just give every subsystem access to the global datastore. We usually namespace our own keys (in each service) but this will bite us eventually. Ideally, service implementations would be free to pick their own key naming schemes (and would never try to use the datastore as an interface to steal information from other services like namesys does with the DHT).

Note: this is totally doable (and actually quite easy) with mounts. However, it would require a migration.

[magik6k]

I'll have a look at this

[magik6k]

I've looked at uses of datastore in the codebase:

• Blockstore: /blocks
• fsrepo fileManager: /filestore
• pin: /local/pins
• mfs: /local/filesroot
• republisher: global keys
• offline routing: global keys
• namesys/publisher: global keys
• namesys/pubsub: global keys
• dht: global keys
• name command: global keys

It looks like all the global stuff are just libp2p records (https://github.com/libp2p/go-libp2p-record).

I think that we should wrap the record stuff into some sort of record-store (js side seems to have something like this, not sure what exactly it does). This would have the advantage of allowing us to do stuff like pinning records which could be a fix for issues like https://github.com/ipfs/go-ipfs/issues/4749

[Kubuxu]

As a point, this needs coordination with JS team.

[Stebalien]

@Kubuxu sure... but I'd like that coordination to be "the datastore is not an interface".

@whyrusleeping should also chime in, there may be a reason we're doing things this way.

[whyrusleeping]

@Stebalien Yeah, currently we let each individual service namespace its keys. We could instead let them use the global namespace, and pass them a pre-namespaced datastore (ending up with the same structure we have no and no migration). The issue with this is that it hurts developer UX a bit, and makes it easier to accidentally have clashing namespaces using pre-existing services.

We can do it, but I don't think its that big of an issue. (I also don't think things like namesys reading the routing systems records is that terrible. Its weirder now that they are split out, but originally namesys was a very light layer on top of the dht that lived the next directory over.)

[Stebalien]

It's not that terrible... but then again, I use Emacs (:heart: global, mutable, shared namespaces).

ending up with the same structure we have no and no migration

Except that I'd expect DHT records to all be stored under a /kad-dht namespace, not /pk and /ipns as they currently are. That's the only place a migration would be necessary (unless we pass multiple namespaced datastores to the DHT).

However, this actually highlights one of my issues with the current system. Currently, /ipns/key maps to a generic record object, not an IPNS record. If we consider the datastore to be an interface, I'd expect it to map to an an IPNS protobuf.

and makes it easier to accidentally have clashing namespaces using pre-existing services.

Avoiding namespace clashes is actually my motivation for this issue. Basically, we need to coordinate namespaces somewhere. Better to coordinate namespaces in a single package (and delegate namespaces to the subpackages) than have the packages themselves coordinate on which namespace should have data formatted in what manner.

[daviddias]

@Stebalien still looking into this one? Shall we move it to ipfs/specs to continue the conversation?

[Stebalien]

SGTM but I can't seem to move anything to that repo.