Server API for databases and data

[gr2m]

:mag_right:👀 Context

The current Hoodie has no APIs on the server side to create new databases, set security, replicate or add/find/update/remove data. There was no need for it yet, because there are no plugins yet and all that the Store Server would do is proxy requests either to CouchDB or to express-pouchdb. The code to create / destroy user databases based on account signups / destroys lives in hoodie/server/utils/user-databases.js

:raising_hand::bulb: Suggestion

I would suggest to provide a server API for simple handling of all things data:

• databases: create / list / rename / destroy
• security: grant / revoke read/write access
• replications: one-time or continuous (persistent)
• data: add / find / update / remove

I’d implement it similar to the Account Server API as part of Store Server even though it wouldn’t use it itself.

💻💭 Dreamcode

require('@hoodie/store-server/api') would return a factory that requires PouchDB as its only argument and then returns a Store constructor.

var PouchDB = require('pouchdb')
var Store = require('@hoodie/store-server/api')(PouchDB.defaults(options))

The Store constructor will be accessible to plugins at server.plugins.store.api.Store

Databases

The Store constructor has class methods to manage databases

// create mydb database
Store.create('mydb')
.then(function (name) {
  // resolves with store instance
})

// destroy
Store.destroy('mydb')
.then(function (name) {})

Security

Inspired by our dreamcode from the shares plugin. grant and revoke methods return promises.

// public read
// everyone who knows the name of the database can read its data
Store.grant('mydb', {access: 'read'})
// public read/write
// everyone who knows the name of the database can read from and write to it
Store.grant('mydb', {access: 'write'})
// private share
// using roles
Store.grant('mydb', {access: 'read', role: ["id:123", "group:456"]})
// revoke all access
Store.revoke('mydb', {access: ['read', 'write'])

// ideas for later
// - password protection
// - invite using roles (e.g. for group accounts)
// - find stores a user has access to

You can also pass security when creating a database

Store.create('mydb', {access: 'read'})

Replications

Replicate data from one database to another. Same as PouchDB’s replication API. Live replications are persisted and resumed when server restarts.

// replicate all documents with publishedAt property set from mydb to mypublicdb
Store.replicate('mydb', 'mypublicdb', {live: true, filter: function (doc} { return !!doc.publishedAt })})

Data

A Store instance exposes the Hoodie Store API

Store.open('mydb')

.then(function (store) {
  store.add({foo: 'bar'}).then(function (doc) {})
})

👪💬 Discussion

This is a corner stone for our village release milestones, we should ship at least parts of these APIs before re-enabling plugins. There is no rush with the implementation, but at least we should agree that the APIs are good to move forward.

Let me know what you think!

[jameswestnz]

Looking good! Cool to see this moving, as I've had to hack this together in our app :/

Just an initial thought: Could we merge the following methods?

Store.grantAccess()
Store.grantWriteAccess()

Something like:

Store.grantAccess('mydb', ['username1', 'username2', ...], {
  read: true,
  write: false
})

This means we could have other rights in the future - like security for adding users to a db, etc. We may also be able to shorten the method grantAccess to simply grant, as the third parameter is defining the "access".

Also, while a Store doesn't exist, this makes sense: Store.create('mydb'), however it feels that once you're dealing with an existing store, the syntax should be something like Store('mydb').grant(...)

Not sure if this helps?

[jameswestnz]

As mentioned on slack (edited):

I’m been keen to understand a bit more about how the PouchDB option could work. (mentioned by @janl on slack)

In the absence of what is outlined above, we have built(hacked might be a better word) something similar as we had a need to create DBs for different entities.

Our structure is something like this: Entities have admins and employees. Admins can create and assign jobs to employees. An employee can be “upgraded” to an admin.

So our Hoodie/CouchDB structure becomes: When a entity is created, I add this new entity to a “entities” db, then create two new dbs specific to that entity - entity/12345 and entity/12345/admin. The latter because there is certain info I don’t want employees having access to, and I couldn’t be bothered with filtered replication for this task (possibly the wrong move, not sure yet). I then simply add a users’ role to the entities db based on their role in the entity. So if a user is an “employee”, I only add their id to the entity/12345 db. Now when an admin adds a job to this db it gets synced to the user, etc. So what is outlined above in theory helps with a large portion of our process.

There are a number of other solutions I considered, too. One option was to replicate individual docs from the entity db to the users personal db. This solution would be more efficient longer term, but has a slightly more complicated setup. And I think that regardless of these two options, I wanted to have a entity/12345 db so that all information for a given entity was stored in a common place.

Maybe this situation helps expand on the concept a little?

CC @gr2m @janl @boennemann

[jameswestnz]

It's worth pointing out that if I went with my other option I mentioned above (per-doc replication to a users' db) It would(may) remove the need to grant/revoke access to a db, as the data would be filtered and synced based on some form of server-side logic.

So if I went down this path, I'd need some logic in the background to make it easy to assign a type of data (most likely a filter function) to a users' db.

I also avoided this example as we may in the future allow a user to belong to multiple entities... meaning I didn't want to deal with changing a users' db structure etc. But that's me getting too deep into our business needs, not hoodie's needs ;)

[gr2m]

I’ve replaced

store = new Store(name)

with

Store.open(name).then(function (store) {})

That way we have an async method to get a store instance in which we can prevent PouchDB from creating a database if it does not exist, so we can make sure that no database gets created without security

[jameswestnz]

@gr2m think that's a nicer way to separate the factory from the constructor/use of the store - think it was slightly ambiguous before on how to differentiate one store from another etc.

[gr2m]

If we have Store.grant() and Store.revoke() methods to grant / revoke access to / from a user, what should we call the method with which we can check if a user has the required access for an operation?

Maybe Store.has(), or in that case Store.hasAccess()? E.g.

Store.hasAccess(dbName, {read: 'id:123'})

[jameswestnz]

Hmmm good question. '.has' is consistent, but ambiguous in this context... you could use '.can' (I.e. 'Store.can({read:[]})'), but doesn't exactly resolve the ambiguity.

Maybe my idea of removing the word access has proven to not work now?

[gr2m]

Maybe my idea of removing the word access has proven to not work now?

Yeah feels like it right now. But this is a process, we’ll come up with something good at the end. I’ll move forward with grantAccess / revokeAccess / hasAccess for now and see how it feels

[jameswestnz]

Totally, sounds good to me!

[gr2m]

The current solution creates a meta database to keep track of all databases, as this is not something that PouchDB provides by default. There is https://github.com/nolanlawson/pouchdb-all-dbs which does the same but in more complicated way, as it most be compatible with the synchronous new PouchDB(name) while we could agree on having an asynchronous Store.open method, which can be async and can do all kind of checks before returning a store API instance.

In my current implementation I also store security in that database, which has the benefit that we can query what databases a user / a role has access to in future. If the backend is CouchDB, we can additionally keep CouchDB’s /_security in sync for double security

[gr2m]

I’ve updated the README with the API, it’s work in progress: https://github.com/hoodiehq/hoodie-store-server/tree/discussion/100/store-server-api/api

As I worked on it and made API docs and examples for every method, I’ve lost myself in a requirement we got a few times in a past to be able to set password protection for a store. What I ended up with is the idea of different "access types". By default the access type is "role", so you can grant read / write access to roles like this:

Store.grant('foo', {
  role: 'acme-inc', // array sets multiple roles
  access: 'read'
})

or give public read access like this

Store.grant('foo', {
  access: 'read'
})

In order to check if a user with the role acme-inc has access, you can do

Store.hasAccess('foo', {
  role: 'acme-inc', // array picks any role
  access: 'read'
})

That is the default behavior that Hoodie will provide, no more. The benefit is that it can be mirrored with CouchDB’s _security settings and custom design docs (with the exception of write only)

In future, we could add APIs where plugins can register custom access types, like 'password', so that a call like this would be handled by the plugin

Store.grant('foo', {
  type: 'password',
  access: 'read',
  role: 'acme-inc',
  password: 'secret'
})

I also ended up calling the methods Store.grant, Store.revoke and Store.hasAccess, even if access repeats itself in the options for the latter, I think it’s better than just Store.has and I didn’t come up with a better name yet.

[gr2m]

I implemented all but Store.replication API and the events, let me know what you think 💭

[jameswestnz]

I see that store.exists resolves with a boolean value stating if the store exists or not. I'm not sure on the answer, but should the promise not resolve if it does exist, and reject if the store doesn't exist? I suppose the user needs to know if there's another reason for rejection, as they may be trying to create if it doesn't exist... Just thought I'd ask?

[gr2m]

My thinking is that the question gets answered correctly with a yes or no, either way the method behaved as expected. It would only reject if something unexpected would go wrong, like a connection error. In that regard I’d say it’s different than e.g. hoodie.store.find(id) because it reads like I expect a document with id to exist, and if it doesn't it’s an error

[gr2m]

the PR is ready for review: https://github.com/hoodiehq/hoodie-store-server/pull/45

[gr2m]

we have https://github.com/hoodiehq/hoodie-store-server-api now :)