The future of modules

[nilclass]

TL;DR? This message is about changing the way modules work in remoteStorage.js. To skip the intro, search for "A) " in this message.

To recap, this is currently how modules work:

• Modules are defined through remoteStorage.defineModule. They consist of a name and a builder function.
• The name corresponds to a directory node directly below the storage root, as well as to an entry in the "scope" used to claim access.
• The builder function receives two BaseClient instances, corresponding to the private and public root of the module respectively.
• Modules return (amongst other things) an object called "exports", which is the public interface of the module, as seen by apps.
• Once a module is defined, it's "exports" object can be accessed via the remoteStorage object as remoteStorage[moduleName].

To illustrate this, here is what a module may look like:

remoteStorage.defineModule('bottles', function(privateClient, publicClient) {
  // declare which paths to use
  privateClient.use('');
  publicClient.use('');

  function generateId() {
    // (code ommitted)
  }

  return {
    exports: {
      add: function(content, size) {
        var id = generateId();
        return privateClient.storeObject('bottle', id, {
          content: content,
          size: size,
          id: id
        });
      },

      remove: privateClient.remove,

      getPublished: function() {
        return publicClient.getObject('publishedItems');
      },

      updatePublished: function(object) {
        publicClient.storeObject('published-items', 'publishedItems', object);
      },

      publish: function(id) {
        var published = this.getPublished();
        publicClient.storeObject('bottle', id, privateClient.getObject(id));
        published[id] = new Date().getTime();
        this.updatePublished(id);
      },

      unpublish: function(id) {
        var published = this.getPublished();
        publicClient.remove(id);
        delete published[id];
        this.updatePublished(id);
      }
    }
  }
});

This leads to a number of restrictions in writing apps:

1. In order to write an app, you need to write a module, unless you only depend on core-modules. This is even true for apps in which the module doesn't do anything that the BaseClient wouldn't be able to do on it's own.
2. The module determines the only public API that the app can use to access storage. This means that even very basic methods, such as getting / storing / deleting / listing objects or files, need to be implemented or explicitly delegated to the baseClient(s) by the module. Thus the structure and naming of the public API varies greatly between individual modules.
3. The separation of access to the "private" and "public" area through individual BaseClients makes it the module's responsibility to connect the two. Thus it becomes very hard to establish conventions for publishing and sharing, as those conventions cannot be implemented of the remoteStorage.js core code.
4. There is no defined way to extend a module. The public remoteStorage[moduleName] object can be extended like any other JavaScript object, but the syntax for this varies greatly from the regular module definition through defineModule().
5. Modules have no mechanism to access another person's storage.

In order to overcome these restrictions and reduce the amount of boilerplate code required to start with an app, I would like to make some changes. Most of the following suggestions are the result of various discussions in the channel (#remotestorage on freenode) and at other places (such as issue #100). But before starting to actually implement those changes, I'd like to give everyone a chance to add to, criticize or approve with any of this. So go ahead :)

A) Drop the public client, enhance the BaseClient

As can be seen in the example above, the code to publish objects is very generic. In fact, quite similar code is already part of the "root" module. As publishing and sharing are the most common usecases for public data, it makes sense to have them as part of the BaseClient.

I propose to add the following methods to BaseClient:

• publishObject(path) - adds the given path with the current timestamp to the "publishedItems" object and copies the object found at "path" to public/{moduleName}/{path}
• shareObject(path, secret) - copies the object found at given path to public/{moduleName}/shared/{secret} and returns the absolute path to it as a String.
• getPublicClient(path) - retrieve a copy of the BaseClient that works on the "public" area. This would be for backwards compatibility with the current publicClient, as well as for custom operations done in the public area.

  B) Expose the BaseClient to the app

To make trivial access to remotestorage as simple as possible, it should be possible to gain access to a BaseClient instance without having to define a module. This would work through a method called remoteStorage.getClient, which receives a moduleName:

var client = remoteStorage.getClient('bottles');
client.getObject('...');
client.publishObject('...');

C) Make modules extensions of BaseClients instead of independent objects

Once (A) is implemented, our module boilerplate would have changed to:

defineModule('bottles', function(client) {
  return {
    exports: {}
  }
});

Then once (B) is implemented, all the generic methods from the module could disappear. All that remains is "add". Now the situation is a bit weird, as some methods (such as remove, publishObject, ...) have to be called on a client, while "add" still has to be called on the module.

So instead of making assigning the "exports" object to remoteStorage[moduleName], the module could serve as an extension for the BaseClient. The BaseClient in turn would be cached, so getClient('bottles') will return that modified BaseClient instead of a fresh one.

Thus once a module 'bottles' has been defined, the following lines become equivalent:

remoteStorage.bottles;
remoteStorage.getClient('bottles');

Note that in this case, the module would better name the "add" method "addBottle", to clearly distinguish it from adding other objects.

This would also enable modules to be extended, so a module that has been defined in the core, can be extended from an app by calling defineModule() again, without having to copy the entire module to the app.

D) Apply modules to ForeignClient instances as well

Another new thing in the upcoming 0.7 release is the ForeignClient. It is basically a read-only BaseClient that is associated with another person's storage. Once (C) has been implemented, the module's "exports" object can additionally extend any created "ForeignClient" for the same scope / moduleName. To provide an example, consider a method "totalSize" for the example above. It would iterate over all "bottles" and add their "size" attributes. If the same method is applied to the ForeignClient, it can be used to perform the same operation on another persons published "bottles".

Ok, that's all for now. If I forgot anything, please add.

-- Niklas

[xMartin]

Great writeup! I like the direction you're heading to.

Some thoughts from the top of my head:

1. I don't like that modules/clients can be accessed as properties of remoteStorage. getClient "feels" much better.
2. While we're at it: module and client being almost the same is confusing. Any idea to clean up the terms?
3. I like the idea of extending for modules. I'd like to see an extend function to help with that. Just pass it an object and mix it in. Addionally would be great to have the posibility to call some kind of super method.
4. This proposal doesn't contain the possibility of not having to define a module at all, right?

[nilclass]

I don't like that modules/clients can be accessed as properties of remoteStorage. getClient "feels" much better.

I'm not sure about that either. I'll keep it at least for a while though, for backwards compatibility. And actually I think it's kind of nice. It gives a straightforward way to access core modules, without having a method call in between:

remoteStorage.contacts.getListing()
// vs
remoteStorage.getClient('contacts').getListing()

While we're at it: module and client being almost the same is confusing. Any idea to clean up the terms?

No, not really. I have started using "module", "scope" and "category" almost interchangeably, where "module" may refer to either the code, or the object returned by the code, or a directory beneath the storage root, while "scope" and "category" only ever refer to the latter :) That suddenly "client" comes into the picture as well is probably even more confusing. Considering (C) from above, the connection is that a "module" in rs.js terms is an extension applied to a BaseClient, but only to a BaseClient that is bound to the "scope" (or "moduleName" as it's called in the code), that the module is intended for.

I like the idea of extending for modules. I'd like to see an extend function to help with that. Just pass it an object and > mix it in. Addionally would be great to have the posibility to call some kind of super method.

Yes, I've thought about the same thing (super method), but I'm not sure it's needed. Also "extend" is not strictly needed, as defineModule will behave quite differently due to (C). When you define a module, you have access to the client as it looks previously, so you can call it's methods:

defineModule('something', function(client) {
  return {
    "exports": {
      getObject: function() {
        doFancyThing();
        client.getObject.apply(this, arguments);
      }
    }
  }
});

After that code has been executed, remoteStorage.getClient('something') will have the new "getObject" method.

This proposal doesn't contain the possibility of not having to define a module at all, right?

It does. There is no need to call defineModule. Though we should encourage to do so, as extending remoteStorage with domain specific functionality can be very useful and simplifies re-usability of both code & data.

[silverbucket]

+1 sounds great to me.

On Fri, Oct 26, 2012 at 7:44 PM, Niklas Cathor notifications@github.com wrote:

TL;DR? This message is about changing the way modules work in remoteStorage.js. To skip the intro, search for "A) " in this message.

To recap, this is currently how modules work:

Modules are defined through remoteStorage.defineModule. They consist of a name and a builder function. The name corresponds to a directory node directly below the storage root, as well as to an entry in the "scope" used to claim access. The builder function receives two BaseClient instances, corresponding to the private and public root of the module respectively. Modules return (amongst other things) an object called "exports", which is the public interface of the module, as seen by apps. Once a module is defined, it's "exports" object can be accessed via the remoteStorage object as remoteStorage[moduleName].

To illustrate this, here is what a module may look like:

remoteStorage.defineModule('bottles', function(privateClient, publicClient) { // declare which paths to use privateClient.use(''); publicClient.use('');

function generateId() { // (code ommitted) }

return { exports: { add: function(content, size) { var id = generateId(); return privateClient.storeObject('bottle', id, { content: content, size: size, id: id }); },

  remove: privateClient.remove,

  getPublished: function() {
    return publicClient.getObject('publishedItems');
  },

  updatePublished: function(object) {
    publicClient.storeObject('published-items', 'publishedItems', object);
  },

  publish: function(id) {
    var published = this.getPublished();
    publicClient.storeObject('bottle', id, privateClient.getObject(id));
    published[id] = new Date().getTime();
    this.updatePublished(id);
  },

  unpublish: function(id) {
    var published = this.getPublished();
    publicClient.remove(id);
    delete published[id];
    this.updatePublished(id);
  }
}

} });

This leads to a number of restrictions in writing apps:

In order to write an app, you need to write a module, unless you only depend on core-modules. This is even true for apps in which the module doesn't do anything that the BaseClient wouldn't be able to do on it's own. The module determines the only public API that the app can use to access storage. This means that even very basic methods, such as getting / storing / deleting / listing objects or files, need to be implemented or explicitly delegated to the baseClient(s) by the module. Thus the structure and naming of the public API varies greatly between individual modules. The separation of access to the "private" and "public" area through individual BaseClients makes it the module's responsibility to connect the two. Thus it becomes very hard to establish conventions for publishing and sharing, as those conventions cannot be implemented of the remoteStorage.js core code. There is no defined way to extend a module. The public remoteStorage[moduleName] object can be extended like any other JavaScript object, but the syntax for this varies greatly from the regular module definition through defineModule(). Modules have no mechanism to access another person's storage.

In order to overcome these restrictions and reduce the amount of boilerplate code required to start with an app, I would like to make some changes. Most of the following suggestions are the result of various discussions in the channel (#remotestorage on freenode) and at other places (such as issue #100). But before starting to actually implement those changes, I'd like to give everyone a chance to add to, criticize or approve with any of this. So go ahead :)

A) Drop the public client, enhance the BaseClient

As can be seen in the example above, the code to publish objects is very generic. In fact, quite similar code is already part of the "root" module. As publishing and sharing are the most common usecases for public data, it makes sense to have them as part of the BaseClient.

I propose to add the following methods to BaseClient:

publishObject(path) - adds the given path with the current timestamp to the "publishedItems" object and copies the object found at "path" to public/{moduleName}/{path} shareObject(path, secret) - copies the object found at given path to public/{moduleName}/shared/{secret} and returns the absolute path to it as a String. getPublicClient(path) - retrieve a copy of the BaseClient that works on the "public" area. This would be for backwards compatibility with the current publicClient, as well as for custom operations done in the public area.

B) Expose the BaseClient to the app

To make trivial access to remotestorage as simple as possible, it should be possible to gain access to a BaseClient instance without having to define a module. This would work through a method called remoteStorage.getClient, which receives a moduleName:

var client = remoteStorage.getClient('bottles'); client.getObject('...'); client.publishObject('...');

C) Make modules extensions of BaseClients instead of independent objects

Once (A) is implemented, our module boilerplate would have changed to:

defineModule('bottles', function(client) { return { exports: {} } });

Then once (B) is implemented, all the generic methods from the module could disappear. All that remains is "add". Now the situation is a bit weird, as some methods (such as remove, publishObject, ...) have to be called on a client, while "add" still has to be called on the module.

So instead of making assigning the "exports" object to remoteStorage[moduleName], the module could serve as an extension for the BaseClient. The BaseClient in turn would be cached, so getClient('bottles') will return that modified BaseClient instead of a fresh one.

Thus once a module 'bottles' has been defined, the following lines become equivalent:

remoteStorage.bottles; remoteStorage.getClient('bottles');

Note that in this case, the module would better name the "add" method "addBottle", to clearly distinguish it from adding other objects.

This would also enable modules to be extended, so a module that has been defined in the core, can be extended from an app by calling defineModule() again, without having to copy the entire module to the app.

D) Apply modules to ForeignClient instances as well

Another new thing in the upcoming 0.7 release is the ForeignClient. It is basically a read-only BaseClient that is associated with another person's storage. Once (C) has been implemented, the module's "exports" object can additionally extend any created "ForeignClient" for the same scope / moduleName. To provide an example, consider a method "totalSize" for the example above. It would iterate over all "bottles" and add their "size" attributes. If the same method is applied to the ForeignClient, it can be used to perform the same operation on another persons published "bottles".

Ok, that's all for now. If I forgot anything, please add.

-- Niklas

— Reply to this email directly or view it on GitHub.

[michielbdejong]

i put in that barrier on purpose, to encourage people to write reusable modules. unix applications may also want direct access to the kernel interrupts without having to go through kernel modules.

everything outside the code that goes into the defineModule() calls should be considered "userland" code and should not have direct access to any data.

i understand that this barrier creates extra work for the app developer, but to me that's not a reason to abolish modules altogether.

[michielbdejong]

i'm not against merging public client and private client into one base client, and adding extra functionality to it.

i am against foreignClient in general, but that's a separate issue. :)

if someone wants to quickly hack spaghetti code together, then they always have the option of doing:

remoteStorage.defineClient('quickhack', function(priv, pub) { return { exports: priv }; });

and then put all their data-related and presentation-related code together in one place, outside their 'quickhack' module.

but the only way to access anything under /contacts/ or /public/contacts/ should imho be by including a module called 'contacts'.

i think most of your points 1. - 5. can be resolved without starting to allow direct module-less access to the client(s)?

Except maybe point 4., but that relates to the bigger topic of module versioning i think (which i think is definitely a currently unsolved problem still, and we'll run into that also, in the near future)

[raucao]

+1 (with the exception that it shouldn't be so much about including a module, as about including a core-recommended schema, plus your own potential extension schema for custom attributes)

[michielbdejong]

yeah, so let's keep it the way it is, not exposing the baseClient directly to apps; as we said, people should just get used to the fact that "user data modules" are a core feature of how you use remoteStorage in an app.

please reopen if you want to bring this issue back up for discussion.