New Module System Discussion - Please contribute.

[MichMich]

Maybe it's a good idea to discuss the new Module System before we writeout the API.

@paviro, I noticed you've contributed. Thanks for this. I really appreciate your help! Just a few questions though:

The functions you proposed, are those method of a module class? I can't really wrap my head around how this would work, could you elaborate?

Also, I think the added regions are a bit unnecessary. In most cases you can just add content in one of the corners (the content will stack) or align right or left in the upper third, center or lower third. I think we should try to keep the layout simple.

Aslo, what do you guys think about using a system like vue.js? I'm not a big fan of jQuery, and prefer a mvvm model. But this might make it to difficult for most users to contribute.

EDIT 2016/3/23: First steps in the development are made. See: https://github.com/MichMich/MagicMirror/issues/80#issuecomment-200411278

[paviro]

To be honest with you I haven't thought all that much about how to implement it. I just added what I would need to port my modules to the new system.

• A way to start helper scripts written in Python and Node.js as background daemons (currently talking to the UI via socket.io - maybe we could provide a general module socket somehow?)
• A way to switch between multiple views (one for general users and one for people who are logged in via facial recognition). I would therefore love the possibility to be able to define more than one module layout in config.js and also to switch them via a module. Might also be useful if you want to have different views for lets say work related and private stuff. A module could again provide an interface to switch between them for example via a GPIO button.

Since my edits in the wiki are not that useful in their current state, I will remove them, they are here now anyway.

Yeah you are right. Stacking is probably enough. I guess I am not experienced enough in JavaScript to judge weather we should use jQuery or vue.js.

[bitte-ein-bit]

This is going to be some random thoughts.

I think if you switch to vue.js most user will not be able to contribute. I'm not sure about stability. I once used a rather new php framework (symfony 1.x) and later regretted it because their api change a LOT when going to 2.0. IMHO jQuery seems to have stabilized. I can't tell for vue.js or angular.js. Never used either. On the other hand not using spaghetti code is nice.

Personally I think it would be a good idea if a module could listen and trigger events. Some events could be defined (e.g. 'User changed', 'User logged out', 'Calendar event starting', 'GPIO x HIGH', ...) and some central instance could route them. This core should be very minimal and all that is needed to run the mirror. I think that everything else should be a module.

I agree with @paviro that some modules should be implemented without any use by themself like socket.io. This makes it necessary that modules can specify depends.

Regarding the helper scripts I think it would be nice to have e.g. a Makefile which will download and install the dependencies and setup an init.d or systemd script. Writing an daemon is rather advanced so IMHO the developer of that plugin should be able to find enough information to provide a Makefile. Other options would be some sort of Chef cookbook, Puppet Manifest, Bash Script, ... which could also setup the system itself. A Makefile would probably be the easiest.

I think the regions will work fine. But we should find a way, that a user can override all region positions without interfering with the main code. Not all mirrors are the same. So they might need some way to adjust margins, etc. For some modules it would be nice to render depending on the screen orientation but that is just eye candy.

[paviro]

I think I will write a Python CLI which could be used to install modules and also list all available modules with a description. I could implement the systemd or init.d scipt setup there :)

CLI

Something like:

• magic-mirror list (show all plugins with short discription)
• magic-mirror info module (show long description, dependencies etc. for module)
• magic-mirror install module (installs module and dependencies)
• magic-mirror remove (removes module and dependencies, if installed via magic-mirror install)
• magic-mirror enable/disable module (?)
• magic-mirror update (Update module list and versions)
• magic-mirror upgrade (install updates)

[MichMich]

Hi @paviro, before we start building command line tools, It might be a good idea to first finish of the basic concept behind the modules. How they work, and how dependencies are loaded. - Also, since most of it is javascript based, we might consider using existing CLI tools, like npm. This way, we can put our effort in the enhancement of the mirror UI.

[MichMich]

@bitte-ein-bit I agree with the idea of event triggering. Besides a module instance, there should be indeed a MagicMirror shared instance (singleton concept) which can receive messages of the module instance (in other words, a delegate for the module).

The MagicMirror share instance (the main instance) could have methods to load and unload modules. A broadcasting system for notifications between modules (so no direct module to module communication). A method to reload the HTML/dom for a specific module ... etc.

Regarding the overriding for region positions: this can all be done by css. The region is just a simple way to load a module in a specific location, but the module is free to adjust it's design using CSS. (As long as it doesn't interfere with other visuals.)

CSS based, it might be a good idea to define some standard elements modules can be used. If all modules use the same css elements, we could also implement a tempting system in later stage.

Regarding dependencies and installing libs. I think this should be taken case of if we have the base of the module system running. IMHO we could use existing systems like NPM.

Most importantly: thanks for your effort and contributions. I love the fact we are working on the future of the Magic Mirror.

[MichMich]

As a small side note: If we start using NPM, we could also consider building a small node web server to host the magic mirror. This way, users don't need to install Apache & PHP.

[gefangenimnetz]

I would love a JS only implementation. That would allow for easy porting to to Android or IOS using build tools like cordova (phonegap), etc. Basically that’s what I am currently doing.

[paviro]

@MichMich Your are right :)

[MichMich]

Not sure yet, but I might have some spare time end of this month to work on this version.

[sclausen]

@gefangenimnetz I'm currently working on this. It's based on angular2 and I'm still not sure, if I'll add a backend. Meteor (still with an angular2 frontend) is one idea and works really well, especially with realtime server side functionality like motion detection.

[paviro]

@MichMich @bitte-ein-bit any news on your side? Any way I can help you with the module system with my limited JS knowledge?

[sclausen]

@gefangenimnetz @MichMich I started a javascript-only clone sclausen/MagicalMirror

[bitte-ein-bit]

Unfortunately I'm packed with other stuff to do. Don't have the time to lock into a new backend right now. Sorry.

Am Do, 10. Mär 2016, um 14:46, schrieb Sebastian Clausen:

@gefangenimnetz @MichMich I started a javascript-only clone sclausen/MagicalMirror

---

Reply to this email directly or view it on GitHub: https://github.com/MichMich/MagicMirror/issues/80#issuecomment-194845798

[sclausen]

@bitte-ein-bit currently it has no backend and works frontend only with calendar, news, weather and time.

[MichMich]

I'll be probable be working on a new version coming weeks. I'll keep you guys posted.

[paviro]

Nice looking forward to it!

[MichMich]

Hi Guys,

Today, I FINALLY started working on the module system. Although I only have the very simple foundation up and running, I think I'm really on the right track.

A small overview so far:

config.js Used to define all the modules that need to be loaded, including the configuration for the specific module. Adding a module to this config will ensure the module will be loaded, including all nessecery .js and .css files. So other than downloading a module to the module folder and adding it to the config, no further actions are required to add it to the mirror.

    var config = {
        modules: [
            {
                module: 'clock',
                position: 'middle_center',
                config: {
                    foo: 'Bar'
                }
            },
            {
                module: 'compliments',
                position: 'middle_center',
                config: {
                    compliment: 'You look hot!'
                }
            },
            {
                module: 'compliments',
                position: 'top_right',
            }
        ]
    };

modules/compliments/compliments.js This is a very simple example of a module. Of course, this doesn't include any logic yes, but shows how simple a module is.

    Module.create({

        // Define the default config. This is optional.

        defaults: {
            compliment: "This is the default compliment"
        },

        // Define a start method. For example if you want to schedule timers,
        // or want to make connections to socket servers. Again: this is optional.

        start: function() {
            Log.info('Starting module: ' + this.name);
        },

        // Return an array of scripts that need to be loaded.
        // All scripts are loaded before the module is started.
        // Loading might be skipped if the file is already loaded.
        // The system contains a vendor folder for default libs like jQuery.
        // See comments below for more info.
        // This method is optional.

        getScripts: function() {
            return [
                'dummy.js', // Loaded (if available) form vendor folder, otherwise from module folder. Loaded only once.
                'http://server.com/js/file.js', // External file. Always loaded.
                this.file('foo.js'), // File from module folder. Multiple foo.js files might be loaded.
            ];
        },

        // Same system as getScripts().
        // This method is optional.

        getStyles: function() {
            return [
                'compliments.css'
            ];
        },

        // Returns the dom to be inserted in the mirror.
        // This method is optional.

        getDom: function() {
            var compliment = document.createTextNode(this.config.compliment);       
            var div = document.createElement("div");
            div.appendChild(compliment);

            return div; 
        }

    });

And thats all there is to write a module! Of course, I will continue to work on this. If I have any news I'll let you know. Hope to share a basic version next week.

All code so far is pure javascript. But of course, modules are free to use additional libs like jQuery.

Open for feedback.

[CFenner]

Looks great. I like the js only approach.

[MichMich]

And, as an example, a super simple module to show something on the mirror:

modules/helloworld/helloworld.js:

Module.create({
    defaults: {
        text: "Hello World!"
    },
    getDom: function() {
        return document.createTextNode(this.config.text);  
    }
});

This allows you to display a text on the mirror via the config:

var config = {
    modules: [
        {
            module: 'helloworld',
            position: 'middle_center',
            config: {
                text: 'This works perfect!'
            }
        },
    ]
};

[paviro]

Looks great! :) How are we going about socket connections? Should each module create their own (also if it uses helper scripts) or should we create a connection module which serves information from one socket to all the modules which need it and helper scripts have to talk to our socket?

[MichMich]

I've implemented a notification system to communicate between modules. This will also be used for the incoming socket messages, and probably also for the outgoing messages (although I don't see a use case for outgoing messages yet ...).

[paviro]

There is a use case. I would like to implement voice control and also use that for example to set my alarm clock which would need outgoing connections.

[MichMich]

But in that case you might want to communicate with a socket other then the mirror's backend. For now, the backend won't include a module system. (Will definitely be on the todo list.)

[MichMich]

I pushed a very simple first version of the v2-beta to a separate branch. Unfortunately I did not have the time yet to write a readme, but most files have sufficient comments. I probably won't be able to work on this version coming few days, but will definitely continue the project next week.

Feel free to give it a spin!

https://github.com/MichMich/MagicMirror/tree/v2-beta

[Miyasashi]

Can someone explain what the module system does I dont really get it.

[MichMich]

It will (eventually) allow everyone to built modules for the magic mirror without the need of modifying the core of the system. This way everyone can provide modules that can be installer on the mirror.

[Miyasashi]

Thanks for the fast response, funny coincidence i had problems with implementing my own stuff and I think this will help a lot.

[paviro]

@MichMich Do you think we could move the whole project to Electron once the module API is finished? Only the calendar part would have to be ported to JS right?

[CFenner]

It's not so easy to get a more complex module to run. This variable structure differs and Q seems not to work.

[MichMich]

@Paviro: Maybe in the future. It's not the current focus. @CFenner: I don't really understand what You mean. Could you elaborate?

[CFenner]

As the module was accessible via the global variable it was a bit easier to chain up the methods with Q promises. Now I need to bind the correct module object to each handler method to be able to call the next method. But I think I'm making progress..

[MichMich]

Hi @CFenner, I don't have any experience with 'Q promises'. Could you give an example what was possible in the old version, which isn't in the new version. Also note: the new version is in a very early state. So a lot of things might break in future versions, so please don't spent to much time on building stuff for the new version yet. :)

[CFenner]

Before I had this:

return Q.fcall(
    netatmo.request.token, netatmo.render.error
).then(
    netatmo.request.data, netatmo.render.error
).then(
    netatmo.render.all
).then(
    netatmo.update.wait
);

Now it looks like this:

var that = this;
return Q.fcall(
    this.request_token.bind(that), this.render_error.bind(that)
).then(
    this.request_data.bind(that), this.render_error.bind(that)
).then(
    this.render_all.bind(that)
).then(
    this.update_wait.bind(that)
);

But I think it is needed because Q is using the methods as a callback from the ajax request, but after all it's working now.

[MichMich]

@paviro the node_helper todo's are all gone. :)

What's more important, is that I integrated a socket system for the node helpers. Using it is pretty simple:

In client side module: To send a notification:

var notification = "JUST_A_STRING"
var payload = {anything: 'you want'}
this.sendSocketNotification(notification, payload);

To receive messages, make sure your client side module has the following method:

socketNotificationReceived: function(notification, payload) {
    console.log(notification);
    console.log(payload);
},

In node_helper.js: Create a socket object:

var MMSocket = require('../../js/socketclient.js');
var socket = new MMSocket('nameofthemodule');

To send a notification:

var notification = "JUST_A_STRING"
var payload = {anything: 'you want'}
socket.sendNotification(notification, payload);

To receive messages, register a callback:

socket.setNotificationCallback(function(notification, payload) {
    console.log(notification);
    console.log(payload);
});

Note 1: The connection will be established as soon as the client side sends it's first message. Note 2: All modules of the same type receive the same notifications. Note 3: For the client side modules, we're using subclassing to keep thing more simple. (No need to create a socket object for instance ...), I might work on a node_helper.js superclass as well in order to streamline the experience.

[paviro]

Nice! I will later update my module to use it :) The weather module looks really gorgeous! One question: Why are you creating socket namespaces for modules that don't even have a helper script that could use that namespace?

[MichMich]

Ok, slight change. I changed the node_helper setup to use a superclass. This way it works the same as the clients side modules, So to recap:

To send a notification:

var notification = "JUST_A_STRING"
var payload = {anything: 'you want'}
this.sendSocketNotification(notification, payload);

To receive messages, make sure your client side module or your node_helper has the following method:

socketNotificationReceived: function(notification, payload) {
    console.log(notification);
    console.log(payload);
},

[MichMich]

About that namespace: it doesn't create a lot of overhead, but never the less it will be fixed in the future. Currently all node helpers connect to the socket with their own connection. In the future they just use the overal socket connection. This will also fix the issue you mentioned.

This should not affect any of your code.

[paviro]

Alright :+1:

[paviro]

Do you think it would make sense to add some kind of notification UI module, so that modules can show alerts? I use sweetalert for my callmonitor but maybe it would make sense to integrate something like that into the core system? I am thinking of slide-in notifications (OS X style) but also UI overlays like I use in the monitor.

Also any chance we will be able to swap modules in the UI on runtime via an API? For example that you can have different sets of modules for different users or just different views (switchable with a button or something).

[MichMich]

Alerts: this is exactly what I was thinking. This would be a nice feature for the future. Swap module: yes this is on the to do list.

[CFenner]

An alerts system would be a nice idea so each module can display configuration or communication issues in a similar way.

Maybe build a module for it and use the socket system.

[MichMich]

There already is a inter-module notification system. So we just need to built this module.

To send a notification to all modules (client side, so no socket connection):

sendNotification: function(notification, payload);

To receive notifications:

notificationReceived: function(notification, payload, sender) {
        if (sender) {
            Log.log(this.name + ' received a module notification: ' + notification + ' from sender: ' + sender.name);
        } else {
            Log.log(this.name + ' received a system notification: ' + notification);
        }
    },

[MichMich]

@paviro: That namespace issue is now solved. https://github.com/MichMich/MagicMirror/commit/b243d25399e9538a0dc563e3fd7dd58c864937c1

[paviro]

Nice! Glad module swapping is on the todo list!

[MichMich]

What would be the best approach in your oppinion?

My idea is to allow the configuration of additional module classes in the config.

var config = {
    modules: [
        {
            module: 'calendar',
            position: 'top_left',
            classes: 'michmich'
        },
        {
            module: 'calendar',
            position: 'top_left',
            classes: 'paviro'
        }
    ]
}

This way you could do stuff like:

// using the default classes
MM.hideModulesWithClass('calendar',  speed, callbackFunction);

//using the custom classes
MM.hideModulesWithClass('michmich',  speed, callbackFunction);
MM.showModulesWithClass('paviro',  speed, callbackFunction);

Any ideas?

[paviro]

I like that way :)

[MichMich]

Cool, will try to implement it asap.

[paviro]

Nice, I will try to finish my facial recognition module later tonight. Maybe MM.hideModulesWithClass('michmich', speed, callbackFunction); should also invoke a suspend method in the module class, so a module can stop timers and resume them when it comes back into view later? Just so that we do not waste any CPU on the Pi refreshing stuff that isn't even visible.

[MichMich]

Great idea. Will keep it in mind.

Calendar module is almost finished. Will push it tomorrow.

[paviro]

Nice! Is there currently a standard way of reading a modules config from the node helper script? Other than by using require?