Work on the develop branch

[zsoltpardi]

I am opening this issue to discuss the latest changes at the develop branch.

tl;dr There are some changes at the develop branch. To review the changes please get the latest from the develop branch, (you need to do the usual npm install to get the npm modules) open a browser session at http://localhost:8888 and click on the "Things" menu item, this will be client "A" session. Open an other browser session at http://localhost:8888 and click on the "Things" menu item, this will be client "B" session. The property values such as "is_open" or "battery_value" and "power_consumption" for the door and switch respectively should appear on both browser windows. Click on the Invoke button next to the "unlock" action and the value of "is_open" should change in both browser sessions as well as the Invoke button next the the "lock" action should close the door and change the "is_open" property. There are the trace and debug logs in the browser development console, the Nodejs window and log files.

In more details: Finally, I could find some times and changed a few things with the aim of having a bit better structured and more maintainable application. I kept all original ideas and concept terms of models, events, property and action handling. I assume there will be many iterations until we find the correct design and implementation, so this is mainly a brain storming exercise.

The changes are for the latest commit

• There were an excellent community suggestion with regards to separating the UI from the back-in. I agree, such separation is really necessary and in first step I have created a "servers" directory which later can be easily moved into a separate repository. The http server related files and directories are moved from the root into the servers directory. The idea is (if I understood correctly) that there will be different servers and interfaces exposed to the clients, such as HTTP server (with web sockets), a REST server to provide mainly API based query/command interaction, or perhaps CoAP and MQTT. I am referring to clients who connect to their devices via the WoT framework, i.e. manage a door lock via the WoT framework. I imagine the majority of the client connections will be via HTTP server using web sockets, for instance the client open the door via the HTML5 application using the mobile enabled UI or via the REST application programming interface. There are sub directories in the "servers" directory, http, restapi, coap, etc. directories that refer to the implementation. I have moved the Express web server and related files into the "http" directory. The start and configuration of the servers is based on a config settings. Probably most implementations will be using a database to store the configuration settings. Currently we get it from the config.js file, the server settings is in the config.servers variable. The \servers\server_handler.js is responsible to manage the server sessions. The idea is to decouple the server management task from the main application file so we can write tests for this as well as probably it will be easier to maintain the application if not everything is embedded in the main application file (framework.js) like as it was before.
• Added a "dbs" directory to store database related implementations. I believe most of clients will be using database to store configuration settings and model definition of the things. Under the dbs directory will be stored the different database implementations such as postgresql, mongodb, mssql, nedb, etc. Currently the application pulls the model settings (which was before hard coded in the demo.js file) from a local db file (under the dbs/file directory), but the find_thing and things_list function returns asynchronous results just like the databases will be, so this hopefully gives an idea how the database based resources will be populated. The changes are about separation of concerns. I think the objects that are responsible for the business logic should not include configuration and settings resources, but in a local repository file or in a database repository.
• I tried to start breaking down the previous files that embed transport and application logic together into objects. I haven't got as far as implementing bubbafat's excellent suggestion about the factory design pattern, but I created a thing object as well as a thing handler. Previously everything was included in the framework.js and the wsd.js files, and I felt some decoupling is really necessary to have a cleaner design. The idea is that for example the transport layer such as the web socket handler should be responsible for the transport and the thing_handler object manages the things. So now the thing_handler under libs/thing/ is responsible for managing the things instead of the web socket file which was before performing basically all application logic. I believe we clearly need to change that as there will be other transport than web sockets and we should have some common objects which the transports can use to interact with the things. If this is the right direction then the next step could be to implement the factory design pattern. Please note the dependencies are commented out. I will wire them up if we think this is the way to go forward.
• In order to have a decoupled design I believe we should utilize the event emitter of NodeJs. I like to use the event emitter, it is efficient, quick and allows a decoupled event driven design. Therefore I started to use it in this project as well. For example there are no asynchronous callbacks nor synchronous returns between the thing and the transport (web socket) but the thing is emitting an event upon property changes and any device events. The web socket transport object listen on the event (at the end of the servers\http\wsd.js file) and then submits it to the UI. Using the event driven design allows that all type of protocols, ws, REST, etc. listen on the very same events in a decoupled manner and then submit them to their connected clients.
• Now there are database based settings and asynchronous calls to populate the UI. Previously the settings were hard coded in the wot.js file at the UI. To address high availability and scalability requirements I believe all real world, load balanced applications will be based on database settings or at least some server side configuration files, and therefore the committed design pulls the UI related resources such as list of things asynchronously from a DB file (currently it is a local file but I will wire up mongodb and postgresql schemas soon). The \servers\http\routes\api_routes.js pulls the model from a local DB file. I understand WoT is mainly a back-in server solution, but I think it is quite important to provide users with ideas how the WoT server can wired up to desktop and mobile UIs and how the WoT framework can be actually used - that's why I put some effort into the HTTP server design. Later it would be nice to roll out a proper MVC design using controllers and proper models for things, users, etc. Also, to have a different type of server we can add later node-restify which was another excellent suggestion from drasko.
• There are many changes in the UI and these I believe are pretty standard changes. There is a new main js file at \servers\http\public\js\wot\wotapp.js which introduces name space, base viewmodel, dialog window handling based on bootstrap, etc. I started to integrate a flexible Model-View-ViewModel pattern javascript framework. I am mainly familiar with knockout.js so that's what I am using here as well, but based on the current design it will be straight forward to include Angular or other frameworks. This early stage UI design hopefully gives some idea about what can be done via the UI. For example device management using some edit templates. I assume the properties, events, actions eventually will be configured via the UI.
• Please note, currently the remote proxy stuff is not implemented as I was not sure what is the best way to handle that. Probably it will be not very hard to add, most likely by extending the "thing" object (so the thing will be interacting with a remote WoT server instead of with a local device), but I am not sure at this stage.

Please let me know if these changes are sort of all right and then I will write tests and continue rolling out new files and device simulators. Alternatively, please let me know that in your opinion how to go forward with this WoT design.

[draggett]

The separation of concerns for the server and web page libraries sounds like a great idea. We need more discussion about not including the thing description in the method call to register a new thing. A common situation would be where a thing has a standard description and data model apart from its name. The API could then provide a link to the thing description in place of an explicit model. In other situations, it would be more convenient to provide the model explicitly in the application script. I therefore think both approaches are needed.

The server code already supported remote proxies, e.g. the register proxy example in demo.js. The web page library only allows you to register a proxy, but as far as I remember, doesn't yet cater for the situation where one or more of the properties for the thing you're proxying is for an other thing. In that situation, the web page library needs to register proxies for the things these properties link to. This is subject to the same complications as when a script on the server wants to register a proxy, and hence should be able to reuse the same design.

In principle, a web page could register a local thing, e.g. for sensors and actuators attached to the same device that is running the browser. These could be exposed to web of things servers via web sockets. I anticipate a new method to register a remote proxy on a server for a thing on this server or browser. This is needed to allow a browser/server behind a firewall/NAT boundary to expose a thing on a server accessible on the open Internet. It would require a new message to export the thing description and bind it to a proxy set up on the target server.

Even further out, a browser could expose a thing to other browsers via a WebRTC data connection. I don't yet have good use cases for this as yet, but want to note it as a possibility.

[zsoltpardi]

I understand the demo.js supports the registration of remote proxies and it is wired up with the wsd.js file, and I could be wrong, but I felt the current experimental design is not robust enough for a WoT Framework.

One of the issues is in my opinion that the "thing" object is embedded in the framework.js file (which is not very ideal for software maintenance and test but that's another issue) and the transport calls are hard coded in the thing object. When for example the thing is registered then it calls the WS server by calling a method on the wsd object. Similarly, when a property changes then the "set" of the property is invoked and then it calls the web socket server also using the wsd object. This design doesn't support other transports - it only supports the currently hard coded web socket server. If someone want to use REST API server or node-restify based HTTP server without web sockets then the line which calls the WS server will have to be replaced with the call to the other transport/module. I thought it is not the most optimal design. I thought that instead of hard coding the transport code in the thing object the thing object could emits events using the Nodejs event emitter and then all transports could listen on the events. Alternatively, if we don't want to use the Nodejs event emitter then there could be callbacks to the transport (multiple transports would be difficult with callbacks that's why I suggest the event emitter). In my opinion shouldn't be hard coded transport calls from the thing. A decoupled event or callback based design allows easier implementation of client portals/servers that communicate with WoT. I thing that's what bubbafat has suggested in his pull request that uses a factory pattern and event emitters, and that was my intention with separating the thing into an object: decoupling the thing implementation from the framework.js's main application logic and start using the event emitter to move away from the current interconnected design.

Other issue is in my opinion that different client portals/servers will be using the WoT Framework such as HTTP server with web socket, node-restify based HTTP server, REST API and CoAP servers and therefore I think there should be handlers and objects that are common to all client portals/servers. So both the Express based HTTP server, the node-restify based HTTP server and the REST API server could interact with very the same thing handler. Currently everything terms of things interaction is in the wsd.js file that means all other implementation (e.g. node-restify based HTTP server and the REST API server) will have to repeat the logic of the wsd.js file terms of things management. I thought a perhaps better design is to have a thing handler which manage the list of things and such thing handler will be common to bot Express HTTP server, node-restify based HTTP server and the REST API server. Also, as I said in my first post of this issue, I think it is quite important to not mix the roles: the transport should be responsible to forward the messages and not for application logic such as construct the messages which is the case currently in the wsd file. In my opinion the message should be common to all clients, and if that's the case then the message should not be constructed and produced by the transport - it should just send it.

If there is an agreement the current design is not optimal then the remote proxy registration and interaction should be re-engineered as well, and I haven't done that. Once we have a better understanding what will be the direction terms of design and implementation, and if we agree that the current design is not optimal then the remote proxy registration needs to be done as well.

I think the question is should we replace the current design with the changes I have proposed and later perhaps even try to move forward with more decoupling such as bubbafat's factory pattern or the framework.js file and the wsd.js is the proper base of WoT Framework, the current architecture of the software is fine and therefore no major redesign is required? I thought - for the reasons described in the above paragraphs - the current design should be replaced with a more robust, extendible, scalable, testable and maintainable design and that's what I was trying to check in with the latest commits.

If we say that as the demo.js file, the framework.js file and the wsd.js file currently perform tasks such as the remote proxy registration and that works and we are happy with that then the current implementation is the base of the WoT Framework, in fact that's the WoT Framework. In this case the changes that I proposed will not be merged, and that's all right, as I said it is a brain storming to find out what is the optimal design.

[draggett]

It is hard for me to properly appraise the changes you suggest, so it makes sense to apply them and see where this takes us, we can always back track if needed. With regards to supporting multiple protocols cleanly, we have proposals for how to represent server metadata as a means for servers to identify which protocol, data formats and encodings to use to communicate with another server.

The idea is to avoid an extended conversation between servers to establish how they should communicate. Instead, the server wishing to communicate with another server first needs to get the metadata for that server, and to the examine that metadata to identify the best approach. The precise way to get a server's metadata will depend upon the protocols it supports. For HTTP and CoAP, the idea is to use a ''well known'' path on the server. The means by which a server was discovered will have already provided information on at least one of the protocols that the server supports. I suggest that we defer detailed discussion on discovery and provisioning to another issue thread.

The existing server code uses setters and getters to intercept reads and writes on thing properties. We need to support multiple transport layer modules, and for the server to configure the setters and getters to use a specific transport layer module having previously determined which protocol to use for a given server. My idea is thus to use a separate NodeJS module for each protocol where this module deals with the messaging and the message encoding and decoding as appropriate to that protocol, and to the communications metadata for that server.

A starting point along this path would be to define separate modules for HTTP and WebSockets. The transport layer modules would expose a common API for property setters and getters, for notifying events, for invoking actions and delivering their results, for registering and unregistering things and proxies for things. In future additional models could be added for CoAP, MQTT and XMPP.

The existing Web Sockets code could be moved into the new module. For HTTP, we have some choices to consider. How does a server hosting a thing asynchronously send a message to another server hosting a proxy for that thing? One possibility is for the latter server to expose an HTTP end point for the first server to invoke. This could be passed when registering the proxy.

Another possibility is for the server hosting the proxy to poll the server hosting the thing in case that server has any messages ready to be sent to it. This provides design requirements for the messages to used with HTTP, e.g. a container for a sequence of messages.

Unfortunately, I am very busy with other work right now and don't have much time to contribute to the development work in the short term.

[zsoltpardi]

I kept the property setters and getters, so the read and writes on the property is intercepted in a similar manner than it is on the current design. The event and action handling logic is similar as well, I haven't changed that. My changes are mainly structural modifications to decouple the components.

I think an HTTP end point on the server that hosts the proxy is a good idea and will be relatively straight forward to implement it. I will do more test and try to integrate the dependency stuff as well as come up with a design for the remote proxy management.

Would you please let me know where can I find the proposals for how to represent server metadata that you have mentioned in your post?

[draggett]

Would you please let me know where can I find the proposals for how to represent server metadata that you have mentioned in your post?

See https://github.com/w3c/web-of-things-framework/issues/13

Of course the detail are not yet firmed up, but the thread covers many of the ideas

[zsoltpardi]

Thanks, that's great! In my latest develop branch commit I came up with something similar and it is in the config.js file:

servers: {
        web: {
            http_port: 8888,
            ws_port: 8080,
            base_uri: 'http://localhost:8888/wot/'
        },
        restapi: {
        },
        coap: {
        },
        mqtt: {
        }
    }

I will read that discussion to get some ideas and refine my current design.