App Organization

[formula1]

So this started off being about the removal of bower, but I think its a good time to understand what Silk does/doesn't, what an app does/doesn't and how we plan to process it

app.json -> package.json

Since npm is the package manager of everything, it makes it easier for us to just add parts onto the package.json. This is common in things like jscs, js-lint and other linters. For our purposes, we ought to allow both implementations but a few things should be made clear

• the dependencies are resolved and installed by npm and bower
• Gui aspects like icons, titles, index.html and etc are handled by us

  Usage of WebComponents: bower, browserify and the future

Currently we are using tablesaw and materialize in an important way inorder to provide applications to the client. However, as applications become more isomorphic the destinction between client and server starts to fade. In my opinion, this puts us in an awkward situation.

1. Web components are used in the wild and allows users to easily make beautiful websites
2. React may be a fad but the shadow dom will eventually become a staple in the future
3. Bower is a common place for these aspects and its difficult to find another place where css, html and javascript are all available in the same place

We probably shouldn't remove bower but we should take a heavy look in how how we want to install "building blocks" for applications and how we want to ensure these aspects are available

Files/Streams, "auto-open", choose an application

I put files and streams in the same boat because at some point I think we will want to implement a similar situation for audio, video and streaming text (ansii characters from the terminal as an example)

As it stands we are using regex which is likely not the best implementation. By basing it off something like accepts we have a further integrated ourselves in a web standard (http headers). And as those get updated, we get updated.

As for streams, this is also particularly important to me simply because streaming video/audio is something I use daily (twitch.tv) and I strongly believe its something that we can benefit from.

Routing: A node application, a seperate environment, a terminal interface or more?

Looking into the future, eventually rendering of an html will likely be done through a node application. Something to the effect of

var request_router = Silk.getRequestRouter();
var live_router = Silk.getLiveRouter();
var monitor = NodeOs.getMonitor();
var speakers = NodeOs.getStereo();
var www = NodeOs.getNetworkAdapter();
var window = new WebRender(monitor, speakers, www, web_router, live_router);
window.goto("./index.html");

Looking towards this future what eventually happens is instead of of client application sending text through websockets we will be sending them directly through node. Since this case we should also be thinking of our apps in this way as well. Apps ability to change through own router is imperitive in addition to our own ability to add and remove them as well.

[zodern]

Thanks for bringing this up. I have been thinking about this for a while.

I agree that we should switch to package.json. If an app uses bower or needs anything else done on install it can add it to the install script. Then the bower components would be in the app's folder and it will have the versions it needs. I don't think there is a reason to also support app.json.

Files There is three parts 1) Defining what files/streams an application accepts * We could use Navigator.registerContentHandler * And/or HTTP accept header 2) Choosing which program to open a file/stream * App defaults * Or something else? 3) Open the app to open the file/stream * Currently we open the window add add the path as a url query. But that won't work for streams.

[zodern]

• [x] Switch to package.json
• [x] Run NPM install
• [x] Save if npm install finished successfully
• [x] Save app id

Since the app's id is in the url, on each restart an app would get a new url which requires the window manager to be reloaded. We could save the id in package.json, or we could save it in the db in ~./silk. We should also save if npm install was run successfully.

• [ ] Fix all server api's
• [ ] Add and create tests for client api's
  • [ ] Set title
  • [ ] Open file
  • [ ] File picker
  • [ ] Folder picker
  • [ ] Minimze window
  • [ ] Close window
  • [ ] Open app
• [ ] Allow running app with node inspector or iron-node
• [ ] Change install location to ~./silk. Otherwise apps will be deleted when silk is updated by npm.
• [ ] Allow server to open apps
• [ ] Remove need for Silk Data to allow server api listen to work.
• [x] Only start apps when the window is first opened

[formula1]

Do you think its a good idea to create a seperate folder for the app manager? I was working with a a couple months ago on a similar project with heavy inspirations from this one and happy fun times.

Would you mind if I reorganized a bit?

[zodern]

Do you think its a good idea to create a separate folder for the app manager?

Are you wanting to move the App Manager app into core?

Would you mind if I reorganized a bit?

Feel free to.

I was working with a a couple months ago on a similar project with heavy inspirations from this one and happy fun times.

Thanks for the link. I have already gained some ideas by looking at it.

[formula1]

The abstraction of it yes. Perhaps its good to put it into a seperate repository (again :P) I'd like to plan this out a little more because when I was writing, I found myself going into a breaking mode. I don't think that is a good Idea and would prefer if we work togethor trying to come up with a solid solution. Something else I believe we should start thinking about is async/await instead of callbacks. Right now its probably not a good idea since it needs to be inside an async function, making things promised based would probably help us move towards that direction

Let me know any thoughts. Should we do this in another repo?

A couple of things that should be seperated out

• Messaging API - I started here, got deeper into it in js-gaming, but we can go deeper
• Data Synchronization API - This is something that I see you got a start on and we should probably go deeper with.
• Permission Layer - Again this should probably be ignored for now.

Important aspects about the Client and Server

• A single user may have the same page open multiple times
• A SharedWorker can likely be leveraged clientside to ensure synchronization within a single computer
• A single user may be logged in from 2 different computers
• The server acts as a shared context for all computers

What we know as a fact is the app manager can ServerSide

• Manager
  • List -Apps available for instalation, Apps installed, Apps with update available, Apps running/not-running
  • Install (application) - download, extract, compile? npm install?
  • Update (application) - npm update?
  • Uninstall (application) - npm uninstall?
  • enable (Application) - Allows Application to accept being started
  • disable (application) - Prevents Application from being started (usually hidden)
  • start (Application) - Starts an application
  • stop (application) - Stops an application
  • Handle Request (httprequest) - Sends it to the correct Application. We can send requests over child_process messeges so the Manager doesn't have to act as a proxy
• Application
  • Handles Request (httprequest) - provides it to a common http router or wraps it in our messaging API
  • Maintain Synchronized data
  • Save Sessions to Be Reused - Similar to how Browsers allow you to open windows that you had before
  • Do other node stuff - this may or may not be a bad thing since it opens permission problems

Client

• Manager
  • openWindow(Application, {constructor_options}) - Open a window for an application to run in
  • Some applications will allow multiple windows
    • Sometimes a user will force a second window
    • Some applictaions will want to maintain synchronization between each window
    • Some applications consider each window entirely different contexts
  • Some applications only use a single window
    • This may need to be enforced?
    • These windows may have "sub windows" handled by its own internal manager
  • focusWindow(ApplicationWindow)
  • This will make the window accept the user inputs above others
  • Close a window
  • Server may need to be notified of the closing of the window
  • Requesting confirmation on closing if there is "data that may be lost"
  • Minimize window
  • This happens when the window effectively isn't rendered but maintains context within javascript
  • This should be implemented always since in some cases the rendering is as problematic as the javascript. How the Window Manager handles displaying what is and isn't seen is the window managers problem
  • openContent(<File|Stream>) - If an application acts as a repository for Tor Links but doesn't handle the tor content themselves, they need to notify the window manager that the Tor Download has been requested and needs to be handled elsewhere. From there the user may choose an application
• Application
  • Rendering system (each application requires its own)
  • openContent (each application requires its own)

From here, how the windows are displayed (whether they are dragged and dropped or full screen) doesn't matter.

So here are some facts

// window manager is a class
var ApplicationManager = function(userInputs, config){
  this.windows = new Map(<window_id>, <window>);
  this.applications = new Map(<application_id>, <application>);
};

config = {
  enabledApplications:[],
  contentHandlers: [
    'text/html': 'text-editor'
  ]
};

// Both install and uninstall should take in an application_id (which likely should resolve to an npm name)
ApplicationManager.prototype.install = function(application_id){};
ApplicationManager.prototype.uninstall = function(application_id){
  var application = this.applictaions.get(application_id)
  return Promise.resolve().then(function(){
    if(application.enabled) return this.disable(application);
    return application;
  }).then(function(){
    this.doUninstall(application);
  });
};
ApplicationManager.prototype.enable = function(application_id){};
ApplicationManager.prototype.disable = function(application_id){
  var application = this.applictaions.get(application_id)
  return Promise.resolve().then(function(){
    if(application.running) return this.stop(application);
    return application;
  }).then(function(){
    application.enabled = false
  });
};
ApplicationManager.prototype.start = function(application_id){};
ApplicationManager.prototype.stop = function(application_id){
  var application = this.applictaions.get(application_id)
  return Promise.resolve().then(function(){
    if(application.windows.length){
      return Promise.all(application.windows.map(this.closeWindow.bind(this)));
    }
    return application;
  }).then(function(){
    application.fork.close();
  });
};
ApplicationManager.prototype.list = function(query){};

ApplicationManager. prototype.openWindow = function(application_id, constructor){
  var application = this.applications.get(application_id);
  if(!application) throw new Error('this application does not exist');
  if(!application.enabled) throw new Error('this application has not been enabled');
  (application.running ? Promise.resolve(application) : this.start(application) ).then(function(application){
    return Application.newWindow(this.createWindow(), constructor);
  });
};
ApplicationManager.prototype.focusWindow = function(window_id){};
ApplicationManager.prototype.closeWindow = function(window_id){}

// Application is a class
var Application = function(){};

Application.prototype.newWindow = function(window, constructor){}
Application.prototype.handleContent = function(content, metadata){};
Application.prototype.handleRequest = function(){
  // they will likely use a routing system here such as express
};
Application.prototype.synchronizeData.set = function(key, value){};
Application.prototype.synchronizedData.get = function(key){};
Application.prototype.synchronizedData.on = function(key, value){};

[zodern]

We should switch to ES2015. For async and await we will need to use babel or something similar since V8 doesn't support it yet. The code for async and await seems more complicated. I will try it in a project this weekend.

Messaging API - We have two implementations, one on the server and one in the client api. We should switch to the client api one. How do you want to go deeper?

I think we should also move the client API to it's own repository and publish it on bower and npm.

Data Synchronization API: There is three places that could be synchronized: 1) Between Silk's API's and an app. This way an app can have up to date information on the app list, remote status, etc without having to poll the information. This doesn't rarely works, and I overcomplicated it.

2) Between the client and server. I think this works well.

3) Between clients. Without this, remote access isn't a traditional remote access. Did you do any work on this in Silk Skeleton?

I agree we shouldn't worry about permissions yet.

I agree with almost everything you wrote about the application manager. Would the window manager use the ApplicationManager? That makes a lot of sense since the window manager is designed to be easily replaced.

We want to eventually allow apps like http://benkaiser.github.io/stretto/ and https://www.nylas.com/n1, and maybe even github atom to work with minimal modification. I think it would be easier with electron than nw.js. I will do some experimenting this weekend. A first step could be allowing silk apps to run outside of silk and in electron.

[zodern]

Electron has an option to disable nodeIntegration in windows:

var mainWindow = new BrowserWindow({
  webPreferences: {
    nodeIntegration: false
  }
});

We would only be able to support apps that disable the node integration. We might also need to have our own module for communication between the window and the main process to support running in silk, in electron, and remotely in silk.

[zodern]

I am going to work on this over the next week.

This is my plans for now:

1) Finish external apps. This allows running apps that are outside of silk's app folder which is useful for debugging. I need to add the required server apis, add a folder picker, and allow adding them in the app manager.

2) Start apps when they are opened in the window manager and close the fork when their last window is closed.

3) Work on getting stretto to run in Silk. It disables node-integration so it should be possible. We will probably need to some of polyfill electron's api's. What we end up with will likely replace our current app framework.

@formula1 Do you have time to help with this?

[zodern]

External apps is finished. They have a green underline on hover in the app menu.

[formula1]

Yes and no, at the moment there was another project that grabbed my attention and/or could bring more developers into NodeOS core

https://github.com/guisouza/tEmbO - tEmbO is a tiny replacement for react but it has far bigger implications. For example:

• We mentioned Terminal Commands being similar to Clientside forms
  • React Enables this in a sense (See renderers section)
  • Create a React Component that acts like a form
  • In browser, it generates html
  • in server, it generates a commander or command router
• It can possibly double as generating the same interface for native/terminal/html as well

The Communication API is absolutely paramount to your work (as well as a number of projects I work on). I absolutely agree we should probably more the communication api into a new repo. I got started on this earlier But I hadn't worked on it for a while because of how crazy/tedious it can get. I'll push up what I have though either today or tomarrow.

[zodern]

That is an interesting project. I am excited for what you create! It could also be used to automate creating UGUI apps.

Let me know when you push the communication api. We have it implemented in 3 or 4 places in Silk and all have some problems. It would be nice to have one library that works well.

[formula1]

https://github.com/formula1/JS-Messaging

Not quite ready though. There are alot of things I have yet to implement. But I'm trying to start things fresh.

[zodern]

I have been thinking about this a lot recently.

Our priorities for the app manager should be (not necessarily in this order): 1) Easy for developers to use when developing an app 2) Simple, compared to the fork framework 3) Reduce/remove any walled gardens

1) Right now it is a lot more complicated than working on an electron app. It could be simplified to:

• silk app add (only needs to be run once)
• Start the app with silk app start --dev
• It opens a new window for the app. Node inspector is available behind a toggle button.

To refresh the app, when running with --dev we can use nodemon or our own watcher to restart the app on a change. The developer could have webpack-dev-server to refresh only the client, and for projects that don't, we can have a refresh button in the task bar.

2) The new app framework is much simpler. Another change we can do is use yarn to install and update apps. It has an option to set the install directory instead of using node_modules or bower_components.

3) With the new app framework, it is much more open at the expense of doing less to help developers. Eventually, the only requirement could be a js file for electron's main process. Using node-x11, we could open any app on a computer with x11.

The problem with that is we lose our "free" remote access. Once option is only allow remote access for apps with nodeIntegration set to false, and maybe in the future have a hybrid approach with part of the screen using a remote desktop protocol.