Feature: support host + mod workflow

[HipsterBrown]

As I've been reading the Moddable book again recently, I was reminded of the workflow of pushing a host program to a device via mcconfig before running "mod" programs via mcrun for quicker feedback. This was brought up as a method of supporting "hot reloading", but this is also useful as a documented pattern used by folks familiar with Moddable.

1. How does this fit with the current build and run commands in xs-dev?
2. Should this be an init template option?
3. What best practices for host+mod programs can we codify into this feature? Security through compartments? Module dependencies?
4. Should this pattern be used for "production" deployments or just the local development workflow?
5. Are there any caveats that could make this feature difficult to understand and debug?
6. Does this work for the simulator or wasm targets? Or is it just for on-device platforms?

One motivator for this feature is the xs-dev starting guide and ensuring folks have a good experience with those instructions. I am thinking of creating a ECMA-419 host for projects using the IO APIs.

This could be the first step towards supporting that "hot reload" ideal and/or over-the-air updates for devices on the same network.

/cc: @phoddie @andycarle @dashcraft

[dashcraft]

This is a topic i'm super excited about honestly. Having a robust host that has out of the box behaviors, similar to a bolts included framework, is something I've been trying to build for awhile, but haven't had time.

The mcrun stuff is still super unfamiliar for me as well, i need to really do a deep dive and use it far more often.

[phoddie]

Great topic. Thanks for getting this started. A little background might help.

We designed mods to allow JavaScript modules to added to a product's firmware . The idea is that the product firmware provides the foundation and a mod plugs into that to extend or modiy the firmware in some way. The most common way to think of that is the the product's firmware is an application and the mods are some kind of extension to the application. Another way to think of it is that the product firmware is the operating system and the mods are applications. We've seen mods used to effectively deliver content -- to change the visual theme of a product's UI or deliver a "tutorial of the month."

One thing that is common in all of these is that the host firmware is created by one person/organization and the mod is created by another. Here, especially with hot reloading, we are talking about something a bit different where the host and mod are the total firmware. The mod isn't an extension. It is just a faster way to install the modules currently under active development.

That brings up an important detail. The modules in a mod are not preloaded, of course. But most of the modules in the host are preloaded. By definition a JavaScript module is loaded exactly once. That means that any preloaded module in the host firmware cannot be overridden by a different verison of the module in a mod. This is good (because it prevents a mod from overriding parts of the host that aren't intended to be overridden) and bad (because we cannot override an arbitrary module). For the hot reloading scenario, it presents a challenge since the modules to be hot-reloaded either need to be absent from the host or at least not preloaded by the host.

How does this fit with the current build and run commands in xs-dev?

It could make sense to have build & run handle both the host and mods. They are very similar (mcrun and mcconfig share much of the same implementation in mcmanifest.js). That said, there isn't currently a 100% reliable signal that a given manifest is a host or a mod, so the user still has to be aware of the intent. But, perhaps that should be addressed.

Should this be an init template option?

That's easier: yes!

It would be good if the manifest indicated that it was a mod and which host(s) it is intended to work with.

What best practices for host+mod programs can we codify into this feature? Security through compartments? Module dependencies?

The right answer depends on the host's requirements. A general ECMA-419 host doesn't have security needs. It is really just a bundling of the native module dependencies. Compartments get in the way. A lightbulb host has real security requirements and Comaprtments are a great solution. If you consider the case of mod as application, a host might even launch each module of a mod into a separate Worker (and the Worker might use a Compartment to restrict he mod).

Should this pattern be used for "production" deployments or just the local development workflow?

All of the uses of mods, apart from hot reloading, have always assumed use in production. It does get a bit more complicated because xsbug isn't there to install the mod. But, there are patterns.

Are there any caveats that could make this feature difficult to understand and debug?

See everything above. ;)

Does this work for the simulator or wasm targets? Or is it just for on-device platforms?

It definitely works in the simulator. We use it there often. It works in wasm too, though it has been a while since I tried.