Lifecycle experiment with limited async steps

[jrburke]

I have been working on the core for a loader that is inspired by some previous sketches of the ES module-related loader, details at amodro-lifecycle. It is not a module system on its own, but a piece that can be wrapped by a module system.

While I do not expect it to be useful as-is for this repo, I wanted to share some aspects of its design in the hopes it helps the effort here.

Namely, it only has two async steps in the loading lifecyle, fetch and depend. The rest of the steps are functions that return their values synchronously.

This resolved some tensions I felt with previous ES-related loader sketches: supporting concepts like AMD loader plugins, but also providing synchronous methods on the AMD module meta for things like normalize and locate.

It also felt like it could bridge between node modules and ES modules well given the limited async steps that might be avoided in some node loading scenarios.

The other design note is the normalize step in the lifecycle, for things like loader plugin IDs that may not be paths, and module bundles that are likely more portable than an approach that uses URLs for the IDs in the bundles.

I put more details in the design forces section of the README.

The goal is to use that core in a new AMD module loader, currently being constructed and tested in the impl/amd directory in that repo. The tests are pulled from the tests used by the other AMD loaders I maintain. So far it is feeling pretty good, the code is operational: loader plugin support, config for ID and location resolution.

Still more work to do, but promising, so it seemed worth passing along the design parts.

[matthewp]

It also felt like it could bridge between node modules and ES modules well given the limited async steps that might be avoided in some node loading scenarios.

Do you mean so that Node can continue to support require() for a period after ES modules are introduced? If so I hadn't considered this. Great point.

I go back on forth on this issue. On the one hand having ever hook return a Promise makes everything so much more powerful. Consider a conditional syntax like: rafPolyfill?has(requestAnimationFrame). This loads the rafPolyfill module if and only if the has module returns false for the argument "requestAnimationFrame". Doing this in Normalize means you can return some empty module in the case where requestAnimationFrame is supported (let's pretend the registry has an empty module called @empty that we use like a noop. Doing this in Normalize means I want to import has in Normalize.

And there are so many other cases in every hook where you might want async behavior. On the other hand, I do worry about performance when every hook is returning a promise, and potentially there are multiple promises per hook... if importing a module results in 30 promises the whole thing will be slow.

[jrburke]

A module dependency ID of 'rafPolyfill?has(requestAnimationFrame)' will work in the lifecycle used by amodro-lifecycle, because of the depend hook is async, and would allow a loader that supports loader plugins to make sure rafPolyfill is loaded async before the system asks that plugin to sync normalize that full ID.

If the requestAnimationFrame test passes, and no loading is needed, then the loader plugin can set the module value for that ID to be the real rAF, so that further steps after normalize, which check to see if the module is already loaded, recognize it is already satisfied, and does not trigger the fetch step. Otherwise, the result of the normalize can be a polyfill module ID that proceeds to the async fetch step.

Fewer async steps helped the AMD module cases I have seen where loader plugins and modules want module meta objects with sync normalize and locate methods, just to simplify their work. For example, to generate paths to package-local resources for HTML structures.

While I was also a bit concerned about the speed of many promise-based steps, the primary motivation was simplifying the module meta uses cases, simplifying internal loader construction, and seeing a pathway to improve the node backcompat story.

[matthewp]

Sorry for the delayed response. That's an interesting solution, but I don't think it will work for all scenarios. Does your lifecycle loader have an equivalent to the whatwg loader loader.import? If you dynamically import with loader.import("rafPolyfill?has(requestAnimationFrame)") it will go straight to normalize, no? So I think you still need a top-level lifecycle method that can handle async needs.

[matthewp]

Ok, so useUnnormalized solves the problem in my above comment. Is a user-facing dynamic import API not defined yet?

[jrburke]

Right, the idea is that this is just a core engine, with different module systems implementing a public API on top of it, because the public API might have other concerns besides just the core operation.

For example, in the AMD-based implementation around this core, its require([]) needs to collect any queued up define() calls and add them to the lifecycle core's registry before calling useUnnormalized.

[matthewp]

Looked at your code and useUnnormalized won't solve this problem as it doesn't use the normalized name returned by depend(): https://github.com/amodrojs/amodro-lifecycle/blob/master/lifecycle.js#L216

[jrburke]

Let's track the use case you are thinking about over in the other repo. Feel free to outline it more over there. Loader plugins are working for me in the amd implementation on top of lifecycle (depend() is called in useUnnormalized, so plugins have a chance to intercept/delay before that normalize call is hit).

[jrburke]

Some follow up, I now have two wrappings over this lifecycle, with different goals and different normalization/code layout practices:

• amodrojs/amodro-base: AMD loader in the browser. Includes loading ES2015 modules, loader plugin support, './' and '../' relative ID normalization to a module ID space, loader config for locations and module ID aliasing.
• amodrojs/node-es-adapt: ES2015 module syntax in node, allowing ES modules to use traditional node modules as dependencies. Uses node's ID normalization and adds loader plugin support.

The AMD wrapper has a variant with a requirejs shim that passes more than 250 of requirejs tests, and that process helped improve the implementation of loader-lifecycle. I feel like the code has a good foundation now, so changes to the APIs can be tried out with some tests behind them.

[caridy]

After the latest refactor, we have a clear separation between sync processes (any interaction with the registry) and async processes (any interaction with a loader or an individual module status object). That should be enough to accomplish what you described here.

[jrburke]

Hopefully there will be a way for a module to synchronously resolve a relative ID for doing things like generating IDs in HTML or paths to package-local files. Similar to require.resolve() in Node, and require.toUrl() in some AMD loaders.

In the case of AMD and the experiment in the original post, the in-module resolution call was synchronous, but if a module body tried with a loader plugin resource ('plugin!./thing') and that plugin was not loaded yet, it would throw. The developer just needs to explicitly depend on the loader plugin module to get that case to work, and that seemed like an acceptable trade-off.