Loader Hooks and State

[EisenbergEffect]

Loader Hooks and State

Currently, we have the following possible states:

• fetch
• translate
• instantiate
• link
• ready

We also have the following hooks:

• resolve
• fetch
• translate
• instantiate

This is certainly not my area of expertise, but there's a bit of a mismatch here that bothers me. Of particular concern to me is that the entire process of loading a resource is not accessible via hooks. The hooks seem to cut off at the instantiate phase, whereas I believe they should follow through to the ready phase. There's also a bit of an inconsistency at the beginning, with a missing resolve state.

I believe there was recently some talk around removing the locate hook, so it seems there's still issues to be discussed around this topic and I think it's pretty critical that we get this right.

Note: I realize stuff has only recently been moved over here and there's still lots of work to do. I just wanted to jump in early on an area that's particularly important to me and my community.

My personal interest is in having a ready hook which would allow me to receive the fully executed module instance along with its metadata. There are a number of important scenarios that this would open up:

• Insight - Track loads, load times, runtime usage patterns, tracing, etc.
• Instrumentation and AOP - Dynamically instrument or transform code based on the runtime context.
• Metaprogramming - Use metadata defined in instances to generate code on the fly.
• Registries - Application-Specific optimized catalogues of loaded modules for use by frameworks/libraries.

I'm sure there are other uses as well. One technique I've used repeatedly across multiple module loaders for a number of years now is to tag exports with their module id of origin, thus enabling convention-based programming, ala Rails. It's not possible to do this reliably 100% of the time without a ready hook of some sort.

I've also attempted to proxy module.execute but this technique doesn't work for real ES6 modules. So, it's not adequate...not to mention that it seems like a hack. Issue #9 is related to this. @matthewp shows a technique for using instantiate. He's got a different scenario, but hits some similar types of inconsistencies.

So, I'd love to see additional refinement in this area. And, as I've mentioned, I'm particularly interested in a ready hook. I'm also pretty sure I've heard this requested by others in different forums including @jrburke and @guybedford (Though I could be wrong, because it was quite some time ago.)

Looking forward to working through this and very excited to see the renewed effort on the loader.

[guybedford]

A post-instantiate hook could certainly be useful. Note that the module object would have the exports, but wouldn't be executed yet in ES6 since execution is late, making this not quite as simple as expected. It's only dynamic modules (CommonJS etc) that are executed during the link stage. I'm not sure of the technicalities here further myself, so perhaps others can clarify.

[EisenbergEffect]

I'd like to learn a bit more about how that works as well. For my own uses, I would need a consistent way to get the executed module, regardless of whether it's ES6 or dynamic.

[guybedford]

So the way it is written currently - the Module object is created as a reflective module record - http://whatwg.github.io/loader/#reflective-module-record. It exists with its bindings at the time of linking (which is after instantiate). The requestReady method then actually calls execution on the module (in turn ensuring execution of its dependencies, this is in the ES6 spec, not this one) - http://whatwg.github.io/loader/#request-ready.

The issue is that executing Module A, can result in executing Module B. Because the code that executes Module B here is not part of the pipeline, I'm not sure how it would be possible to spec a post-execution hook, as it is actually the ES6 spec driving this process.

I can just try and explain the details (badly) though - others will be able to comment on the actual viability.

[guybedford]

An alternative might be to ammend all methods that return module objects to ensure post-operations are applied.

[guybedford]

Note that the module instance exists as a unique entity before execution, just with no exports. Would that be useful?

[EisenbergEffect]

Unfortunately no. I actually need the exports for my own use cases. I'm thinking of a hook that would be called after a module's exports are ready.

[guybedford]

Evaluation is defined in the ES6 specification, not this one. See https://people.mozilla.org/~jorendorff/es6-draft.html#sec-moduleevaluation. There are no hooks available for evaluation that I can see.

[guybedford]

The best bet may be through module reflection added to the source itself through translate.

[EisenbergEffect]

So, if I have module A that imports module B and then A is imported...Isn't there some point where the loader knows that B is ready and now A can be executed? Isn't there some point when it knows that A is finished executing and can return it to the caller?

[guybedford]

No - this execution is all in the ES6 spec. As far as I'm aware the loader has a module object for A, then asks to make sure A is executed, then returns the module object. The module object exists before execution, there's not sense of it being undefined then defined, but rather simply moving from an unexecuted to executed state.

[guybedford]

As I say though - I have limited knowledge of the spec interactions and am just inferring, @dherman, @caridy or @wycats can most likely comment more clearly on the true viability.

[EisenbergEffect]

Hmm. Well that's disappointing to me. Seems like there isn't as nice an integration with the loader as I would have hoped.

@guybedford Can you elaborate more on the module reflection idea? For my own use cases, I basically have an export and I need to figure out what module it was originally exported from (not re-exported).

[guybedford]

The loader registry may well be iterable in which case you could probably do a sort of reverse lookup. It may be worth creating an issue to track this.

[matthewp]

https://github.com/whatwg/loader/issues/17

[EisenbergEffect]

I could iterate every module in the registry...and iterate every export to find a match, then tag the exports with the matched module id....seems kind of expensive, but maybe it would work since I don't need to do it except in certain scenarios...

[EisenbergEffect]

So, how does that registry get populated? It seems....that it must populate items after the module is executed, yes?

[guybedford]

This implementation actually creates a registry entry the moment the module is resolved, along with its state. The public API that is exposed here seems to be described in the loader lookup function - http://whatwg.github.io/loader/#reflect-loader-lookup.

That is the object -

{
    state: 'fetch', // etc
    fetch: fetchPromise,
    translate: translatePromise,
    instantiate: instantiatePromise,
    module: module,
    error: null
}

The module object itself is undefined until instantiate is complete. Then it is defined but not executed, before finally being executed.

Given a module object, I'm not sure there will be a public API though for determining if it is executed or not. As mentioned - the object exists from instantiate regardless of execution state.

[EisenbergEffect]

I see. What that means is that it will be possible to query the registry, find the module I'm looking for, but not be able to determine that it is in fact the module I'm looking for because it hasn't been executed yet.

The only think I can think to do is monkey patch the registry itself so I can determine when things are added and then monkey patch the module property so I can know when it's set. A custom setter function could do what I need. Sounds questionable though....

[guybedford]

You would completely be able to determine if it was the right module - the module object can be used in comparisons regardless of execution state.

[EisenbergEffect]

Actually, I think it's ok. If I've got the export in user code, then the module property must be set on the registry entry, correct? So, I should be able to enumerate the exports of each defined module to find the correct entry.

[guybedford]

Yes that's my impression from the current information.

[caridy]

guys, I don't fully understand the use-case here. Can someone articulate what you're trying to achieve, to see if we have a solution for it?

[EisenbergEffect]

@caridy My personal need is to be able to get from an exported value back to the module id for the module it was originally exported from. In code it means being able to do something like this:

foo.js

export class Foo { ... }

bar.js

import {Foo} from './foo';
import {getOrigin} from 'metadata';

let x = new Foo();
let moduleId = getOrigin(x);

assert(moduleId === 'foo');

The question is, can I implement getOrigin()?

This is my own personal need as it's how we do convention-based rendering in Aurelia. If you have an instance of foo and you want to render that in the browser, then we can determine its module id and then use that to map to an html file with contains the template. Again, this is a very specific example of what we have done in the past with require.js that we'd like to be able to do with ES6 modules. We were able to do this because require.js has a hook that is called when a module is ready. We used that to tag module exports with their normalized module id.

It seems as if we could use the module registry to accomplish the same thing. If something isn't tagged with it's id, we could iterate the registry looking for a match, tag it for future lookup optimization, then return it. So, I think we are good in terms of there being "a way" to accomplish this.

However, I have a more general concern over the black boxing of certain aspects of ES6 modules and the loader. I'd really like to see a clean, fully accessible resource loading pipeline. To me it seems odd to have a loader with hooks...but no hook that is called when a module is actually loaded. The states and hooks seem a bit lopsided to me. This isn't a deal killer for me, especially if the registry exists, but it would make things nicer and open up some other possibilities. As I've mentioned, you could use this to do metaprogramming and AOP.

Does that help clarify things a bit more?

[caridy]

@EisenbergEffect I understand. We have discussed some features for ES2016 that might provide access to the module metadata, something like import {loader, name} from this module;.

That being said, I will not recommend to continue doing what you are doing since that sort of indirection is very limited, and really if the template is tied to the class exported by ./foo, it should either export the associated template or the class should provide static access to the associated imported template. I like the idea of exporting a template along side with the class because that means you can tap into the loader hooks to provide different templates per bucket, per device, etc, since the loader will allow to normalize and fetch the template.

[EisenbergEffect]

It's clear from above explanations that the design of ES6 prevents actually implementing a loader that addresses the state/hook inconsistencies. That's quite disappointing, but it is what it is.

Since there is an issue tracking the module registry, and that can be used as a workaround, at least for my own use cases, I'm going to close this issue out.

Hopefully, some of this can be improved in future versions.