search

WICG / import-maps

How to control the behavior of JavaScript imports
https://html.spec.whatwg.org/multipage/webappapis.html#import-maps
Other
2.7k stars 72 forks source link

Proposed overhaul to be more URL-based #53

Closed domenic closed 6 years ago

domenic commented 6 years ago

Introduction

The Chrome team is keenly interested in being able to use package name maps both as a way of bringing the bare-import-specifier experience to the web, and as a way of enabling web platform features to be shipped as modules (the layered APIs project). In particular we want to enable the LAPI-related user stories in https://github.com/drufball/layered-apis/issues/34.

The current proposal was created specifically to solve the bare import specifier problem, and is pretty good at that, ongoing tweaks aside. But it only has some tentative gestures in the direction of web platform-supplied modules. The proposed syntaxes are half-baked and feel tacked on to the existing proposal, instead of integrating well with it.

My best attempt to use the current package name maps proposal to solve the LAPI use cases is https://github.com/drufball/layered-apis/pull/33. Its biggest drawback is the introduction of the secondary layeredapi: scheme in addition to the std/x (or @std/x) syntax for importing LAPIs. But we are forced into this awkward situation by the current proposal's separation of mapping import specifiers (the left-hand side) to URLs (the right-hand side).

The below is an alternative proposal that was developed from the ground-up to support both use cases in a unified way. It incoporates ideas from several other open issues and PRs along the way. Note that this is written as an evolution of the current proposal, for discussion and to gather thoughts. I'll probably also write a pull request that replaces the existing README with one implementing this proposal, i.e. as if we'd thought of this proposal in the first place. That'll be easier to read top-to-bottom. But I want to have this discussion first with existing collaborators, for which the below framing is probably better.

Proposal details

URL-based mapping, and the import: scheme

As alluded to in #23, it'd be ideal to have a URL scheme that says "use the package name map to resolve the contents". Let's call that scheme import:.

In the current proposal, the bare import specifiers are thought of as "primary", and import: as an add-on feature. That is, we have two namespaces: import specifiers, and URLs, and the purpose of the package name map is to map between them.

This proposal flips things around. Modules are always imported via URL. A URL is the module's primary identifier. There is just one piece of sugar: in JS import statements and import() expressions, the import: part will get auto-prepended for you, when you use a bare import specifier.

With this in hand, we reframe package name maps to be about URL-to-URL mapping. They are no longer about mapping from the import specifier namespace into the URL namespace. They operate entirely within the URL namespace. And most users will be using them to control the import: URL namespace. But you can also use them to control other parts of the URL namespace, which is useful for LAPI user story (C).

Recursive mapping

Now that we have URL-to-URL mapping, one naturally has to wonder: what happens when you map an import: URL to another import: URL? It recurses, of course!

The key question though is error-handling behavior. If you map an import: URL to some other import: URL which is known not to exist, what happens? In this proposal, the mapping gets dropped, perhaps with a warning in your dev console. This works out really well for the LAPI fallback user story (B), as we'll see.

The left- and right-hand sides of the mapping

We've talked about the above as a URL-to-URL mapping. But it's a bit more complex than that, I think.

The current proposal's setup is about mapping a class of module specifiers to a class of URLs, to support submodules. That is, "lodash": { "path": "/node_modules/lodash-es", "main": "lodash.js" } is designed to map both "lodash" -> "/node_modules/lodash-es/lodash.js" and "lodash/*" -> "node_modules/lodash-es/*".

Even if we change the left hand side to a URL (e.g. "import:lodash") instead of a module specifier (e.g. "lodash"), we want to keep this property.

Furthermore, we want to enable the fallback cases discussed in https://github.com/drufball/layered-apis/issues/34 user story (B), or https://github.com/drufball/layered-apis/issues/5. And personally, I want to do so in a way that isn't tied to LAPIs, and works for all modules; that seems way better if we can.

The solution is to extend the right-hand-side of the mapping to allow multiple forms:

Similarly, the left-hand side keeps its meaning today: it's not only a URL, but also a URL prefix for any submodules.

Examples

Bare import specifiers

Consider the existing examples from this repository. In this new URL-based world, they would be

{
  "mappings": {
    "import:moment": { "path": "/node_modules/moment/src", "main": "moment.js" },
    "import:lodash": { "path": "/node_modules/lodash-es", "main": "lodash.js" }
  }
}

Using the #52 behavior, we can just write this as

{
  "mappings": {
    "import:moment": "/node_modules/moment/src/moment.js",
    "import:lodash": "/node_modules/lodash-es/lodash.js
  }
}

We'll prefer this abbreviated form from now on.

Bare import specifiers with fallbacks

Let's say we wanted to use moment from a CDN, but if that CDN was down, fall back to our local copy. Then we could do this:

{
  "mappings": {
    "import:moment": [
      "https://unpkg.com/moment@2.22.2/src/moment.js",
      "/node_modules/moment/src/moment.js"
    ]
  }
}

LAPI fallbacks, user story (B)

Refresh yourself on https://github.com/drufball/layered-apis/issues/34, then consider this map:

{
  "mappings": {
    "import:@std/async-local-storage": [
      "import:@std/async-local-storage/index",
      "/node_modules/als-polyfill/index.mjs"
    ]
  }
}

This assumes that LAPIs modules are registered (by the browser) at import:@std/lapi-name/*, with an index module in particular existing for each LAPI.

In browser class (1): import "@std/async-local-storage" maps to the URL import:@std/async-local-storage/index which the browser has pre-registered a module for. It works!

In browser class (2): import "@std/async-local-storage" maps to the URL "/node_modules/als-polyfill/index.mjs", after trying import:@std/async-local-storage and getting a failure. It works!

LAPI fallbacks, user story (C)

Refresh yourself on https://github.com/drufball/layered-apis/issues/34, then consider this map:

{
  "mappings": {
    "/node_modules/als-polyfill": "import:@std/async-local-storage/index"
  }
}

In browser class (1): import "/node_modules/als-polyfill/index" maps to the URL import:@std/async-local-storage/index", which the browser has pre-registered a module for. It works!

In browser class (2): this mapping gets dropped from the package name map, per the "recursive mapping" rules above. So such browsers just use the original import statements, pulling in the polyfill. It works!

In browser class (3): the browser doesn't know about package name maps at all, so again the original import statements work, as desired.

Discussion

Overall this proposal accomplishes my goals. It allows package name maps to solve the LAPI use cases, while being more unified; they didn't grow any special capabilities or keys specific to LAPIs. It also solves #23, not in a tacked-on way, but in a way that gets integrated deeply into the mapping.

I see two drawbacks with this proposal:

As an example, if we used the dedicated fallbacks key and the new extension key, a package name map for user story (B) might look more like this:

{
  "mappings": {
    "import:moment": {
      "path": "https://unpkg.com/moment@2.22.2/src",
      "main": "moment",
      "extension": ".js"
    }
  },
  "fallbacks": {
    "import:moment": [{
      "path": "/node_modules/moment",
      "main": "moment",
      "extension": ".js"
    }],
    "import:@std/virtual-scroller": [{
      "path": "/node_modules/virtual-scroller-polyfill",
      "main": "index",
      "extension": ".mjs"
    }]
  }
}

Thoughts welcome, either on these points or more generally.

robpalme commented 6 years ago

Scope

I don't mind expanding the scope of PNMs to include fallbacks. It helps to have all resolution control in one place. It's good to have things that improve runtime efficiency by making more information available early, and help tooling understand whats going on by centralizing the information.

Observability

In a module loaded via a fallback, I assume the importee module can discover its resolved/physical identity via import.meta.url. Will there be a builtin way for the importer module to know what they loaded? Given we're introducing non-determinism in loaded modules when using an array of mappings, we'll need debugging/telemetry to remotely report on what modules your app truly loaded for a particular run.

Bare specifier definition

the import: part will get auto-appended for you, when you use a bare import specifier.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

Extensions

On the file extension drawback, I have a mild personal preference to lean in and drop _class-of-URLs_mappings. It promotes deep imports into arbitrary files within external packages, that may or may not be considered implementation details. I think it best to surface inter-package dependencies in the PNM to aid static analysis. My expectation is that PNMs will primarily be tool-generated (e.g. as a pre-deployment step), so I'm not worried about extra lines.

Misc

Two typos:

guybedford commented 6 years ago

Very exciting to see these directions being embraced.

arrays of the above, which result in trying each URL in sequence and falling back on network error or non-ok fetching status.

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Now for my concerns :)

The two arguments for URL mapping seem to be (1) standard module are URLs and (2) For use case (C) we want to support URL mapping.

I don't think this is an adequate justification for moving to the more complex URL -> URL mapping.

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

  1. Standard modules are URLs

You're introducing an indirection anyway here as well with an index for the standard modules to distinguish anyway.

If users just use a package map to map into the standard module URL instead of writing a URL (say import @std/async-local-storage), then the package name map can solve use case B fine.

  1. URL mapping for browsers without package map support (use case C)

This feels to me like trying to do too much in one go. The web traditionally lets the ecosystem handle compatibility problems, and instead focuses on the future workflows. If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

domenic commented 6 years ago

Thanks both for the comments!

@robpalme

Observability

An interesting question. I think the platform has some things that help with this, such as the resource timing API. Ideally I'd like to reuse those.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

No. In this scheme, you import URLs, always. And package name map left-hand sides are URLs. So "/node_modules/als-polyfill" is a URL, just like "import:lodash" is. Bare import specifiers are no longer really a thing, except that there's that auto-prepending sugar which gives you the same effect.

Extensions

Interesting viewpoint. On reflection, I think I agree. That makes this all a lot simpler, which helps address some of @guybedford's concerns. We can just have string left hand sides, and pure URL right-hand sides.

Two typos:

Fixed, thanks!

@guybedford

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

URL mapping for browsers without package map support (use case C) ... If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

Unfortunately this is a hard constraint we must meet. So I think it's best we work on a way to meet it while avoiding the complexity you're worried about. @robpalme's simplification might help in that regard... I'll start writing it up.

guybedford commented 6 years ago

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

I really like this form, and I think the way it expands is quite predictable and intutitive.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Do you allow infinite loops? Eg:

{
  packages: {
    "/x": {
      "path": "/x/x"
    }
  }
}

how do you know when to stop the recursion in this scenario?

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

Why should we be able to map standard modules, when indirection is fine?

Unfortunately this is a hard constraint we must meet.

Who is "we"? Don't change the web to solve things it doesn't need to solve. Why not go the tried and tested polyfill route rather?

domenic commented 6 years ago

how do you know when to stop the recursion in this scenario?

The same way you usually do? Keeping track of where you've been?

Why should we be able to map standard modules, when indirection is fine?

I don't understand what "indirection" is in this instance, or why it is fine.

Who is "we"? Don't change the web to solve things it doesn't need to solve.

"We" is the proposal authors and implementers. We are not interested in working on package name maps if they don't also solve the important use cases for built-in modules. I understand you might not think this is important to solve, but from talking with our customers and partners we've found that without a solution for these cases, there's not enough interest in this feature to be worth our devleopment effort.

Why not go the tried and tested polyfill route rather?

I don't understand what this means, as this whole proposal is about enabling the polyfill route to even be possible.

domenic commented 6 years ago

Writing up the proposal in more detail I've run into one fly in the ointment. It's weird that package name maps allow remapping of all URLs, but only the remapping of import: URLs works outside of import statements and import() expression contexts. Trying to explain this is quite weird, because I talk about the benefits of import: URLs and how they are usable everywhere, and then I talk about how you configure them using package name maps, but note that you can also configure other URLs using package name maps, and those aren't usable everywhere...

Not sure what to do with this. When the namespaces were separate, import: meant "use module specifier resolution", and that story was pretty easy. Now that there's just the URL namespace, but applied unevenly, it's more troubling. Might need to back up a few steps... Sigh.

Edit: current potential solution I'm thinking of is to say that (a) the package name map only changes import: URL resolution, even in import and import(); but (b) import: always gets auto-prepended to import and import() statements. So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs". We'll see what that looks like. In this version you wouldn't use import: prefixes in the package name map.

domenic commented 6 years ago

Started on a writeup, but ran out of time for today. Still, should give the basic idea, for those very actively engaged: https://github.com/domenic/package-name-maps/blob/url-based/README.md

littledan commented 6 years ago

I'm a big fan of just about everything in this iteration of the proposal, and agree with @domenic that polyfilling built-in modules is very important. This proposal seems to work well for both the case of an entirely missing built-in module, and the ability to wrap a built-in module in another module (e.g., in case a function is added, and you want to polyfill it in old browsers; see https://github.com/msaboff/JavaScript-Standard-Library/issues/2).

Potential solution: lean into it, and get rid of the class-of-URLs-to-class of URLs mapping entirely? I.e. make everything 1:1, so you'd need to enumerate the submodules of each package, both for LAPIs and non-LAPIs.

I'm wondering whether there's something halfway to that point: Could we say that, if you use a string, you're indicating just a single mapping, and then the object can be used to permit submodules as well, if we want that feature?

daKmoR commented 6 years ago

To be honest, when this first started I was a little hesitant as the "current" implementation with path_prefix, packages, scopes, main and everything looked really complicated.

However, this new flatter approach is sooo much easier to understand, generate and I hope applying it would also be not too complex.

So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs"

So especially in this usecase the "import:" would do nothing right? as it's followed by a /. If it where import:foo/bar then it would actually look up foo in the map. Just wanna be sure about my understanding.

@guybedford I'm curious if you would consider this "new" approch in your shim maybe in a branch? I would like to play around with a generator of this map... most likely on top of yarns "beta" plug and play implementation

domenic commented 6 years ago

Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look.

MylesBorins commented 6 years ago

This looks great! Any expectation for a name change?

On Tue, Oct 2, 2018, 8:48 PM Domenic Denicola notifications@github.com wrote:

Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look https://github.com/domenic/package-name-maps/blob/336ccbcc2000235733ce3fd93b98934b47882c8e/README.md .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/domenic/package-name-maps/issues/53#issuecomment-426477231, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV9H7CO-T8u4LF8oiFcnzZBaZmbleks5uhAlbgaJpZM4WEvVh .

domenic commented 6 years ago

Yeah, tentatively "module import maps" instead of "package name maps" since it's now focused more on controlling imports in general and deemphasizing the package concept.

ljharb commented 6 years ago

@domenic This looks great! Not sure where to provide feedback:

michael-ciniawsky commented 6 years ago

What's the reasoning behind import:./x, import:../y, import:/z? Can't those not just use a 'bare specifier' aswell and left as is otherwise?

url('bare/logo')
<link href="bare/style">
import module from 'bare'
{
   modules: {
      bare: {
         '/': '/path/to/bare', // Path (Context)
         index: './lib/main.js',
         style: './lib/css/main.css',
         logo: './lib/assets/logo.svg',
         ...
      }
   }
}
domenic commented 6 years ago

@michael-ciniawsky I don't really understand your example, but the reasoning is to support https://github.com/domenic/package-name-maps/blob/url-based/README.md#for-built-in-modules-in-browsers-without-module-import-maps

michael-ciniawsky commented 6 years ago

What is unclear can you elaborate?

michael-ciniawsky commented 6 years ago

Why would pollyfilling need support for ./, ../, / in particular?


{
  "modules": {
    bare: {
      index: [
        "import:@std/buildin",
        "/node_modules/polyfill/index.js",
        "https://cdn.fallbacks.com/polyfill/index.js"
      ]
    }
  }
}
ljharb commented 6 years ago

So you can default to your own implementation but use module maps to gracefully replace it with the builtin when available, is my understanding.

robpalme commented 6 years ago

This new version is good. The trailing / convention is neat.

dcleao commented 6 years ago

I like the uniformity and power of the proposed import: scheme, as it would be usable in all contexts where a url can be specified.

littledan commented 6 years ago

The newest version looks even better. Two small comments:

nyaxt commented 6 years ago

The latest draft lgtm.

FYI: @yutakahirano

I think we need to clarify:

Would it make sense to implement this by having import: URLs issue redirect to the resolved URLs?

RangerMauve commented 6 years ago

How would this new import: scheme work with the work being done on changing navigator.registerProtocolHandler to use a blocklist instead of an allow list?

I assume import: would need to be added to the blocklist.

kinu commented 6 years ago

The proposal looks exciting. Let me list some of the comments/questions I had (some might have been already discussed or explained).