Failed package: 'ProcessTerminatedError: cancel after 0 retries!' 502 Bad Gateway

Expected behavior

Can import our npm-published module @stdlib/esm via:

// the whole library:
import stdlib from 'https://cdn.skypack.dev/@stdlib/esm';

// or a single function from it:
import besselj0 from 'https://cdn.skypack.dev/@stdlib/esm/math/base/special/besselj0.js';

I've confirmed the library can be loaded in this manner in deno, via github (see below).

Observed behavior

A page with this script loaded in Chrome 87.0.4280.88:

<script type="module">
import stdlib from 'https://cdn.skypack.dev/@stdlib/esm';
</script>

yields the network error:

Request URL: https://cdn.skypack.dev/error/build:@stdlib/esm@v0.0.2-ProcessTerminatedError:%20cancel%20after%200%20retries!

Or running via deno,

Download https://cdn.skypack.dev/new/@stdlib/esm@v0.0.2/dist=es2020
Download https://cdn.skypack.dev/error/build:@stdlib/esm@v0.0.2-ProcessTerminatedError:%20cancel%20after%200%20retries!
error: Import 'https://cdn.skypack.dev/error/build:@stdlib/esm@v0.0.2-ProcessTerminatedError:%20cancel%20after%200%20retries!' failed: 502 Bad Gateway
    at https://cdn.skypack.dev/@stdlib/esm:15:0

Trying to import a single small function instead of the whole library yields the same error.

Background and context

I'm working on an ES module distribution for the stdlib-js (via skypack.dev) project. The goal of the project is to be something akin to numpy/scipy but with careful consideration for the idiosyncrasies of browser distribution. The library is moderately large (94MB unpacked, with sample datasets), so that ES modules seem like a good fit since you want to be able to import any piece of the library easily but would almost never want to import the entire library at once.

You can import single functions in deno from github, and it works smoothly and downloads subsets of the library, as desired.

// Single imports (imports only the desired functions):
import besselj0 from 'https://github.com/stdlib-js/esm/raw/main/math/base/special/besselj0.js'
import ellipk from 'https://github.com/stdlib-js/esm/raw/main/math/base/special/ellipk.js'

// Namespace imports (imports all of math/base/special):
import { besselj0, ellipk } from 'https://github.com/stdlib-js/esm/raw/main/math/base/special.js'

// The *whole thing* 🙈 
import stdlib from 'https://github.com/stdlib-js/esm'

console.log( besselj0( 1.0 ), ellipk( 0.5 ) );
// => 0.7651976865579666, 1.854074677301372

We'd love to make the library work smoothly with module-aware CDNs, but it fails on skypack.dev, perhaps because the entire module is simply too large—or perhaps for some unrelated reason. I've run the package-check tool to hopefully get it in the ballpark of correct, but it's not clear to me quite what to do about the failure.

Any suggestions or alternate approaches would be greatly appreciated! You can find the full source of the ES module distribution here on github.

Of possible note is that the primary stdlib repo is cjs and that I've used a build process to construct the esm dist repo. That includes minification, browser shims, and sourcemaps so that it can (ideally, I have to fix a bug in the sourcemaps) be served as-is. My naive guess is that skypack does some tasks which we've also done and is perhaps reaching a timeout due to the size of the library.

Thanks for raising this! I’m not completely certain, but I do believe this may have something to do with a memory leak in Rollup that happens from building such a large package. We’re working on a behind-the-scenes change now that will allow us to build larger packages like this.

But as an aside, just want to flag something:

// or a single function from it:
import besselj0 from 'https://cdn.skypack.dev/@stdlib/esm/math/base/special/besselj0.js';

I noticed that in your package.json, this is your "exports" field:

"exports": {
  "import": "./index.js",
  "default": "./index.js"
},

If you only allow ./index.js in exports, then you don’t be able to import ../besselj0.js. From the official Node docs:

The "exports" field provides an alternative to "main" where the package main entry point can be defined while also encapsulating the package, preventing any other entry points besides those defined in "exports".

Skypack will likewise mimic this behavior. In other words, if you have "exports", then all files in your package become private by default and can’t be imported unless they appear in "exports" as well.

To make this easier, you can specify wildcards, e.g.:

"exports": {
  ".": "./index.js",
  "./*": "./*.js"
}

But it should be noted that wildcards can only go one level deep (child folders must be re-declared). And also Skypack doesn’t support wildcards yet but we’re working on it! We hope to release support of full export map functionality, including wildcards, early next year. Until then, unfortunately, we recommend listing out every public file in "exports" which we do support today.

Thanks for the thoughtful reply, @drwpow! If there's anything I can do to either help debug or make this easier for your tooling, I'm glad to do what I can.

Regarding exports, thanks for confirming my suspicion! I considered listing every single file in exports (maybe ~2600 🙈), but was hoping to avoid it unless definitely necessary. Can add though 👍

I also had a question about minifying and sourcemaps. As part of our cjs->esm script, I've bundled, minified and (almost; they have a bug I'm not eager to fix) created sourcemaps. Bundling is worthwhile on our side to boil each unit down to a single file, but I wasn't sure whether esm-aware CDNs would take their own approach for the minification/sourcemap phase. Unpkg, for one, strips out sourcmap info when you append the (experimental) ?module flag.

Backing up just slightly without getting too grandiose, the use-case seems to me valid, which is to say, a coherent, comprehensive, numpy/scipy-like library in js, and that ES modules are a somewhat novel way to distribute software and manage dependencies, but if there's something different we could be doing that would obviate the need for your tooling to handle such a stress test, I'm certainly interested in/willing to adjust our approach. Splitting into separate modules would simplify matters, though something like import-maps for cross-module dependencies would seem necessary before that would be possible.

Thanks again!

skypackjs / skypack-cdn