search

thi-ng / umbrella

⛱ Broadly scoped ecosystem & mono-repository of 199 TypeScript projects (and ~180 examples) for general purpose, functional, data driven development
https://thi.ng
Apache License 2.0
3.41k stars 151 forks source link

[all] Phasing out CJS & UMD builds in favor of ESM #315

Closed postspectacular closed 3 years ago

postspectacular commented 3 years ago

ESM support has been pretty solid in all major browsers for a while now AND recent versions of NodeJS seemingly cope with these modules very well too (at least in my experience). It's long been a bug bear (and major time waster) for me having to emit 3 different versions (ESM, CJS, UMD) of each package and maintain extra tooling for producing these. I don't know if there're still people out there using CJS or UMD in 2021, but going forward and staying with the times, all packages in this monorepo will only be published as ESM, with these (top off my head) benefits:

Since this will mean breaking changes for some users (depending on how/where the projects were used so far), this is just a heads up and note to say that intense work is/has already been ongoing (1700+ source files already updated at the time of writing) on the feature/export-maps branch...

The readme files on this branch have already been updated re: installation & import instructions, take a look and let me know please!

As part of this major (structural) update there will also be some further internal restructurings (or even split ups) needed for some of the packages... All in all, a somewhat overdue house cleaning exercise & I hope as such, it's a welcome one for you too!

👍

postspectacular commented 3 years ago

Some promised statistics to illustrate file size savings in the ~100 examples bundled in the repo.

The first table shows a few of the larger examples which have been updated with selective deep imports and their resulting new file sizes ranging between an incredible 5%(!) - 64% compared to previous builds:

example old new ratio
dominant-colors 122.83 KB 61.78 KB 50.3%
fft-synth 171.25 KB 73.20 KB 42.7%
geom-voronoi-mst 132.46 KB 66.58 KB 50.3%
imgui 486.44 KB 111.86 KB 23.0%
parse-playground 397.03 KB 49.25 KB 12.4%
poly-spline 151.96 KB 54.54 KB 35.9%
rdom-dnd 353.82 KB 16.13 KB 4.6%
rdom-lissajous 403.46 KB 32.18 KB 8.0%
rstream-spreadsheet 70.35 KB 44.87 KB 63.8%
shader-ast-raymarch 146.99 KB 83.89 KB 57.1%
shader-ast-workers 94.15 KB 15.14 KB 16.1%
soa-ecs 141.30 KB 75.06 KB 53.1%
text-canvas 103.11 KB 34.61 KB 33.6%
webgl-msdf 169.20 KB 91.98 KB 54.4%
xml-converter 71.79 KB 34.94 KB 48.7%

Some of these more extreme savings are seemingly due to the previous inability of snowpack/webpack to efficiently filter/tree-shake larger packages like the @thi.ng/hiccup-carbon-icons collection (1000+ icons), @thi.ng/vectors (850+ functions) or the @thi.ng/geom geometry package (to name a few). I don't understand why these didn't get optimized properly before, but being able to e.g. directly import only the functions/icons we care about, the problem now disappears entirely. Nice! Most of these import updates are nicely assisted by VSCode and the fact that packages now provide an export map... Only takes a few mins per (lager) example...

The next table shows the impact of the new ESM-only packages without doing any code/import changes to the examples with a range of new file sizes between 37% - 99%:

example old new ratio
adaptive-threshold 101.00 KB 60.36 KB 59.8%
async-effect 31.81 KB 28.59 KB 89.9%
bitmap-font 61.07 KB 40.19 KB 65.8%
canvas-dial 83.37 KB 63.02 KB 75.6%
cellular-automata 22.12 KB 19.26 KB 87.1%
color-themes 121.17 KB 87.34 KB 72.1%
crypto-chart 113.20 KB 64.28 KB 56.8%
devcards 30.75 KB 29.95 KB 97.4%
dominant-colors 122.83 KB 100.50 KB 81.8%
ellipse-proximity 100.57 KB 60.99 KB 60.6%
fft-synth 171.25 KB 120.46 KB 70.3%
geom-convex-hull 114.78 KB 73.51 KB 64.0%
geom-fuzz-basics 140.57 KB 81.25 KB 57.8%
geom-knn 121.43 KB 64.74 KB 53.3%
geom-tessel 152.74 KB 106.38 KB 69.6%
geom-voronoi-mst 132.46 KB 104.33 KB 78.8%
gesture-analysis 158.58 KB 112.29 KB 70.8%
grid-iterators 53.61 KB 35.48 KB 66.2%
hdom-basics 15.11 KB 12.76 KB 84.5%
hdom-benchmark 54.34 KB 33.33 KB 61.3%
hdom-benchmark2 34.44 KB 31.69 KB 92.0%
hdom-canvas-clock 68.83 KB 46.56 KB 67.7%
hdom-canvas-draw 109.76 KB 69.49 KB 63.3%
hdom-canvas-particles 115.51 KB 87.71 KB 75.9%
hdom-canvas-shapes 178.63 KB 151.90 KB 85.0%
hdom-dropdown-fuzzy 40.39 KB 39.12 KB 96.9%
hdom-dropdown 34.08 KB 32.75 KB 96.1%
hdom-dyn-context 28.49 KB 27.70 KB 97.3%
hdom-elm 20.48 KB 18.24 KB 89.0%
hdom-inner-html 15.40 KB 13.05 KB 84.8%
hdom-local-render 56.26 KB 35.19 KB 62.5%
hdom-localstate 16.62 KB 14.37 KB 86.5%
hdom-skip-nested 15.70 KB 13.35 KB 85.0%
hdom-skip 15.43 KB 13.08 KB 84.8%
hdom-theme-adr-0003 17.87 KB 15.62 KB 87.4%
hdom-toggle 17.92 KB 15.02 KB 83.8%
hdom-vscroller 202.50 KB 199.70 KB 98.6%
hiccup-canvas-arcs 137.67 KB 108.42 KB 78.8%
hydrate-basics 31.63 KB 30.31 KB 95.8%
imgui-basics 164.40 KB 106.07 KB 64.5%
imgui 486.44 KB 452.80 KB 93.1%
interceptor-basics 33.26 KB 31.94 KB 96.0%
interceptor-basics2 33.24 KB 32.45 KB 97.6%
iso-plasma 116.03 KB 74.75 KB 64.4%
json-components 17.32 KB 14.51 KB 83.8%
login-form 32.60 KB 32.52 KB 99.8%
mandelbrot 76.97 KB 55.93 KB 72.7%
markdown 93.69 KB 47.67 KB 50.9%
multitouch 109.06 KB 51.36 KB 47.1%
parse-playground 397.03 KB 376.94 KB 94.9%
pixel-basics 46.04 KB 22.01 KB 47.8%
pixel-indexed 79.28 KB 59.76 KB 75.4%
pixel-sorting 114.84 KB 79.80 KB 69.5%
poisson-circles 115.70 KB 87.33 KB 75.5%
poly-spline 151.96 KB 92.70 KB 61.0%
porter-duff 46.34 KB 22.31 KB 48.1%
ramp-synth 73.19 KB 50.94 KB 69.6%
rdom-basics 58.96 KB 35.13 KB 59.6%
rdom-delayed-update 53.57 KB 30.25 KB 56.5%
rdom-dnd 353.82 KB 316.93 KB 89.6%
rdom-lissajous 403.46 KB 346.40 KB 85.9%
rdom-search-docs 68.58 KB 44.85 KB 65.4%
rdom-svg-nodes 96.57 KB 61.09 KB 63.3%
rotating-voronoi 178.61 KB 132.79 KB 74.3%
router-basics 42.66 KB 41.90 KB 98.2%
rstream-dataflow 70.00 KB 50.38 KB 72.0%
rstream-event-loop 55.53 KB 35.33 KB 63.6%
rstream-grid 130.91 KB 78.63 KB 60.1%
rstream-hdom 57.30 KB 36.30 KB 63.3%
rstream-spreadsheet 70.35 KB 54.75 KB 77.8%
scenegraph-image 133.78 KB NaN bytes 0.0%
scenegraph 117.42 KB 90.38 KB 77.0%
shader-ast-canvas2d 125.25 KB 85.88 KB 68.6%
shader-ast-evo 126.30 KB 80.49 KB 63.7%
shader-ast-noise 145.57 KB 106.43 KB 73.1%
shader-ast-raymarch 146.99 KB 107.86 KB 73.4%
shader-ast-sdf2d 145.52 KB 106.39 KB 73.1%
shader-ast-tunnel 166.55 KB 114.40 KB 68.7%
shader-ast-workers 94.15 KB 35.10 KB 37.3%
shader-graph 141.70 KB 110.36 KB 77.9%
soa-ecs 141.30 KB 97.08 KB 68.7%
spline-tangent 107.47 KB 81.30 KB 75.7%
stratified-grid 115.51 KB 87.13 KB 75.4%
svg-barchart 18.81 KB 16.03 KB 85.2%
svg-particles 20.12 KB 17.31 KB 86.1%
svg-waveform 86.34 KB 48.72 KB 56.4%
talk-slides 76.63 KB 55.58 KB 72.5%
text-canvas-image 50.85 KB 24.57 KB 48.3%
text-canvas 103.11 KB 76.72 KB 74.4%
todo-list 31.31 KB 30.00 KB 95.8%
transducers-hdom 56.52 KB 35.50 KB 62.8%
triple-query 79.99 KB 61.31 KB 76.7%
webgl-cube 137.03 KB 92.55 KB 67.5%
webgl-cubemap 146.12 KB 82.71 KB 56.6%
webgl-grid 137.85 KB 93.26 KB 67.6%
webgl-msdf 169.20 KB 122.85 KB 72.6%
webgl-multipass 136.96 KB 94.81 KB 69.2%
webgl-shadertoy 120.67 KB 73.83 KB 61.2%
webgl-ssao 157.14 KB 139.51 KB 88.8%
wolfram 73.10 KB 52.24 KB 71.5%
xml-converter 71.79 KB 50.75 KB 70.7%
postspectacular commented 3 years ago

All new packages are released and several downstream projects successfully updated. Closing this now.

Ps. Should you run into any issues on your end, please re-open and explain... thx!

dmnsgn commented 2 years ago

Hi @postspectacular,

We (@vorg) are hitting an issue with the use of the "Conditional exports" fields (since v5 in paths and color package at least) when using const paths = require("@thi.ng/paths");:

Module not found: Error: Package path . is not exported from package 
/Users/.../thing-paths-test/node_modules/@thi.ng/paths 
(see exports field in /Users/.../thing-paths-test/node_modules/@thi.ng/paths/package.json)

Quoting Node.js docs, I think using "import" only makes sense when a package exports both cjs and esm:

When using environment branches, always include a "default" condition where possible. Providing a "default" condition ensures that any unknown JS environments are able to use this universal implementation, which helps avoid these JS environments from having to pretend to be existing environments in order to support packages with conditional exports. For this reason, using "node" and "default" condition branches is usually preferable to using "node" and "browser" condition branches.

So this issue is easily resolve by not specifying the condition or adding the default key to the package.json alongside the "import" (@thi.ng/paths example below):

    ".": "./index.js",
    "./api": "./api.js",
    "./delete-in": "./delete-in.js",
    "./get-in": "./get-in.js",
    "./getter": "./getter.js",
    "./mut-in-many": "./mut-in-many.js",
    "./mut-in": "./mut-in.js",
    "./mutator": "./mutator.js",
    "./path": "./path.js",
    "./set-in-many": "./set-in-many.js",
    "./set-in": "./set-in.js",
    "./setter": "./setter.js",
    "./update-in": "./update-in.js",
    "./updater": "./updater.js"
"exports": {
    ".": {
      "import": "./index.js",
      "default": "./index.js"
    },
    "./api": {
      "import": "./api.js",
      "default": "./api.js"
    },
    "./delete-in": {
      "import": "./delete-in.js",
      "default": "./delete-in.js"
    },
// ...
  },
postspectacular commented 2 years ago

Hi @dmnsgn - I don't think I understand what you're asking... For the past 7 months all the packages have ONLY been published as ESM and aren't usable in a CJS environment anymore. So I don't see how an updated export map would help to restore CJS compatibility if the actual syntax in the source files are incompatible with this older format... can you please explain how that would (suppose to) work? Also, may I ask why you're still using CJS?

dmnsgn commented 2 years ago

For the past 7 months all the packages have ONLY been published as ESM and aren't usable in a CJS environment anymore. if the actual syntax in the source files are incompatible with this older format

The fact that they are published as ESM doesn't mean that they can't be used in a CJS env where loading ESM modules has been backported to the supported Node.js versions. The difference is that you need to dynamically import them.

Simple example with concat-typed-array (which is published as ESM):

package.json

{
  "name": "a-cjs-module",
  "version": "1.0.0",
  "main": "index.cjs",
  "scripts": {
    "test": "node ."
  },
  "dependencies": {
    "concat-typed-array": "^2.1.0"
  }
}

index.cjs

// import concatTypedArray from "concat-typed-array";
// Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
// SyntaxError: Cannot use import statement outside a module

// const concatTypedArray = require("concat-typed-array");
// Error [ERR_REQUIRE_ESM]: require() of ES Module /my-path/a-cjs-module/node_modules/concat-typed-array/index.js from /my-path/a-cjs-module/index.cjs not supported.
// Instead change the require of index.js in /my-path/a-cjs-module/index.cjs to a dynamic import() which is available in all CommonJS modules.

(async () => {
    const concatTypedArray = (await import("concat-typed-array")).default;

    console.log(
        concatTypedArray(Uint8Array, Uint8Array.of(1, 2), Uint8Array.of(3, 4))
    );
        // Uint8Array(4) [ 1, 2, 3, 4 ]
})();

So I don't see how an updated export map would help to restore CJS compatibility can you please explain how that would (suppose to) work?

My understanding of the error is that because the thing packages provide conditional exports with the "import" key and no "default", bundlers or babel don't know what to do when mixing ESM and CJS (trying to load an esm module inside a cjs one).

The "default" key would allow fallbacks for these not-so straightforward loading cases (from Node.js docs):

"default" - the generic fallback that always matches. Can be a CommonJS or ES module file. This condition should always come last.

Also, may I ask why you're still using CJS?

Avoiding it as much as possible but still indirectly needed inside Nodes as they are written in legacy cjs (although this might change in a near future).

postspectacular commented 2 years ago

Thanks @dmnsgn - have just pushed changes using default instead of import everywhere. Seems to work indeed, didn't know (also didn't want to know, was glad to leave the CJS vs ESM nightmares behind me...) Will do some more tests, then hopefully release later today...

dmnsgn commented 2 years ago

I moved all my packages to ESM last year and looking forward to forget about CJS too.. I have sticked to "exports": "./index.js" though where I export everything public; no conditional exports yet, hoping tree-shaking works correctly.

postspectacular commented 2 years ago

@dmnsgn could you please check the latest pkg versions and let me know if the changes helped your cause? Thank you!

dmnsgn commented 2 years ago

Yes it did! Thanks 🙏

postspectacular commented 2 years ago

Amazing! :)

stevedevel commented 1 year ago

I dont know if this is related but I have a very strange error saying that:

Error [ERR_REQUIRE_ESM]: require() of ES Module C:\Users\xxx\node_modules\@thi.ng\grid-iterators\index.js not supported. Instead change the require of index.js in null to a dynamic import() which is available in all CommonJS modules.

There is no require in my code and as far as I can see and I could not find require in the grid-iterators/index.js. I am using windows, visual studio code and have already deleted caches, reinstalled node_modules, etc.

Do you have any advise?

postspectacular commented 1 year ago

Sorry @stevedevel - I've got no idea what this is supposed to mean. Is your project using CJS? Are you using a bundler? Can you reproduce the issue in CodeSandbox or CodePen etc. or create a temporary dummy repo/gist? Also, which version of Node are you using?

stevedevel commented 1 year ago

Hi Karsten,

Thanks for your reply. I have setuped a simple ts project (on node 16.20):

import { floodFill } from "@thi.ng/grid-iterators"; let structure:number[] = [
1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0 ] let size = 5; const region = floodFill((x, y) => structure[x + y * 5] === 1, 1, 0, size, size);

Thats the package.json:

{ "name": "test", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start:dev": "npx nodemon" }, "author": "", "license": "ISC", "dependencies": { "@thi.ng/grid-iterators": "^4.0.19" }, "devDependencies": { "@babel/core": "^7.23.0", "@types/node": "^20.7.0", "nodemon": "^3.0.1", "ts-node": "^10.9.1", "typescript": "^5.2.2" } }

Same error here. There is no require in the code above. I have a large project with all sorts of js/ts code and did not experience this error message. Very strange.

postspectacular commented 1 year ago

I have recreated your project, downloaded & installed node 16.20.2 and can confirm the same error...

However, looking at your package.json, I can also see you'll still need to:

1) add the line "type": "module" to allow your code using ESM imports (unless you want to stay in CommonJS land, and if so, why?)... 2) add some barebones tsconfig.json, something like:

{
    "compilerOptions": {
        "module": "es2020",
        "target": "es2020",
        "esModuleInterop": true,
        "experimentalDecorators": true,
        "moduleResolution": "node",
        "strict": true,
        "strictNullChecks": true,
        "verbatimModuleSyntax": true
    },
   "ts-node": { "esm": true },
    "exclude": ["./**/node_modules"]
}

Once you've got these, you can then run your script via yarn ts-node index.ts or node --loader ts-node/esm index.ts...

(Ps. None of these things have anything to do with this particular project though...)

stevedevel commented 1 year ago

Thank you so much for your answer - greatly appreciated. Especially because this has indeed nothing to do with your code. Now the small test works!

I am using so many 3rd party libs with different JS dialects (CJS, TS, ESM...) within a create-react-app. Thats why I thought there was maybe something related with the library - which is fantastic by the way.

Now my challenge: I cant really modify the toolchain without ejecting and do everything manually :-((

postspectacular commented 1 year ago

Don't envy you & I really have no idea how most JS devs accept & manage to cope with this kind of absolutely insane & unnecessary complexity...

earthlyreason commented 1 year ago

I really have no idea how most JS devs accept & manage to cope with this kind of absolutely insane & unnecessary complexity

That's because you knew better than to accept any dependencies of your own.