Create ESM out of source files on NPM pre-publish

[ma2ciek]

📝 Provide a description of the new feature

First, thanks to @phaux for the proposal. Firstly I thought it's impossible, but after a talk, I'd give it a try as the benefits side of this feature is so huge.

The problem

Right now it's very hard to add something to the build without rebuilding it. It makes also the CKEditor 5 project very dependent on the webpack bundler (and on its configuration as well) and creates other issues when it comes to bundling CKEditor 5 sources inside frameworks.

The solution

What if we could just allow users to use ESM and just type the following in the JS file that won't need any bundling / transpilation and will just work in the browser:

import ClassicEditor from '@ckeditor/ckeditor5-editor-classic/esm/classiceditor';
import Essentials from '@ckeditor/ckeditor5-essentials/esm/essentials';
import Paragraph from '@ckeditor/ckeditor5-paragraph/esm/paragraph';

// or:
import { ClassicEditor } from '@ckeditor/ckeditor5-editor-classic';
import { Essentials } from '@ckeditor/ckeditor5-essentials';
import { Paragraph } from '@ckeditor/ckeditor5-paragraph';

ClassicEditor.create( div, {
   plugins: [ Essentials, Paragraph ]
} );

Benefits:

• Support for other bundlers (like rollup, parcel) and for webpack with 0 configuration
• Support for building inside other complex environments (React, Vue, Angular (and others!)) without hacking huge webpack configurations
  • https://github.com/ckeditor/ckeditor5-angular/issues/26
• Support for the node.js environment (probably as the CSS and SVG extensions would be gone) (but actually the gain is low since our plugins are split into the editing and UI parts)
• Getting rid of the source + build duplication error
• Support for CKEditor 5 source code directly in the browser's <script type="module">
• Support for online tools like https://stackblitz.com/
• Much easier setup for non-tech guys
• Ability to kill our lovely CKEditor 5 builds in the future

Cons:

• code size will probably increase
• another build and build tools to maintain
• loss of the PostCSS possibilities (although I'm not sure how many users actually changes the default theme lark to something different)
• additional pre-publish step to create EMS builds
• larger package sizes (who cares? 😄 )

Some more technical details:

There're at least three potential problems I'm aware of:

• PostCSS imports
• SVG imports
• translations

Since all our JS files are documented and can be imported, all our JS files should be entry points during the ESM generation process. SVG files and CSS files that are shared across multiple files should be moved to some shared code chunks.

But... I'm not sure if it gonna work at all 🙃The ESM building would need to be very precise to not duplicate CSS styles, which I think is the critical issue here.

Waiting for your critical opinions ;)

---

If you'd like to see this feature implemented, add a 👍 reaction to this post.

[phaux]

Things that cause problems in the code right now:

• Using import syntax to inject CSS
• Using import syntax to get SVG images as strings
• Custom syntax in CSS itself (e.g. mixins)
• Translations need a custom transform to generate necessary files.
• No .js extensions in import paths (it works in node but not in browsers)

The first thing we need to decide is which tool we will use ourselves to prepare the ESM versions of packages before publishing.

Things we should care about are:

• Scope hoisting so that the module system is entirely compiled away without inserting any module loader runtime into the published package. Another option is to not do any bundling at all and only do the necessary transforms per source file.
• Multiple entry points because we allow importing most of the files directly, and Code splitting so that the shared code ends up in a common chunk. Another option is to add an index.js file to each of our packages which re-exports everything so that we end up with a single bundle per package. That would be a breaking change, though.
• Minimal runtime. The tool can generate some helper functions (e.g. to inject CSS imports), but shouldn't do much more than that.
• Tree-shaking and dead code elimination might be useful to reduce published bundle size.
• ES Module output, duh.

Possible tool options are:

• 👍 Rollup is probably the best choice for packages like CKEditor. It has a small core and is very configurable via plugins. It's used by React, Redux, Mobx and many more.
• 👍 Parcel is mostly used for web apps, but it also has all the features we need and it does most of this stuff by default without any configuration.
• ❔ Microbundle seems like it doesn't have the necessary features. Some small libraries use it.
• ❔ Use Babel and PostCSS directly. This would alleviate the need for code splitting and scope hoisting because the output would have the same file structure. It doesn't transform the CSS and image imports though.
• ❌ Webpack injects its own module loading runtime and only supports partial scope hoisting. The output is always wrapped in at least one "virtual module" and the runtime is always there. It also doesn't support ESM output.
• ❌ Browserify – same as above and also doesn't have the necessary plugins.

So after a bit of thought, the first sensible options that come to my mind are:

• Use Rollup or Parcel, add index.js and get a single bundle per package that works with any bundler, node.js, and browsers.
• Use Rollup or Parcel and define an entry point for every file that the user might import directly and we end up with a few dist files which also work everywhere.
• Use Babel and PostCSS to transform some of the custom syntax (PostCSS syntax, translations stuff, etc) and let the user's bundler handle the other non-standard things like CSS imports. This won't work in node nor browser natively :(
• Don't do any pre-publish transforms at all. Just get rid of as much of the custom syntax as possible in the source itself.

In any case, the remaining blockers are:

• [ ] Figure out what to do with translations because they require a custom transform right now.
• [ ] Find a way for changing editor theme other than exposing PostCSS mixins, because we want to compile them away.

[Reinmar]

This is a really interesting idea and would definitely solve many issues.

However, I can't stop thinking about the blockers:

• Each package and even each module in a package can require a CSS file now. If we build this into pure ES modules, we need to bundle that CSS into those files somehow... but how? 100 modules, each adding its own stylesheet to a page. What about common dependencies? What about order of those stylesheets? It's not about code size here – it's about this not blowing up completely.
• Similar thing for translations. No idea how this could work :(.

Things that I think should be fairly easy:

• SVG imports are easy to transform into local variables so that's fine.
• Modifying themes happen via CSS variables most of the time, so I don't think many people will miss this option. Plus, there'd still be an option to go lower lever.

My other thought is that ESM has some common problems to the "adding plugins to builds" problem. Those blockers would concern me in both cases. So I've got a feeling that we'd have to tackle both things at the same time to not end up with a solution for ESM which later won't work for adding plugins to builds.

[ma2ciek]

Each package and even each module in a package can require a CSS file now. If we build this into pure ES modules, we need to bundle that CSS into those files somehow... but how? 100 modules, each adding its own stylesheet to a page. What about common dependencies? What about order of those stylesheets? It's not about code size here – it's about this not blowing up completely.

The first issue is probably solvable. The order of stylesheets could be a much bigger problem.

I could imagine that from such three endpoints with their dependencies:

- A.js
   - A.css
   - B.js
- B.js
   - B.css
- C.js
   - B.css

We could generate the following ESM:

 - A.js
    - A.css.js 
    - B.js
 - B.js
    - B.css.js
 - C.js
    - B.css.js
 - A.css.js 
 - B.css.js

So the CSS output won't be duplicated and will be imported once. However, that might won't work with PostCSS @import tags. (Though, I think that it might duplicate the CSS code even now)

My other thought is that ESM has some common problems to the "adding plugins to builds" problem. Those blockers would concern me in both cases. So I've got a feeling that we'd have to tackle both things at the same time to not end up with a solution for ESM which later won't work for adding plugins to builds.

The only way to solve the "adding plugins to builds" problem is to kill builds at some point IMHO.

[ma2ciek]

Similar thing for translations. No idea how this could work :(.

Maybe even something like this would do the trick if we collect all available translations:

import { polishTranslations } from '@ckeditor/ckeditor5-translations';
import translationService from '@ckeditor/ckeditor5-utils/src/translation-service';

translationService.add( 'pl', polishTranslations );

ClassicEditor.create( ... )

This way it could be super easy to fill the missing translations by hand if we skip the CKEditorWebpackPlugin:

import { polishTranslations: officialPolishTranslations } from '@ckeditor/ckeditor5-translations';
import translationService from '@ckeditor/ckeditor5-utils/src/translation-service';

translationService.add( 'pl', officialPolishTranslations );
translationService.add( 'pl', { Add: 'Dodaj' } );

[ma2ciek]

Actually this approach is used in other packages as well - https://date-fns.org (i18n example).

[Reinmar]

The only way to solve the "adding plugins to builds" problem is to kill builds at some point IMHO.

❤️

[ma2ciek]

It seems that the libraryTarget: 'module' is planned for the webpack 5 release, but it's still not implemented or implemented as an experimental feature - webpack/webpack#9802, https://webpack.js.org/configuration/experiments/#root. It will be interesting to see if it can bring us some value.

[CKEditorBot]

There has been no activity on this issue for the past year. We've marked it as stale and will close it in 30 days. We understand it may be relevant, so if you're interested in the solution, leave a comment or reaction under this issue.

[CKEditorBot]

We've closed your issue due to inactivity over the last year. We understand that the issue may still be relevant. If so, feel free to open a new one (and link this issue to it).