Tracking Issue: Script Modules API

luisherranz commented 1 year ago

Part of https://github.com/WordPress/gutenberg/issues/36716.

The goal of this initiative is to implement the necessary low-level primitives to add native support in WordPress for registering and enqueueing JavaScript modules, including generating an import map. This will allow developers to leverage the native JavaScript modules system available in modern browsers.

Tasks

[x] Basic version of the Modules API
- Trac: https://core.trac.wordpress.org/ticket/56313
- PR: https://github.com/WordPress/wordpress-develop/pull/5818
[x] Renaming from Modules API to Script Modules API
- Trac: https://core.trac.wordpress.org/ticket/56313
- PR: https://github.com/WordPress/wordpress-develop/pull/5869
[x] Print the import map and enqueued modules in the footer in classic themes
- Trac: https://core.trac.wordpress.org/ticket/60240
- PR: https://github.com/WordPress/wordpress-develop/pull/5931
[x] Update the code in Gutenberg and move it to the compat/wordpress-6.5 folder
- PR: https://github.com/WordPress/gutenberg/pull/57778
[x] Add import map polyfill
- Trac: https://core.trac.wordpress.org/ticket/60348
- PR: https://github.com/WordPress/wordpress-develop/pull/5947
[x] Figure out how to overwrite the script modules registered by WordPress Core in the Gutenberg plugin
- Trac: https://core.trac.wordpress.org/ticket/60463
- PR: https://github.com/WordPress/wordpress-develop/pull/6061

luisherranz commented 1 year ago

First experiment: implement the simplest system possible

I've opened a PR with the first experiment:

https://github.com/WordPress/gutenberg/pull/56092

This is the simplest system possible, delegating dependencies completely to the client.

An API to register modules using:
- A module identifier (required)
- The full URL or the path (required)
- Whether the module is for the admin, the frontend or both (required)
- A version (optional)

Feel free to review or add any comments.

luisherranz commented 12 months ago

Second experiment: add the possibility to preload static dependencies of enqueued modules

I've opened a PR with the second experiment:

https://github.com/WordPress/gutenberg/pull/56118

This is a slightly more complex system that includes an optional graph of static dependencies to be able to avoid the initial waterfalls:

An API to register modules using a module identifier, a version, a path and whether the module is for the admin, the frontend, or both.
An API to enqueue modules.
An optional server graph of static dependencies.
All registered modules add an entry in the import map.
Enqueued modules add a <script> tag in the head.
Static dependencies of enqueued modules add a <link rel="modulepreload"> in the head.

Feel free to review or add any comments.

westonruter commented 12 months ago

There are also various comments about modules on Core-12009, the work to implement script loading strategies.

luisherranz commented 12 months ago

Thanks, Weston. I'll take a look 🙂

thomas-price commented 12 months ago

For reference, some earlier tickets:

luisherranz commented 12 months ago

Thanks, Thomas 🙂

Third experiment: use a server dependency graph to know which modules are used in the page

I've opened a PR with the third experiment:

https://github.com/WordPress/gutenberg/pull/56143

This is a more complex system than the first ones, including a required array of both the static and dynamic dependencies to be able to populate the import map only with the necessary entries:

An API to register modules using:
- A module identifier (required)
- The full URL or the path (required)
- The static and dynamic dependencies (required)
- A version (optional)
An API to enqueue modules.
Only the dependent modules add an entry in the import map.
Enqueued modules add a <script> tag in the head.
The static dependencies are preloaded when the module is enqueued using a <link rel="modulepreload"> tag in the head.

Feel free to review or add any comments.

youknowriad commented 12 months ago

@luisherranz to be honest, all of the experiences are decent. I do think at the minimum we should have the "preload" deps. So I'm thinking we should go either with 2 or 3 but I suspect that the decision is going to be hard and not really objective. So how do you think should we go about this? Is there a world where we can land one, explore it in Gutenberg plugin for some weeks/months before committing entirely.

I also think that it would be good to explore automating the "deps" extraction in a follow-up and in the Gutenberg plugin automate all the gutenberg_register_module calls.

luisherranz commented 12 months ago

So I'm thinking we should go either with 2 or 3 but I suspect that the decision is going to be hard and not really objective

Yeah, I also think 2 or 3 are the best options, as I think there should be an easy way to avoid the initial waterfalls.

I also think that it would be good to explore automating the "deps" extraction in a follow-up and in the Gutenberg plugin automate all the gutenberg_register_module calls.

Yes, that's one of the next things I want to explore: how easy it is to extract the dependencies to some sort of manifest (or autogenerated PHP file similar to the one we have for scripts), and make Gutenberg consume it automatically.

Do you have any suggestions on how to approach this? I was thinking about experimenting with a viewModule property in the block.json file.

Is there a world where we can land one, explore it in Gutenberg plugin for some weeks/months before committing entirely.

Absolutely. I think we can easily land this for the frontend (Interactivity API), and once it's in trunk, try to use it in the Block Lazy Loading experiments as well. I'll prepare everything and test the polyfill.

youknowriad commented 12 months ago

Do you have any suggestions on how to approach this? I was thinking about experimenting with a viewModule property in the block.json file.

I think it should be independent of blocks entirely, it should just be a Webpack plugin like the existing dependency extraction one.

luisherranz commented 12 months ago

Ok. I'll work on that as a first step 🙂

luisherranz commented 12 months ago

I added the polyfill to all the experiments. For now, I used es-module-shims, which seems to be the most battle-tested one. I tried it in old versions of Chrome and Safari (both in macOS and iOS) using BrowserStack and it works great.

However, I won't use this version for WordPress Core, as it includes other shims that are not part of the spec, and if people start using them, it will be impossible to remove the polyfill later. So I'd either talk with Guy Bedford to see if he can generate a slimmed-down version for WordPress, or I'd create a custom version ourselves.

youknowriad commented 12 months ago

Import maps seem to be already supported by all the browsers supported by WordPress so it seems that in theory we don't need the shim. By the time the API reaches Core, the support will be even better. Or do you want to be extra safe here?

luisherranz commented 12 months ago

I was under the impression that WordPress was meant to support all browsers with a market share bigger than 1%, but I'll review that policy again. In any case, I'd like to ensure that something is ready in case it's needed 🙂

youknowriad commented 12 months ago

oh you're right I misread this. There's still one version of chrome and another of Safari that doesn't meat the threshold for the moment.

sgomes commented 11 months ago

Thank you for working on all these experiments, @luisherranz! I'm sure it was a ton of work, but it really is invaluable to have something concrete to analyse 🙇

All three options are good, as @youknowriad mentioned, but the scalability of the first two concerns me.

WordPress ships with a lot of scripts, and if we imagine a future where a large portion of them are loadable through this mechanism, we're looking at a pretty large import map for Core alone, never mind any plugins that add extra dependencies. Adding a large import map to every page request is going to make the document larger, which can have a noticeable impact on performance (as some recent performance regressions with the inline CSS generated by the Fonts API proved).

Since option 3 is the only one that allows for WordPress to tailor the import map to the actual needs of the page, that's the only one that can help us avoid any scalability issues. Yes, it is is a more complex solution that requires a bit more work from developers, but we could try to reduce that burden with automated extraction, as @youknowriad suggested. If on the other hand we were to simplify the API (by picking option 1 or 2) in order to increase adoption, I would expect that decision to backfire on us, as vigorous adoption would mean rushing to those scalability issues. So it seems best to address them from the start and make sure they never happen.

Beyond that, I do like the shape of the API a lot, as I think it achieves a nice balance between familiarity with wp_(enqueue|register)_script and meeting the specific needs of module scripts 👍 Thanks again, @luisherranz!

luisherranz commented 11 months ago

Thanks to you for your insights, Sérgio. I really appreciate them 🙂🙏

Is there a world where we can land one, explore it in Gutenberg plugin for some weeks/months before committing entirely.

We've prepared everything to land the third one in Gutenberg and test it out in the frontend:

https://github.com/WordPress/gutenberg/pull/56143#pullrequestreview-1753305594

Feel free to review!

luisherranz commented 11 months ago

Yesterday, we merged the PR that includes the first Modules API version in Gutenberg, and we modified the interactive blocks to start using modules in the frontend:

https://github.com/WordPress/gutenberg/pull/56143

This will be released in GB 17.2 next week.

luisherranz commented 11 months ago

Let's start talking about some of the questions that @youknowriad asked in the last PR.

Specifically, I'd like to start talking about using modules in scripts and vice-versa:

How can existing scripts make use of modules?

Imagine a new package called @wordpress/utils, only released as a module.
- How existing scripts (that can't be migrated to modules due to backward compatibility issues) can import that package?
- How can they make sure that the @wordpress/utils module is available on the page?
How can modules make use of existing scripts?

Imagine developers that want to start using modules for their block, not only in the frontend, but also in the Editor.
- How can they import existing scripts like @wordpress/block-editor in their edit.js module?
- How can they make sure that the @wordpress/block-editor script is available on the page?

luisherranz commented 11 months ago

I've been thinking about Riad's questions, and I believe it makes sense to focus on them and create a few experiments to explore possible solutions. The reasoning for exploring this path is:

Migrating from scripts to modules is not possible for scripts that have external dependants due to backward compatibility issues. This means there are going to be scripts that will remain scripts forever.

To increase the adoption of modules, it makes sense that:

New modules can still use those remaining scripts as dependencies.

Ideally, developers should be able to use modules instead of scripts, and the need for script dependencies should not be an obstacle to the decision to use modules instead of scripts.
Those remaining scripts can use new modules as dependencies.

Ideally, all new packages should be modules from now on, and the need for existing scripts to use those modules as dependencies should not be an obstacle to the decision to use modules instead of scripts.

I also believe that the priority should be on the modules:

From now on, WordPress developers should be able to use modules instead of scripts for all their needs.
All the existing WordPress packages should be available to be consumed as modules from now on, even if some of them are linked to existing scripts under the hood.
Developers should not need to know which WordPress packages are linked to existing scripts. For them, everything should look like regular modules from now on.
Developers should not need to use a script-related bundling tool while building their modules anymore.
Existing scripts should be able to import modules, but in this case, the API can be explicit: developers should know when they are importing a module instead of a script.

I have a couple of ideas to make this work. I'll continue working on this and open PRs with the experiments once they are ready, but feedback and ideas are welcomed 🙂

youknowriad commented 11 months ago

@luisherranz We've introduced a new package recently (dataviews) and it's used only in edit-site for the moment. It's a package that is "bundled" right now which means breaking changes are still possible for this package. This package could potentially be rewritten as a module as part of these experiments (if you're looking for something to explore with)

luisherranz commented 11 months ago

That's perfect. Thanks, Riad. We can try making it a module and importing other existing packages (scripts) as if they were modules, and importing this module from existing packages (scripts).

luisherranz commented 11 months ago

@adamziel made a proof of concept plugin for Playground that exposes all wordpress packages as modules:

https://github.com/WordPress/playground-tools/pull/133

felixarntz commented 10 months ago

@luisherranz Not sure this is the right place (feel free to reroute me), but I had one specific question and one broader consideration related to supporting modules and import maps after using the Gutenberg API for the first time today:

For the import maps polyfill, it looks like that JS file is always loaded? I was using latest Chrome and it loaded for me, which it probably shouldn't (since Chrome supports import maps). Is there a way to perform a quick support check with inline JS instead so that the polyfill can only be loaded as needed? It's over 12kB, so if we can avoid that for most sites, that would be a good win.
The more extensive point, which certainly requires more thinking and discussion is whether there is a potential place here to support web worker modules here as well (see e.g. https://web.dev/articles/module-workers#preload_workers_with_modulepreload). While web worker modules aren't directly imported, they are still referenced by the relevant regular modules that use them. It might be worth thinking about having some API like wp_register_worker_module() potentially, and allow regular modules to provide them as dependencies? The main consideration here would be to include modulepreload link tags for the relevant worker modules that are dependencies of the enqueued modules, while omitting them from the import map (unless I miss something and there is actually value in including worker modules there).

gziolo commented 10 months ago

For the import maps polyfill, it looks like that JS file is always loaded? I was using latest Chrome and it loaded for me, which it probably shouldn't (since Chrome supports import maps). Is there a way to perform a quick support check with inline JS instead so that the polyfill can only be loaded as needed? It's over 12kB, so if we can avoid that for most sites, that would be a good win.

I opened https://github.com/WordPress/gutenberg/pull/57256 to test whether we can replicate the technique used in WordPress core to print polyfills only when the feature is missing in the browser.

luisherranz commented 10 months ago

I improved the tests and the API, specifically the dependency array:

https://github.com/WordPress/gutenberg/pull/57231

Then, I opened a PR in WordPress Core with this Modules API to gather more feedback:

https://github.com/WordPress/wordpress-develop/pull/5818

luisherranz commented 10 months ago

Hey @felixarntz,

Is there a way to perform a quick support check with inline JS instead so that the polyfill can only be loaded as needed?

Absolutely. I'm talking with Guy Bedford, the creator of es-module-shims, to create a polyfill-only version that can be loaded dynamically:

https://github.com/guybedford/es-module-shims/issues/406

I'd appreciate any help on that front 🙂

It might be worth thinking about having some API like wp_register_worker_module() potentially, and allow regular modules to provide them as dependencies?

It makes sense, although I've barely worked with workers, so I'm not familiar with the requirements. Is this something that needs to be considered from the beginning, or could it be added in the future as an enhancement?

felixarntz commented 10 months ago

@luisherranz

It makes sense, although I've barely worked with workers, so I'm not familiar with the requirements. Is this something that needs to be considered from the beginning, or could it be added in the future as an enhancement?

It could certainly be added later, e.g. with a new function. I believe supporting it could be relatively straightforward though, as from the PHP API perspective it would be quite similar to regular modules. The main difference would be in the JS code itself (a regular module wouldn't import a worker module, but rather register that script as a worker).

I think it could work as follows:

gutenberg_register_worker_module() does something similar to gutenberg_register_module(), except it puts modules into its own dependency chain, separate from the regular modules.
Regular modules should be able to depend on worker modules, but worker modules should not be able to depend on regular modules. The rationale:
- A worker module should be able to import another worker module.
- A worker module should not be able to import a regular module.
- A regular module should be able to register a worker module as a web worker.
When a regular module depends on a worker module, the relevant worker modules would be added to the modulepreload.

sirreal commented 10 months ago

I created https://github.com/WordPress/gutenberg/issues/57492. It's a proposal to support editorModule, module, and viewModule fields in block.json to use modules instead of scripts.

https://github.com/WordPress/gutenberg/issues/57492

cbravobernal commented 10 months ago

I created https://github.com/WordPress/gutenberg/pull/57778 to follow the standard procedure of having the Modules API in Gutenberg and refer to a backport PR for 6.5. I hope the Core Editor Tech Lead team does not get too confused.

Feel free to ping of needed!

https://github.com/WordPress/gutenberg/pull/57778

luisherranz commented 10 months ago

Core Editor Tech Lead team does not get too confused

Confused why?

luisherranz commented 10 months ago

At the request of some core committers, I've made a PR in Core to rename everything to "script module" instead of just "module":

https://github.com/WordPress/wordpress-develop/pull/5869

luisherranz commented 10 months ago

This is still a local PR until https://github.com/WordPress/wordpress-develop/pull/5869 is merged, but these should be the changes required to print everything correctly in classic themes:

https://github.com/luisherranz/wordpress-develop/pull/1

Kudos to @c4rl0sbr4v0 for coming up with the proper fix.

luisherranz commented 9 months ago

Now that https://github.com/WordPress/wordpress-develop/pull/5869 has been committed, I opened the PR to print everything correctly in WordPress Core:

https://github.com/WordPress/wordpress-develop/pull/5931

luisherranz commented 9 months ago

I've also updated the description of this Tracking Issue to reflect the accomplished tasks and next steps:

https://github.com/WordPress/gutenberg/issues/55942#issue-1981760006

luisherranz commented 9 months ago

With the polyfill backport, everything is now merged in WP Core and ready for WP 6.5.

The only thing left is to ensure that we can overwrite the script modules registered by WordPress Core in the Gutenberg plugin.

cbravobernal commented 9 months ago

The only thing left is to ensure that we can overwrite the script modules registered by WordPress Core in the Gutenberg plugin.

I guess the best way is to allow extenders to deregister modules. https://github.com/WordPress/wordpress-develop/pull/6061

gziolo commented 8 months ago

Everything is ready for WordPress 6.5 release scheduled for March 26th. Major props to everyone involved that helped to make it happen 🎉

WordPress / gutenberg