search

WordPress / gutenberg

The Block Editor project for WordPress and beyond. Plugin is available from the official repository.
https://wordpress.org/gutenberg/
Other
10.5k stars 4.2k forks source link

[Discussion] Upgrade `node` and `npm` to latest LTS versions #48588

Closed fluiddot closed 10 months ago

fluiddot commented 1 year ago

Intro

The purpose of this issue is to serve as the source of truth for the topic related to upgrading the node and npm versions, as well as be the central point for discussing how to move it forward.

See also a related PR in WordPress core: https://github.com/WordPress/wordpress-develop/pull/4028.

Context

Currently, Gutenberg allows the following versions:

The problem is that the recommended node version is in maintenance and the End-of-life is April, 30th, 2023, so it would be great to upgrade it. However, new node versions use npm versions above 6, which are currently blocked due to issues with dependencies (see below under the Blockers section).

Goal

Upgrade to the latest LTS version of node:

Or alternatively, if we find blockers using npm 9, upgrade to the latest LTS version that supports npm 8:

NOTE: End-of-life of node 18 is 2025-04-30.

Blockers

The main blockers at this point are related to dependencies/peer dependencies, in the following sections are described:

1. @testing-library/react-native dependency ✅

Error logs ``` npm ERR! code ERESOLVE npm ERR! ERESOLVE could not resolve npm ERR! npm ERR! While resolving: @testing-library/react-native@11.3.0 npm ERR! Found: jest@27.4.5 npm ERR! node_modules/jest npm ERR! dev jest@"27.4.5" from the root project npm ERR! peer jest@"^27.0.0" from jest-watch-typeahead@1.0.0 npm ERR! node_modules/jest-watch-typeahead npm ERR! dev jest-watch-typeahead@"1.0.0" from the root project npm ERR! 7 more (snapshot-diff, @wordpress/e2e-test-utils, ...) npm ERR! npm ERR! Could not resolve dependency: npm ERR! peerOptional jest@">=28.0.0" from @testing-library/react-native@11.3.0 npm ERR! node_modules/@testing-library/react-native npm ERR! dev @testing-library/react-native@"11.3.0" from the root project npm ERR! npm ERR! Conflicting peer dependency: jest@29.4.0 npm ERR! node_modules/jest npm ERR! peerOptional jest@">=28.0.0" from @testing-library/react-native@11.3.0 npm ERR! node_modules/@testing-library/react-native npm ERR! dev @testing-library/react-native@"11.3.0" from the root project ```

🔧 How to solve it: There's an ongoing effort to upgrade Jest version to 29 in https://github.com/WordPress/gutenberg/pull/47388, so we can either downgrade the dependency @testing-library/react-native to a previous version (10.1.1) or wait to the PR to be merged.

UPDATE: ✅ This issue has been solved with https://github.com/WordPress/gutenberg/pull/47388.

2. react-native dependency ❌

Error logs ``` npm ERR! code ERESOLVE npm ERR! ERESOLVE could not resolve npm ERR! npm ERR! While resolving: react-native@0.69.4 npm ERR! Found: react@18.2.0 npm ERR! node_modules/react npm ERR! dev react@"18.2.0" from the root project npm ERR! peer react@">=16.8.0" from @emotion/primitives-core@11.0.0 npm ERR! node_modules/@emotion/primitives-core npm ERR! @emotion/primitives-core@"^11.0.0" from @emotion/native@11.0.0 npm ERR! node_modules/@emotion/native npm ERR! dev @emotion/native@"11.0.0" from the root project npm ERR! 83 more (@emotion/react, @emotion/styled, ...) npm ERR! npm ERR! Could not resolve dependency: npm ERR! peer react@"18.0.0" from react-native@0.69.4 npm ERR! node_modules/react-native npm ERR! dev react-native@"0.69.4" from the root project npm ERR! peer react-native@">=0.14.0 <1" from @emotion/native@11.0.0 npm ERR! node_modules/@emotion/native npm ERR! dev @emotion/native@"11.0.0" from the root project npm ERR! 19 more (@react-native-clipboard/clipboard, ...) npm ERR! npm ERR! Conflicting peer dependency: react@18.0.0 npm ERR! node_modules/react npm ERR! peer react@"18.0.0" from react-native@0.69.4 npm ERR! node_modules/react-native npm ERR! dev react-native@"0.69.4" from the root project npm ERR! peer react-native@">=0.14.0 <1" from @emotion/native@11.0.0 npm ERR! node_modules/@emotion/native npm ERR! dev @emotion/native@"11.0.0" from the root project npm ERR! 19 more (@react-native-clipboard/clipboard, ...) ```

🔧 How to solve it: The optimal approach for this would be to upgrade React Native to version 0.71.0. However, in the meantime, we could override the dependency as a workaround.

3. Other warnings related to peer dependencies ⚠️

Warning logs ``` npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: @mdx-js/react@1.6.22 npm WARN Found: react@18.2.0 npm WARN node_modules/react npm WARN dev react@"18.2.0" from the root project npm WARN 86 more (@emotion/primitives-core, @emotion/react, ...) npm WARN npm WARN Could not resolve dependency: npm WARN peer react@"^16.13.1 || ^17.0.0" from @mdx-js/react@1.6.22 npm WARN node_modules/@mdx-js/react npm WARN @mdx-js/react@"^1.6.22" from @storybook/addon-docs@6.5.7 npm WARN node_modules/@storybook/addon-docs npm WARN npm WARN Conflicting peer dependency: react@17.0.2 npm WARN node_modules/react npm WARN peer react@"^16.13.1 || ^17.0.0" from @mdx-js/react@1.6.22 npm WARN node_modules/@mdx-js/react npm WARN @mdx-js/react@"^1.6.22" from @storybook/addon-docs@6.5.7 npm WARN node_modules/@storybook/addon-docs npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: ajv-keywords@3.4.1 npm WARN Found: ajv@8.7.1 npm WARN node_modules/ajv npm WARN dev ajv@"8.7.1" from the root project npm WARN 4 more (ajv-draft-04, ajv-errors, ajv-formats, table) npm WARN npm WARN Could not resolve dependency: npm WARN peer ajv@"^6.9.1" from ajv-keywords@3.4.1 npm WARN node_modules/ajv-keywords npm WARN ajv-keywords@"^3.4.1" from webpack@4.46.0 npm WARN node_modules/@storybook/core-common/node_modules/webpack npm WARN 4 more (webpack, webpack, schema-utils, schema-utils) npm WARN npm WARN Conflicting peer dependency: ajv@6.12.6 npm WARN node_modules/ajv npm WARN peer ajv@"^6.9.1" from ajv-keywords@3.4.1 npm WARN node_modules/ajv-keywords npm WARN ajv-keywords@"^3.4.1" from webpack@4.46.0 npm WARN node_modules/@storybook/core-common/node_modules/webpack npm WARN 4 more (webpack, webpack, schema-utils, schema-utils) npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: react-element-to-jsx-string@14.3.4 npm WARN Found: react@18.2.0 npm WARN node_modules/react npm WARN dev react@"18.2.0" from the root project npm WARN 86 more (@emotion/primitives-core, @emotion/react, ...) npm WARN npm WARN Could not resolve dependency: npm WARN peer react@"^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1" from react-element-to-jsx-string@14.3.4 npm WARN node_modules/react-element-to-jsx-string npm WARN react-element-to-jsx-string@"^14.3.4" from @storybook/react@6.5.7 npm WARN node_modules/@storybook/react npm WARN npm WARN Conflicting peer dependency: react@17.0.2 npm WARN node_modules/react npm WARN peer react@"^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1" from react-element-to-jsx-string@14.3.4 npm WARN node_modules/react-element-to-jsx-string npm WARN react-element-to-jsx-string@"^14.3.4" from @storybook/react@6.5.7 npm WARN node_modules/@storybook/react npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: react-element-to-jsx-string@14.3.4 npm WARN Found: react-dom@18.2.0 npm WARN node_modules/react-dom npm WARN dev react-dom@"18.2.0" from the root project npm WARN 52 more (@floating-ui/react-dom, @react-spring/web, ...) npm WARN npm WARN Could not resolve dependency: npm WARN peer react-dom@"^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1" from react-element-to-jsx-string@14.3.4 npm WARN node_modules/react-element-to-jsx-string npm WARN react-element-to-jsx-string@"^14.3.4" from @storybook/react@6.5.7 npm WARN node_modules/@storybook/react npm WARN npm WARN Conflicting peer dependency: react-dom@17.0.2 npm WARN node_modules/react-dom npm WARN peer react-dom@"^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1" from react-element-to-jsx-string@14.3.4 npm WARN node_modules/react-element-to-jsx-string npm WARN react-element-to-jsx-string@"^14.3.4" from @storybook/react@6.5.7 npm WARN node_modules/@storybook/react npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: react-inspector@5.1.1 npm WARN Found: react@18.2.0 npm WARN node_modules/react npm WARN dev react@"18.2.0" from the root project npm WARN 86 more (@emotion/primitives-core, @emotion/react, ...) npm WARN npm WARN Could not resolve dependency: npm WARN peer react@"^16.8.4 || ^17.0.0" from react-inspector@5.1.1 npm WARN node_modules/react-inspector npm WARN react-inspector@"^5.1.0" from @storybook/addon-actions@6.5.7 npm WARN node_modules/@storybook/addon-actions npm WARN npm WARN Conflicting peer dependency: react@17.0.2 npm WARN node_modules/react npm WARN peer react@"^16.8.4 || ^17.0.0" from react-inspector@5.1.1 npm WARN node_modules/react-inspector npm WARN react-inspector@"^5.1.0" from @storybook/addon-actions@6.5.7 npm WARN node_modules/@storybook/addon-actions npm WARN ERESOLVE overriding peer dependency npm WARN While resolving: webpack-filter-warnings-plugin@1.2.1 npm WARN Found: webpack@5.65.0 npm WARN node_modules/webpack npm WARN dev webpack@"5.65.0" from the root project npm WARN 35 more (@pmmmwh/react-refresh-webpack-plugin, ...) npm WARN npm WARN Could not resolve dependency: npm WARN peer webpack@"^2.0.0 || ^3.0.0 || ^4.0.0" from webpack-filter-warnings-plugin@1.2.1 npm WARN node_modules/webpack-filter-warnings-plugin npm WARN webpack-filter-warnings-plugin@"^1.2.1" from @storybook/builder-webpack4@6.5.7 npm WARN node_modules/@storybook/builder-webpack4 npm WARN npm WARN Conflicting peer dependency: webpack@4.46.0 npm WARN node_modules/webpack npm WARN peer webpack@"^2.0.0 || ^3.0.0 || ^4.0.0" from webpack-filter-warnings-plugin@1.2.1 npm WARN node_modules/webpack-filter-warnings-plugin npm WARN webpack-filter-warnings-plugin@"^1.2.1" from @storybook/builder-webpack4@6.5.7 npm WARN node_modules/@storybook/builder-webpack4 ```

🔧 How to solve it: It's likely that these warnings won't block using npm 8 but it would be interesting to address them in the future, especially for using npm 9.

Affected dependencies:

Related opened issues/PRs

NOTE: Please feel free to edit the above with issues/PRs related to this topic.

fluiddot commented 1 year ago

In relation to the React Native dependencies (@testing-library/react-native and react-native), I proposed a solution in this draft PR. The idea is to use the setting overrides and override these dependencies to the required versions: https://github.com/WordPress/gutenberg/blob/9827fcbd0e8387d05703c2d52e3c3ce02b12c498/package.json#L246-L253

NOTE: This solution is a temporary workaround until we upgrade the React Native and Jest versions.

tomdevisser commented 1 year ago

@fluiddot Thanks for this issue and the work you put in already, I think it's very important to address this. I have no experience with upgrading Node in a project like this, but will follow this to learn about it and think along where I can.

As there's not much time left before the end of April, is there a way we can give this a higher priority? Should it be added to a particular milestone?

And if we're not able to figure this out and merge it before the EOL, what are the risks of being late? Will things break, are there security implications or will it "just" not receive support anymore?

gziolo commented 1 year ago

@fluiddot, thank you for opening this issue and collecting related efforts to synchronize work.

I updated the description and included a reference to a related PR in WordPress core: https://github.com/WordPress/wordpress-develop/pull/4028.

After April 30th, once Gutenberg runs with NodeJS 18 and npm 9, we will need to follow up with upgrading the minimum required version to Node 16.0.0 and npm 7.10.0 that ships with it. @gvgvgvijayan made me realize that after opening #48368 last week. We will have to update engines field in package.json files, notes in README files, and list those changes in CHANGELOG files as a breaking to enforce a major version bump for updated packages. I found some prior art in https://github.com/WordPress/gutenberg/pull/43141.

fluiddot commented 1 year ago

I'm thinking about a potential plan in order to move the upgrade forward:

1. Fix the current dependency conflicts

Following the approach mentioned in this comment, we could fix the current dependency conflicts related to React native (@testing-library/react-native and react-native). This will allow starting using npm 7 or 8, although the package-lock.json file will be modified every time we install the dependencies.

ℹ️ I can work on this and have a PR ready today/tomorrow.

2. Decide the target version of node to upgrade to

We have two options:

If we go with the former, we'd need to fix other dependency conflicts that only happen when using npm 9. If we go with the latter, I think we could successfully migrate although we'd have the warnings mentioned in 3. Other warnings related to peer dependencies ⚠️ when installing the dependencies.

3. Upgrade node version

As @gziolo mentioned in this comment, we'd need to update engines field in package.json files, notes in README files, and list those changes in CHANGELOG files as a breaking to enforce a major version bump for updated packages. We could use https://github.com/WordPress/gutenberg/pull/43141 as a guide to know what needs to be updated when creating a PR for this.

How does this plan sound to you? Please let me know if you have any thoughts/concerns. Thanks 🙇 !

gziolo commented 1 year ago

An excellent action plan, @fluiddot. I don't have strong opinions about npm version. We enforce the maximum version of Node and npm so increasing that to node 18.13.0 / npm 8.19.3 is still better than waiting more time. If that helps splitting that into two steps sounds reasonable.

desrosj commented 1 year ago

Thanks for working on this all! I haven't taken a deep dive into the challenges on the Gutenberg side yet, but I have on the Core side in Trac-56658 (the updates in both locations will need to be coordinated to happen relatively at the same time). For the related Core changes, see WordPress/wordpress-develop#4028.

It's my strong preference to upgrade to the latest Node and npm versions in one change instead of a two step process if possible. Each time we change the version requirements for these tools, we have to communicate it to contributors so that everyone can make any needed adjustments to their workflows. I know nvm makes this really easy, but in reality, not everyone has this installed and configured.

Another thing to keep in mind is that 7.x of npm upgrade the lock file format. When we update to npm >= 7.0.0, we should also update the engine at the same time. The lock file versions are forwards and backwards compatible, but contributors on 6.x will see the lock file convert back to the v1 format making it impossible for someone using npm 6.x to make any changes to the lock file.

gziolo commented 1 year ago

For visibility, @kevin940726 opened PR https://github.com/WordPress/gutenberg/pull/48950 that upgrades Node to v18 and npm to v9. It contains more changes the discussed above, like:

fluiddot commented 1 year ago

Now that https://github.com/WordPress/gutenberg/pull/47388 is merged, I updated the PR's description to reflect that the issue with @testing-library/react-native dependency dependency is solved 🎊 .

fluiddot commented 1 year ago

Regarding https://github.com/WordPress/gutenberg/pull/48950 PR, I also opened a companion PR that aims to solve the issue with the React Native dependency: https://github.com/WordPress/gutenberg/pull/48990.

noahtallen commented 1 year ago

This should be partially resolved now with #53426, putting Gutenberg on Node 16 and NPM 8. Legacy peer dep mode is still enabled until we can fix the remaining issues! More here: https://make.wordpress.org/core/2023/08/09/updating-wordpress-to-use-more-modern-versions-of-node-js-npm/

gziolo commented 1 year ago

We still need to pick more changes prepared by @kevin940726 ing https://github.com/WordPress/gutenberg/pull/48950:

There might be more work ready to land in the branch 😄

Finally, we should remove the legacy-peer-deps flag override and start using the new npm handling of peer dependencies. This might require some audit of how we handle peer dependencies in WordPress packages and increase the usage of this good practice.

fluiddot commented 1 year ago

⚠️ I'd like to share that node 16 (the current version we use in Gutenberg) has already reached its End-Of-Life (2023-09-11).

Based on the package.json file, I understand that we could use node 18 but the main restriction comes from npm and using version 9. As I shared in the Goal section of issue's description, I wonder if we could apply the workaround of using node 18 by pointing to version 18.13.0, as it still uses npm 8. WDY?

In relation to this, upgrading to node 18 will be a blocker for future upgrades of React Native as starting in version 0.73 it's likely that the minimum supported node version is 18.

gziolo commented 1 year ago

@fluiddot, can you also comment here https://make.wordpress.org/systems/2023/02/09/upgrade-nodejs-npm-on-the-build-server/ about EOL for Node 16?

swissspidy commented 1 year ago

Node 18 is now supported.

Ticket and PR for core to go to Node 18:

noahtallen commented 1 year ago

By the way, Node 20 is now the current active LTS. @fluiddot, should we reach out to systems, try to get on v20, and skip v18 in the meantime?

swissspidy commented 1 year ago

Worth reaching out, but I'm skeptical about the timeline for that.

GitHub Actions is also skipping Node 18 and going straight to Node 20, see https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

If we could do that as well, it would mean having to maintain fewer Node versions across all branches.

On the other hand, the JS ecosystem isn't usually fast to catch up, so not sure how easy the migration would be for Node 20.

noahtallen commented 1 year ago

I don't think Node 20 is supposed to have many breaking changes, though I suppose it's worth trying in a branch!

noahtallen commented 1 year ago

Draft here, just to see what failures come up: #55561

noahtallen commented 1 year ago

Looks like CI is passing, so I think on the Gutenberg side, this will be an easy update. I'm not familiar with the core WP side though

fluiddot commented 1 year ago

By the way, Node 20 is now the current active LTS.

Hey @noahtallen 👋. Good point, following this documentation, node 18 is now in Maintenance mode until 2025-04-30. Hence, it would be ideal to use node 20 now that it's the current LTS.

In this regard, I'd like to note that the issues we had in the past have been related to upgrading npm, specifically when upgrading from version 6 to 8. Currently, we exclude npm version above 8 (reference), so I'm wondering if we'd identify potential blockers when upgrading to npm 10 🤔.

In any case, I'd lean toward choosing the easiest and quicker approach, either using version 18 or 20, as we still use node 16 which is already deprecated.

@fluiddot, should we reach out to systems, try to get on v20, and skip v18 in the meantime?

Sure, we could reach out to systems to notify them about using node 20. But as shared here, I'm also skeptical about a potential ETA.

desrosj commented 1 year ago

Worth reaching out, but I'm skeptical about the timeline for that.

I agree. It's unlikely we will get this support added on the build server this calendar year. But we can still make a request. If there are specific features in 20.x that could be utilized, we could build a more compelling case for it to be made available.

GitHub Actions is also skipping Node 18 and going straight to Node 20, see https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

Just noting that this only really applies to actions such as actions/checkout, though. actions/setup-node still supports any version of Node/npm. Good to look at what others are doing (and skipping 18 is also my preference), but it's not technically a restriction for our decision here.

18.x will also be in LTS maintenance mode for 18 months. There is a longer term security being on 18.x. 16.x was a bit of a unicorn because the EOL date was moved up by 7 months, and we were also on 14.x for a really long time.

I don't think Node 20 is supposed to have many breaking changes, though I suppose it's worth trying in a branch!

I think the issue that we may encounter will be dependencies having an upper limit to the version they allow similar to how we do. I think it's worth exploring so we know what the challenges will be, but there may be a bit of a waiting game for packages to update.

I'd also like to see us get to a state where we can remove legacy-peer-deps = true from .npmrc. I think that should be explored before 20.x.

With the WordPress 6.4 release coming out on November 7th, let's hold off on this until after that when contributors will have more time. I plan to circle back to this the following week and we can coordinate updating Core SVN and Gutenberg at the same time.

noahtallen commented 1 year ago

In my PR above, I did update to npm 10 at the same time, and everything appears to be working correctly in CI.

I'd love to remove legacy peer does too, but I don't think it makes a big difference which Node/npm version we're on when we do that.

By the way, do we know why it takes so long to make another Node binary available on the build servers?

noahtallen commented 1 year ago

I opened a wordpress-develop PR to test Node 20 and npm 10 for core: https://github.com/WordPress/wordpress-develop/pull/5571

noahtallen commented 1 year ago

I don't think I have access to create posts on https://make.wordpress.org/systems/. Could someone else create the request so that we could at get the process started?

fluiddot commented 1 year ago

I don't think I have access to create posts on https://make.wordpress.org/systems/. Could someone else create the request so that we could at get the process started?

I don't have permission either to create posts there. @desrosj since you are the author of the post Upgrade NodeJS/npm on the Build Server, I wonder if you could help us with this. Thanks 🙇 !

desrosj commented 11 months ago

I couldn't find an open PR that stages updating to Node.js 18.x, so I've gone and made one: #56331.

I also updated WordPress/wordpress-develop#5515. One thing that's different than the usual approach is that I removed the upper limit to the versions that can be used. In my testing, there are no problems when using 20.x. Even if we can't use 20.x as the version used to build the final software, I think it's reasonable to allow use of it and test against it as long as the results are the same.

For example, recently the format of the package-lock.json file completely changed. Running the install command on version 7 vs. 6 would cause the format to flip back and forth, which was not ideal because it would potentially create a ton of churn on lock files. If a situation like this is encountered, we would need to explore pinning a version maximum again.

Happy to post again on the systems blog as well. But I'd like to have a better reasoning than "please install the latest version" because that will put the request into their backlog instead of at the forefront.

noahtallen commented 11 months ago

But I'd like to have a better reasoning than "please install the latest version" because that will put the request into their backlog instead of at the forefront.

I think the main reason to update is because Node version 16 is now at end of life. I'm not sure exactly what it means, but I assume no more bug (and maybe even security) fixes

Node 20 is the current LTS, and I think it should be any project's priority to remain on the current, stable, supported versions of software. We also technically don't want the latest version (21), just the current LTS, which is Node 20. ;)

I didn't realize that Node 18 was already available on that build server; that makes things easier for us! :)

However, I'm not sure there will ever be a more urgent reason, until Node 18 reaches EoL in 2025. I feel like this shouldn't be an ad-hoc process -- we should just be in the habit of updating to the current LTS as it becomes active each year. It often becomes more work the longer you wait. So maybe it'd still be worth creating the request, because we don't want it to suddenly become urgent at the last minute a year down the road. I guess the other approach is just doing it at the last minute when it becomes a security issue (e.g. Node 14 and now Node 16).

The other thing on my mind is that if we could just skip Node 18 and go to 20 right away, that makes developer's lives slightly easier :)

Just my opinion about it!

flootr commented 11 months ago

I second what @noahtallen says and would argue staying on top of the current LTS version of Node.js is what we should aim for.

desrosj commented 10 months ago

Thanks for all the work folks! The changes have been merged to both Gutenberg and WordPress Core, and I've gone and published an announcement: https://make.wordpress.org/core/2023/12/20/updating-wordpress-to-use-more-modern-versions-of-node-js-npm-2/.

See you all in April for 22.x 😉