Update npm dependencies in CHT Conf - post-Node >= 18

m5r commented 2 weeks ago

Describe the issue Last time we updated dependencies we were still supporting node 8. Now that we've dropped support for node versions < 18, we can move forward and update the dependencies we couldn't update before.

Describe the improvement you'd like Update npm dependencies and test to make sure nothing is broken after the updates. We are behind many major versions of most packages we depend on, some of them are no longer maintained and need replacing like eslint-loader and eslint-plugin-node.

I made an attempt at upgrading ESLint to allow it to accept more recent syntax but it led me to replace eslint-loader with eslint-webpack-plugin, replace eslint-plugin-node with eslint-plugin-n, upgrade webpack to version 5, and then I eventually ran out of time to focus on the issue I was initially dealing with :man_shrugging:

One important thing to note is that since cht-conf v3, CHT deployments configurations have their own ESLint configuration (documented here). ESLint v8 deprecates the json config file approach .eslintrc for a javascript config file eslint.config.js instead. With ESLint v9, they dropped support for it and it no longer works without migrating to the new config format. They provide a CLI that automates some of the work for you but it still needs some manual tweaks to make it work for each project. This represents a breaking change for app devs that needs to be well documented and means a cht-conf v5. For now I would suggest updating ESLint to v8 to support both formats for now and update to v9 (or latest if v9 becomes obsolete) at a later time, when we're comfortable with releasing a cht-conf v5. EDIT: feedback from a team who recently migrated to ESLint v9 with Flat Config:

I expect similar occurrences with some of our dependencies, ESM/CJS incompatibilities with some others of our dependencies. Node 8.0 was released in 2017 (with the last version 8.17 in 2019) and the ecosystem has evolved a lot in that timeframe, we have 10 major versions of Node to catch up on and I estimate this maintenance to take quite some time (along the lines of weeks, not months - but definitely not days). IMO the part of the code that need extra attention during this process is the compilation code in src/lib/compilation/*.js that compiles the contact summary, tasks, and targets when compiling app settings.

garethbowen commented 2 weeks ago

Ideally we'd have the e2e testing suite merged before embarking on this to give us more confidence that nothing has regressed.

m5r commented 1 week ago

I opened PR #623 to update non-breaking dependencies. That leaves us with those dependencies left to upgrade:

  ◉ chai                            ^4.3.10  →    ^5.1.1
  ◉ chai-as-promised                 ^7.1.1  →    ^8.0.0
  ◉ chai-exclude                     ^2.1.0  →    ^2.1.1
  ◉ eslint                           ^6.8.0  →    ^9.7.0
  ◉ googleapis                      ^84.0.0  →  ^140.0.1
  ◉ open                             ^8.4.2  →   ^10.1.0
  ◉ pouchdb-adapter-http             ^7.2.2  →    ^9.0.0
  ◉ pouchdb-adapter-memory           ^7.2.2  →    ^9.0.0
  ◉ pouchdb-core                     ^7.2.2  →    ^9.0.0
  ◉ pouchdb-mapreduce                ^7.2.2  →    ^9.0.0
  ◉ pouchdb-session-authentication   ^1.3.0  →    ^1.4.0
  ◉ semantic-release                ^22.0.0  →   ^24.0.0
  ◉ terser-webpack-plugin            ^1.4.3  →   ^5.3.10
  ◉ webpack                         ^4.46.0  →   ^5.93.0
  ◉ xpath                            0.0.33  →    0.0.34

This list doesn't include the deprecated/unmaintained dependencies we might have to replace. WebStorm calls out request as vulnerable and its readme makes note that the package is deprecated and points to this issue for potential alternatives.

Like mentioned in the PR description we're:

keeping semantic-release on v22 instead of updating to latest v24 because starting with v23 they stopped supporting Node.js 18
not updating chai because chai-exclude still relies on chai <= 4 in their peerDependencies - note that this is changing, they recently merged a PR that resolves this, so we can expect a release soon allowing us to update chai then EDIT: see next item
keeping chai, chai-as-promised, and chai-exclude respectively on v4.3.10, v7.1.1, v2.1.0 because starting with their next major version they become ESM-only packages
keeping open on v8.4.2 instead of updating to latest v10 because starting with v9 it's an ESM-only package and ESLint is so outdated it can't parse the await import('package') syntax :man_facepalming:
not updating xpath from v0.0.33 to v0.0.34 because no changelog is provided, I'm not sure what changed in that version and this is an important dependency for parsing XPath paths in forms
keeping @parcel/watcher on v2.1.0 because v2.2.0-2.4.0 have incompatibilities with npm 10.4 (see https://github.com/parcel-bundler/watcher/issues/156) and v2.4.1 has a weird bug where the watcher hangs ~5% of the time when unsubscribing (see https://github.com/medic/cht-conf/pull/623#issuecomment-2237265325)

I'm not sure yet whether updating pouchdb-* dependencies will have to wait until we also update it in cht-core, I haven't looked into it yet.

My next initiative will be to update eslint to v8 to keep compatibility with our current eslint config format allowing us to give app developers and ourselves enough time to migrate to the new eslint config format before updating to v9.

m5r commented 3 days ago

Tackling the webpack upgrade, I'm running into same issue Gareth ran into when trying to update webpack a while ago. Our compile-contact-summary.js tests are failing because the compilation hangs indefinitely. I haven't found the cause yet but I'm building a minimal repro case to replicate the bug and locate where it's coming from.

medic / cht-conf

Update npm dependencies in CHT Conf - post-Node >= 18 #621