Explore Yarn PnP - Take 2

tschaffter commented 2 years ago

I recently upgraded this monorepo to Yarn 2+, which greatly reduces the time required to install/update npm dependencies. One issue left unaddressed is the activation of the PnP mode generated an error. See https://github.com/Sage-Bionetworks/challenge-registry/issues/588#issuecomment-1225835942. Using PnP should further reduce the runtime of Yarn locally and in CI/CD workflows.

Checklist

❌ Can add binaries to the path, for example to use nx instead of yarn nx.
❌ Jest works with PnP.
❔ All VS Code extensions that relies on npm packages work.
✔️ Can run npm install and npm install --immutable.
✔️ Accessing package type definition in VS Code
✔️ CI/CD workflow can still cache the npm dependencies.

References

Advanced package manager features for npm, Yarn, and pnpm
Getting started with Yarn 2 and TypeScript
- See relation between pnpify and Yarn SDK for VS Code.

tschaffter commented 2 years ago

Clean the yarn cache: yarn cache clean
Cache location:
- By default: .yarn/cache
- With enableGlobalCache: true: ~/.yarn/berry/cache

$ yarn config get cacheFolder
/home/vscode/.yarn/berry/cache

Storing the cache outside of the git workspace folder contributes to make git faster as git will have to do less work when it comes to .gitignore evaluations.

tschaffter commented 2 years ago

(✔️) VS Code extensions breaks if they don't find binaries in the path, e.g. `prettier` in the screenshot below:

TODO:

Review all VS Code extensions that depend on npm packages.

tschaffter commented 2 years ago

As mentioned in the first ticket I opened about migrating to Yarn 2+ with PnP, some package may break if they don't specify their dependencies properly. I just bumped into this issue with Nx not specifying that it depends on dotenv. This issue has been previously reported here.

I can installed dotenv manually but it should be removed next time I'm updating Nx, which should solve the above issue.

tschaffter commented 2 years ago

❌ Jest is broken when using PnP

Multiple tests are failing with the same error:

    ✖  nx run challenge-registry-org-search:test
 FAIL   web-org-search  libs/challenge-registry/org-search/src/lib/org-search.component.spec.ts
  ● Test suite failed to run

           TypeError: Cannot read properties of undefined (reading 'resolvedFileName')

             at module.exports (../../../../../home/vscode/.yarn/berry/cache/@nrwl-jest-npm-14.5.4-571814ec0a-8.zip/packages/jest/plugins/resolver.ts:85:22)

See:

The issue is project specific:

yarn nx test challenge-registry-ui --skip-nx-cache works
yarn nx run challenge-registry-challenge-search:test --skip-nx-cache fails

TODO:

Try yarn add -D jest-pnp-resolver => does not solve the issue

Running yarn jest from the project folder generates the same issue.

vscode@d8a8967a009a:/workspaces/challenge-registry/libs/challenge-registry/org-search$ yarn jest
 FAIL   web-org-search  src/lib/org-search.component.spec.ts
  ● Test suite failed to run

    TypeError: Cannot read properties of undefined (reading 'resolvedFileName')

      at module.exports (../../../.yarn/unplugged/@nrwl-jest-npm-14.5.4-571814ec0a/packages/jest/plugins/resolver.ts:85:22)

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        1.559 s
Ran all test suites.

yarn.lock has the following peer dependencies:

jest-pnp-resolver
jest-resolve

The file .yarn/unplugged/@nrwl-jest-npm-14.5.4-571814ec0a/packages/jest/plugins/resolver.ts does not exists. The closest one I could find is .yarn/unplugged/@nrwl-jest-npm-14.5.4-571814ec0a/node_modules/@nrwl/jest/plugins/resolver.js.

Nx version installed: 14.5.4
Current Nx version: 14.8.3

The migration didn't success but trying my luck after package.json has been updated by the migration process.

Issue 1

Test environment jest-environment-jsdom cannot be found. Make sure the testEnvironment configuration option points to an existing node module.

Solution: yarn add -D jest-environment-jsdom

Issue 2

$ yarn nx run challenge-registry-challenge-search:test --skip-nx-cache

> nx run challenge-registry-challenge-search:test

 >  NX   jest-preset-angular tried to access jest-util, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

   Required package: jest-util
   Required by: jest-preset-angular@virtual:c90490eb2ad1863a8b234a152a63ef6b1e3395e58b6a870ee84e43041440f27fae9b212a409125e99a745ab235e9cd29b2c3141d8f30ed464f0452d2433a03f7#npm:12.2.2 (via /workspaces/challenge-registry/.yarn/__virtual__/jest-preset-angular-virtual-eac328e650/0/cache/jest-preset-angular-npm-12.2.2-5fbc5bb22d-0cb3738744.zip/node_modules/jest-preset-angular/build/config/)

   Require stack:
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-preset-angular-virtual-eac328e650/0/cache/jest-preset-angular-npm-12.2.2-5fbc5bb22d-0cb3738744.zip/node_modules/jest-preset-angular/build/config/ng-jest-config.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-preset-angular-virtual-eac328e650/0/cache/jest-preset-angular-npm-12.2.2-5fbc5bb22d-0cb3738744.zip/node_modules/jest-preset-angular/build/ng-jest-transformer.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-preset-angular-virtual-eac328e650/0/cache/jest-preset-angular-npm-12.2.2-5fbc5bb22d-0cb3738744.zip/node_modules/jest-preset-angular/build/index.js
   - /workspaces/challenge-registry/.yarn/cache/jest-util-npm-28.1.3-9ae2283a08-fd6459742c.zip/node_modules/jest-util/build/requireOrImportModule.js
   - /workspaces/challenge-registry/.yarn/cache/jest-util-npm-28.1.3-9ae2283a08-fd6459742c.zip/node_modules/jest-util/build/index.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-config-virtual-e08a44d8a6/0/cache/jest-config-npm-28.1.1-8c4e855059-8ce9f6b8f6.zip/node_modules/jest-config/build/getCacheDirectory.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-config-virtual-e08a44d8a6/0/cache/jest-config-npm-28.1.1-8c4e855059-8ce9f6b8f6.zip/node_modules/jest-config/build/Defaults.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-config-virtual-e08a44d8a6/0/cache/jest-config-npm-28.1.1-8c4e855059-8ce9f6b8f6.zip/node_modules/jest-config/build/normalize.js
   - /workspaces/challenge-registry/.yarn/__virtual__/jest-config-virtual-e08a44d8a6/0/cache/jest-config-npm-28.1.1-8c4e855059-8ce9f6b8f6.zip/node_modules/jest-config/build/index.js
   - /workspaces/challenge-registry/.yarn/cache/@nrwl-jest-npm-14.8.3-0be5500c1d-65c2ed91ad.zip/node_modules/@nrwl/jest/src/executors/jest/jest.impl.js
   - /workspaces/challenge-registry/.yarn/unplugged/nx-virtual-a6e8072397/node_modules/nx/src/config/workspaces.js
   - /workspaces/challenge-registry/.yarn/unplugged/nx-virtual-a6e8072397/node_modules/nx/src/command-line/run.js
   - /workspaces/challenge-registry/.yarn/unplugged/nx-virtual-a6e8072397/node_modules/nx/bin/run-executor.js
   Pass --verbose to see the stacktrace.

Solution: yarn add -D jest-util

Issue 3

$ yarn nx run challenge-registry-challenge-search:test --skip-nx-cache

> nx run challenge-registry-challenge-search:test

 FAIL   web-challenge-search  libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts
  ● Test suite failed to run

    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:1:34 - error TS2307: Cannot find module '@angular/common/http' or its corresponding type declarations.

    1 import { HttpClientModule } from '@angular/common/http';
                                       ~~~~~~~~~~~~~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:2:40 - error TS2307: Cannot find module '@angular/core' or its corresponding type declarations.

    2 import { CUSTOM_ELEMENTS_SCHEMA } from '@angular/core';
                                             ~~~~~~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:3:43 - error TS2307: Cannot find module '@angular/core/testing' or its corresponding type declarations.

    3 import { ComponentFixture, TestBed } from '@angular/core/testing';
                                                ~~~~~~~~~~~~~~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:8:1 - error TS2593: Cannot find name 'describe'. Do you need to install type definitions for a test runner? Try `npm i --save-dev @types/jest` or `npm i --save-dev @types/mocha` and then add 'jest' or 'mocha' to the types field in your tsconfig.

    8 describe('ChallengeSearchComponent', () => {
      ~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:12:3 - error TS2304: Cannot find name 'beforeEach'.

    12   beforeEach(async () => {
         ~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:12:14 - error TS2354: This syntax requires an imported helper but module 'tslib' cannot be found.

    12   beforeEach(async () => {
                    ~~~~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:21:3 - error TS2304: Cannot find name 'beforeEach'.

    21   beforeEach(() => {
         ~~~~~~~~~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:27:3 - error TS2593: Cannot find name 'it'. Do you need to install type definitions for a test runner? Try `npm i --save-dev @types/jest` or `npm i --save-dev @types/mocha` and then add 'jest' or 'mocha' to the types field in your tsconfig.

    27   it('should create', () => {
         ~~
    libs/challenge-registry/challenge-search/src/lib/challenge-search.component.spec.ts:28:5 - error TS2304: Cannot find name 'expect'.

    28     expect(component).toBeTruthy();
           ~~~~~~

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        6.678 s
Ran all test suites.

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Ran target test for project challenge-registry-challenge-search (11s)

TODO:

Try updating Nx without PnP, then come back to this issue. => Did not solve the issue

tschaffter commented 2 years ago

Install Yarn SDK for VS Code

$ yarn dlx @yarnpkg/sdks vscode
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 1s 372ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ has-own-prop@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ repeat-string@npm:1.6.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ tinylogic@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ treeify@npm:1.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ typanion@npm:3.12.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 0s 888ms
➤ YN0000: ┌ Link step
➤ YN0000: └ Completed
➤ YN0000: Done in 2s 476ms

➤ YN0000: ┌ Generating SDKs inside .yarn/sdks
➤ YN0000: │ ✓ Eslint
➤ YN0000: │ ✓ Prettier
➤ YN0000: │ ✓ Typescript
➤ YN0000: │ • 3 SDKs were skipped based on your root dependencies
➤ YN0000: └ Completed
➤ YN0000: ┌ Generating settings
➤ YN0000: │ ✓ Vscode (new ✨)
➤ YN0000: └ Completed

This scans package.json for supported technologies. In the context of this project, it's ESLint, Prettier and TypeScript. For example, prettier works again in VS Code. The command prettier is not available, Instead, I would have to use yarn prettier.

tschaffter commented 2 years ago

✔️ Issue related to `/node_modules/.ngcc_lock_file`

Sometimes yarn install or yarn add ... fail.

$ yarn add -D jest-pnp-resolver
...
➤ YN0002: │ challenge@workspace:. doesn't provide styled-components (pd8d31), requested by redoc
➤ YN0002: │ challenge@workspace:. doesn't provide unzipper (p32a3b), requested by @nxrocks/nx-spring-boot
➤ YN0002: │ challenge@workspace:. doesn't provide xmlbuilder2 (p9250a), requested by @nxrocks/nx-spring-boot
➤ YN0002: │ challenge@workspace:. doesn't provide xpath (paae6e), requested by @nxrocks/nx-spring-boot
➤ YN0002: │ jest-preset-angular@npm:11.1.2 [c9049] doesn't provide jest (p9a8ff), requested by ts-jest
➤ YN0002: │ jest-preset-angular@npm:11.1.2 [c9049] doesn't provide typescript (pc596a), requested by ts-jest
➤ YN0002: │ redoc-cli@npm:0.13.20 doesn't provide core-js (p6cfa3), requested by redoc
➤ YN0002: │ redoc-cli@npm:0.13.20 doesn't provide react-is (pb4ff6), requested by styled-components
➤ YN0002: │ redoc@npm:2.0.0 [0682b] doesn't provide webpack (p1f851), requested by style-loader
➤ YN0002: │ redoc@npm:2.0.0 [c9049] doesn't provide webpack (pe2104), requested by style-loader
➤ YN0002: │ redoc@npm:2.0.0-rc.77 [98264] doesn't provide webpack (p9d479), requested by style-loader
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 0s 898ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 1s 861ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ challenge@workspace:. must be built because it never has been before or the last one failed
➤ YN0009: │ challenge@workspace:. couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-b154d8ad/build.log)
➤ YN0000: └ Completed in 2s 537ms
➤ YN0000: Failed with errors in 5s 599ms

Following the link to the log file:

# This file contains the result of Yarn building a package (challenge@workspace:.)
# Script name: postinstall

Error: EROFS: read-only filesystem, open '/node_modules/.ngcc_lock_file'
    at makeError$1 (/workspaces/challenge-registry/.pnp.cjs:36775:24)
    at EROFS (/workspaces/challenge-registry/.pnp.cjs:36802:10)
    at ZipFS.prepareWriteFile (/workspaces/challenge-registry/.pnp.cjs:38636:13)
    at ZipFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:38620:53)
    at /workspaces/challenge-registry/.pnp.cjs:39702:20
    at /workspaces/challenge-registry/.pnp.cjs:39918:60
    at ZipOpenFS.getZipSync (/workspaces/challenge-registry/.pnp.cjs:40047:14)
    at ZipOpenFS.makeCallSync (/workspaces/challenge-registry/.pnp.cjs:39918:17)
    at ZipOpenFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:39699:17)
    at VirtualFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:39032:24)

Previous discussions:

I thought that maybe the lock file was in one of Angular zip archives, but couldn't find it.

for file in .yarn/cache/*.zip; do unzip -l "$file" | grep "ngcc_lock_file"; done

The lock file is created when another Angular process is running. Here is one path where the lock file could be found in non-PnP environment (not tested): rm node_modules/@angular/compiler-cli/ngcc/__ngcc_lock_file__.

It seems that a program (nx compiler CLI?) tries to write to /node_modules/.ngcc_lock_file. While this would work in non-PnP environments, it seems that it's not possible to write to the node_modules folders that live in the zip archives managed by PnP.

Workaround

Unplugging @angular/compiler-cli solves the issue to instruct yarn to unzip the package, which will enable the lock file to be written. Information about unplugged packages is located in package.json.

Here is where the lock file would be written:

$ ls -all .yarn/unplugged/@angular-compiler-cli-virtual-a50605b143/node_modules/@angular/compiler-cli/ngcc/    
total 8
drwxr-xr-x  3 vscode vscode  57 Jun 22  1984 .
drwxr-xr-x  7 vscode vscode 182 Oct  4 15:20 ..
-rwxr-xr-x  1 vscode vscode 980 Jun 22  1984 index.d.ts
-rwxr-xr-x  1 vscode vscode  94 Jun 22  1984 main-ngcc.d.ts
drwxr-xr-x 12 vscode vscode 319 Jun 22  1984 src

tschaffter commented 2 years ago

✔️ Accessing package type definition in VS Code

Consider import { Component } from '@angular/core';. I can successfully access the type definition shown as tooltip:

However, it is not possible to open the definition file, which hinders package exploration and debugging. Note that I have already installed the Yarn SDK for VS Code.

See:

Workaround

Unplugging an npm package enables to access its code in VS Code. For example, yarn unplug @angular/core followed by closing and reopening the .ts file in VS Code, then click on @angular/core to access the index file of the library.

Note When I installed the Yarn SDK for VS Code, the VS Code extension arcanis.vscode-zipfs, which is supposed to enable VS Code to read files directly from zip archives. As of today, this extension may be used to show the type definitions as tooltip but not to access the source code of npm packages. However, a long term solution should be provided by this extension or natively by VS Code (https://github.com/microsoft/vscode/issues/75559). Unplugging individual packages when we want to explore their source code is a pain.

tschaffter commented 2 years ago

Conclusion

I'm timeboxing the exploration of PnP to one day. The short answer is that our project can not use it yet mainly because there is still a lack of support from tools like VS Code, Nx, Jest, etc. PnP requires npm packages to be thorough when defining their dependencies. I could probably resolve these dependency issues if I had time. Instead, I'll wait 3-6 months before trying again.

Setup

Here is what I did to enable PnP and try to get it to work with this Nx workspace:

Remove the folder node_modules.
Comment out nodeLinker: node-modules in .yarnrc.yml.
Run yarn install --immutable.
Unplug Ng compiler CLI to solve one of the issue: yarn unplug @angular/compiler-cli.
Install Yarn SDK for VS Code: yarn dlx @yarnpkg/sdks vscode

Unfortunately, running tests with Jest are now failing.

Other considerations

Using PnP would speedup yarn commands - locally and in CI/CD workflow - as dependencies are stored in memory and the bulky node_modules is no longer around.
PnP has a feature called "zero-install" that consists in storing npm packages as zip archives. The recommendation by the Yarn team is to push these zip archives to Git(Hub). However, git is not designed to track binary and/or large files. Even if most of the zip archives are small and do not exceed a total of a few hundreds MBs, I'm concerned about the long term decrease of performance of git as npm packages are added/removed/updated and past commits slow down git. This is especially true in the context of a monorepo that may handle a large number of dependencies.
Without PnP, it is easy to add node_modules/.bin to the path and simply run commands such as nx, serverless, etc. This approach is no longer possible with PnP.

Issues to monitor

hakimio commented 2 years ago

@tschaffter have you tried PnPify to work-around some of the issues like the jest issue?

tschaffter commented 2 years ago

@hakimio I tried yarn pnpify --sdk vscode at some point, then switched to yarn dlx @yarnpkg/sdks vscode to as recommended here.

Before you could simply yarn pnpify --sdk vscode (I think that was the old command?), however, with Yarn 3, you’ll need to use @yarnpkg/sdks in order to add needed support. This time you have two options, executable or dependency-based SDK support.

What is your experience using pnpify?

hakimio commented 2 years ago

The linked article talks about adding yarn pnp support to VS Code. I was referring to the issue you had with jest where it couldn't find external libraries (Cannot find module '@angular/common/http' or its corresponding type declarations). My idea is that it can't find them since it's trying to access node_modules directly and if you run yarn pnpify jest, pnpify would act as a proxy and allow jest find the modules. Personally I don't have much experience with pnpify. It's just an idea which might be worth a try.

arcanis commented 1 year ago

❌ Jest works with PnP.

That seems strange - Jest has supported PnP for a while now, both codebases have tests to enforce it 🤔

tschaffter commented 1 year ago

@arcanis Thanks for the info. There are still issues related to Nx-Jest according to this comment.

Sage-Bionetworks / sage-monorepo