search

0x80 / isolate-package

Isolate a monorepo package with its internal dependencies to form a self-contained directory with a pruned lockfile
MIT License
121 stars 13 forks source link

PNPM prune fails to read 'specifiers' #97

Open insanity54 opened 3 months ago

insanity54 commented 3 months ago

Hi thanks for isolate-package. I'm interested in this project, I think it could help me build docker images from my pnpm monorepo. I followed the prerequisites as best I could, but I ran into an error when I tried to isolate one of my packages.

npx isolate-package isolate                                                                        ✔  ▼ 
info Using isolate-package version 1.18.0
Error: Failed to generate lockfile: Cannot read properties of undefined (reading 'specifiers')
    at generatePnpmLockfile (file:///home/cj/.npm/_npx/27834e9986fbcd8f/node_modules/isolate-package/src/lib/lockfile/helpers/generate-pnpm-lockfile.ts:175:11)
    at processLockfile (file:///home/cj/.npm/_npx/27834e9986fbcd8f/node_modules/isolate-package/src/lib/lockfile/process-lockfile.ts:69:7)
    at isolate (file:///home/cj/.npm/_npx/27834e9986fbcd8f/node_modules/isolate-package/src/isolate.ts:175:29)
    at run (file:///home/cj/.npm/_npx/27834e9986fbcd8f/node_modules/isolate-package/src/isolate-bin.ts:9:3)

I'm not sure what the 'specifiers' thing is. I searched isolate-package source code for that string and it looks like it's maybe the package version specifier in package.json?

example: "@futurenet/scout": "workspace:^",

Curiously, isolate-package did generate a bunch of output in the isolate directory. I'm not sure if it completed it's job, though, since I haven't seen what a successful run of isolate looks like.

Not sure if it's something I did wrong with my project configuration but I figured I'd document what I came across. I think my next move is to double check that my package build process is correct. Right now, all of my packages have a build step. I read a bunch of articles and docs pages about isolate-package and I'm learning about internal packages for the first time. Maybe I will try using internal packages instead.

Another thing I'm suspicious of, maybe it's causing conflicts, is how Temporal loads it's workflows. My project integrates with Temporal and Temporal insists on using webpack to load workflow files. Normally I wouldn't use webpack at all, I'd use tsc instead. I'm not sure how that ties into everything but maybe what I'll do is see if I can re-create this error on a simpler project with fewer dependencies.

Edit 1

I made a simplified git repo and successfully used isolate-package to isolate the taco package. taco package depends on meat package and isolate-package ran without error. This points to the issue being with my code.

0x80 commented 3 months ago

@insanity54 I have just released an update that improves error logging by re-throwing the original error so you should get a full stack trace. Hope it helps.

If you can't find the problem in your codebase, and you are willing to give me access, then I could take a look.

moonrockfamily commented 2 months ago

@insanity54 I was getting the same error and created a small app to try to reproduce without success; it worked just fine! I then tried to replicate with the same scope of dependencies on a fresh directory and it still worked as expected... Hmm, very strange! The following does not identify the root cause, but helped me get unstuck. I hope it helps you.

Error output with debug logging

PS C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration> npx isolate
info Using isolate-package version 1.19.0
debug Using buildDirName from config: .
debug Workspace root resolved to C:\Users\thiles\gitrepo\nodered
debug Isolate target package (root)\app\sfUserIntegration
debug Isolate output directory (root)\app\sfUserIntegration\isolate
debug Cleaned the existing isolate output directory
debug Detected package manager pnpm 9.5.0+sha512.140036830124618d624a2187b50d04289d5a087f326c9edfc0ccd733d76c4f52c3a313d4fc148794a2a9d81553016004e6742e8cf850670268a7387fc220c903       
debug Use PNPM pack instead of NPM pack
debug Detected pnpm packages globs: [ 'app/sfUserIntegration' ]
debug Registering package app\sfUserIntegration
debug Packed (temp)/sfUserIntegration-1.0.0.tgz
debug Generating PNPM lockfile...
debug Relevant importer ids: [ 'app/sfUserIntegration' ]
debug Pruning the lockfile
error Failed to generate lockfile: Cannot read properties of undefined (reading 'specifiers')
TypeError: Cannot read properties of undefined (reading 'specifiers')
    at pruneLockfile (C:\Users\thiles\gitrepo\nodered\node_modules\.pnpm\@pnpm+prune-lockfile@6.1.4\node_modules\@pnpm\prune-lockfile\src\index.ts:56:56)
    at generatePnpmLockfile (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/lib/lockfile/helpers/generate-pnpm-lockfile.ts:145:15)
    at processLockfile (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/lib/lockfile/process-lockfile.ts:69:7)       
    at isolate (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/isolate.ts:175:29)
    at run (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/isolate-bin.ts:9:3)
PS C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration> npx isolate
info Using isolate-package version 1.19.0
debug Using buildDirName from config: .
debug Workspace root resolved to C:\Users\thiles\gitrepo\nodered
debug Isolate target package (root)\app\sfUserIntegration
debug Isolate output directory (root)\app\sfUserIntegration\isolate
debug Cleaned the existing isolate output directory
debug Detected package manager pnpm 9.5.0+sha512.140036830124618d624a2187b50d04289d5a087f326c9edfc0ccd733d76c4f52c3a313d4fc148794a2a9d81553016004e6742e8cf850670268a7387fc220c903       
debug Use PNPM pack instead of NPM pack
debug Detected pnpm packages globs: [ 'app/sfUserIntegration' ]
debug Registering package app\sfUserIntegration
debug Packed (temp)/sfUserIntegration-1.0.0.tgz
debug Generating PNPM lockfile...
debug Relevant importer ids: [ 'app/sfUserIntegration' ]
debug Pruning the lockfile
error Failed to generate lockfile: Cannot read properties of undefined (reading 'specifiers')
TypeError: Cannot read properties of undefined (reading 'specifiers')
    at pruneLockfile (C:\Users\thiles\gitrepo\nodered\node_modules\.pnpm\@pnpm+prune-lockfile@6.1.4\node_modules\@pnpm\prune-lockfile\src\index.ts:56:56)
    at generatePnpmLockfile (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/lib/lockfile/helpers/generate-pnpm-lockfile.ts:145:15)
    at processLockfile (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/lib/lockfile/process-lockfile.ts:69:7)       
    at isolate (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/isolate.ts:175:29)
    at run (file:///C:/Users/thiles/gitrepo/nodered/node_modules/.pnpm/isolate-package@1.19.0/node_modules/isolate-package/src/isolate-bin.ts:9:3)

Cleanup pnpm-lock files

The error message related to pruning the lock file gave me the hunch to remove all existing pnpm-lock.yaml files as follows:

It now produced the expected isolate output without the error! Conclusion/assumption: An old lock file was being processed

Output after Cleanup with debug logging

PS C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration> pnpm install
Scope: all 2 workspace projects
 WARN  7 deprecated subdependencies found: are-we-there-yet@2.0.0, gauge@3.0.2, glob@7.2.3, inflight@1.0.6, ldapjs@2.3.3, npmlog@5.0.1, rimraf@3.0.2
../..                                    | +631 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
../..                                    | Progress: resolved 631, reused 631, downloaded 0, added 631, done
../../node_modules/.pnpm/bcrypt@5.1.1_encoding@0.1.13/node_modules/bcrypt: Running install script, done in 593ms
../../node_modules/.pnpm/bcrypt@5.1.0_encoding@0.1.13/node_modules/bcrypt: Running install script, done in 604ms

dependencies:
+ @thiles/node-red-contrib-amqp 2.4.4
+ do-red 0.3.5 (1.0.1 is available)
+ moment 2.30.1
+ node-red 3.1.11 (4.0.2 is available)
+ node-red-contrib-activedirectory 0.3.2
+ node-red-contrib-better-ldap-servicely 0.1.7
+ node-red-contrib-countula 1.1.9
+ node-red-contrib-cron-plus 1.5.7 (2.1.0 is available)
+ node-red-contrib-flogger 1.2.2
+ node-red-contrib-flowtest 0.1.2
+ node-red-contrib-moment 4.0.0 (5.0.0 is available)
+ node-red-contrib-mongodb3 2.0.1
+ node-red-contrib-oauth2 4.0.3 (6.2.1 is available)
+ node-red-contrib-objectid 1.1.0
+ node-red-contrib-oracledb-2 0.3.3
+ node-red-contrib-powershell 0.0.3 (0.0.5 is available)
+ node-red-contrib-reusable-flows 1.0.0
+ node-red-contrib-salesforce-fix 0.8.5
+ node-red-contrib-semaphore 1.0.1
+ node-red-node-daemon 0.4.0 (0.6.0 is available)
+ oracledb 5.5.0 (6.6.0 is available)
+ xml2js 0.4.23 (0.6.2 is available)

Done in 9.3s
PS C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration> npx isolate 
info Using isolate-package version 1.19.0
debug Using buildDirName from config: .
debug Workspace root resolved to C:\Users\thiles\gitrepo\nodered
debug Isolate target package (root)\app\sfUserIntegration
debug Isolate output directory (root)\app\sfUserIntegration\isolate
debug Detected package manager pnpm 9.5.0+sha512.140036830124618d624a2187b50d04289d5a087f326c9edfc0ccd733d76c4f52c3a313d4fc148794a2a9d81553016004e6742e8cf850670268a7387fc220c903       
debug Use PNPM pack instead of NPM pack
debug Detected pnpm packages globs: [ 'app/sfUserIntegration' ]
debug Registering package app\sfUserIntegration
debug Packed (temp)/sfUserIntegration-1.0.0.tgz
debug Generating PNPM lockfile...
debug Relevant importer ids: [ 'app/sfUserIntegration' ]
debug Setting target package importer on root
debug Pruning the lockfile
debug Created lockfile at C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration\isolate\pnpm-lock.yaml
debug Copied .npmrc file to the isolate output
debug Deleting temp directory (root)\app\sfUserIntegration\isolate\__tmp
info Isolate completed at C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration\isolate
PS C:\Users\thiles\gitrepo\nodered\app\sfUserIntegration>

isolate.config.json

{
    "logLevel": "debug",
    "buildDirName": "."
}

app\sfUserIntegration\package.json with Catalog: scheme pnpm version specifiers

{
  "description": "Nodered - Flows for provisioning Georgian Users; eg. sensing Salesforce Contact's Admission Group needs",
  "name": "sfUserIntegration",
  "version": "1.0.0",
  "scripts": {
    "prestart": "nvs auto",
    "start": "node --max-old-space-size=8196 --env-file .env node_modules/node-red/red.js --userDir=. --port=1884"
  },
  "files": [
    "flows.json",
    "flows_cred.json",
    "settings.js",
    "Microsoft.PowerShell_profile.ps1",
    "pnpm-lock.yaml"
  ],
  "dependencies": {
    "@thiles/node-red-contrib-amqp": "catalog:messaging",
    "do-red": "~0.3.5",
    "moment": "catalog:essential",
    "node-red": "catalog:essential",
    "node-red-contrib-activedirectory": "^0.3.2",
    "node-red-contrib-better-ldap-servicely": "~0.1.7",
    "node-red-contrib-countula": "catalog:essential",
    "node-red-contrib-cron-plus": "catalog:essential",
    "node-red-contrib-flogger": "catalog:essential",
    "node-red-contrib-flowtest": "~0.1.2",
    "node-red-contrib-moment": "catalog:essential",
    "node-red-contrib-mongodb3": "catalog:db",
    "node-red-contrib-oauth2": "catalog:api",
    "node-red-contrib-objectid": "~1.1.0",
    "node-red-contrib-oracledb-2": "catalog:db",
    "node-red-contrib-powershell": "0.0.3",
    "node-red-contrib-reusable-flows": "^1.0.0",
    "node-red-contrib-salesforce-fix": "catalog:api",
    "node-red-contrib-semaphore": "~1.0.1",
    "node-red-node-daemon": "~0.4.0",
    "oracledb": "catalog:db",
    "xml2js": "^0.4.23"
  }
}
0x80 commented 2 months ago

@moonrockfamily did you have lockfiles in packages folders (outside of the isolate directories)? I think in a monorepo setup there should be only one ever, in the root.

Maybe there's some sort of version mismatch going on. I can imagine that if the lockfile was created using an older version, and isolate-package detects a newer pnpm version, it might invoke the newer prune-lockfile version without respecting the version recorded in the lockfile itself.

But I'm just speculating here :)

0x80 commented 2 months ago

@moonrockfamily @insanity54 If you still have a way to reproduce your error, please give 1.20.0-0 a try and let me know if it solved things. You can install it with @next

This version now respects/preserves the actual lockfile version when pruning, instead of going by the declared version in the packageManager field of the manifest.

moonrockfamily commented 2 months ago

Thanks @0x80!

0x80 commented 2 months ago

@moonrockfamily have you been able to test it on your initial problem?

I have been using 12.0.0-0 in my own projects and it seems at least it didn't break anything so I will release it soon. Wish I had some proper tests to cover this 👀

moonrockfamily commented 2 months ago

I tried to reproduce with 1.19 but couldn't find the recipe to make a reproducible failure!

insanity54 commented 2 months ago

I can still reproduce the error on 1.19.0 and 1.20.0-0. I made a reproduction repo

https://github.com/insanity54/isolate-issue-97

0x80 commented 2 months ago

Thanks @insanity54 I will get back to it in a week or two to see if I can find the real cause then