[turborepo] [yarn 3] pruned lockfile missing resolution entries

[erj826]

What version of Turborepo are you using?

1.6.3

What package manager are you using / does the bug impact?

Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Mac

Describe the Bug

To optimize our Docker builds we are attempting to use turbo prune to create a lighter weight image. After running prune we copy our pruned lockfile and .json files into a directory and then attempt to run an immutable install.

We have observed that when the root package.json includes yarn resolutions we are unable to run yarn --immutable successfully in the pruned monorepo.

For example

In this reproduction ajv is a dependency of eslint so it is a necessary dependency in the pruned monorepo.

ericjacobson@Erics-MacBook-Air pruned % yarn why ajv
├─ @eslint/eslintrc@npm:0.4.3
│  └─ ajv@npm:6.12.6 (via npm:^6.12.4)
│
├─ eslint@npm:7.32.0
│  └─ ajv@npm:6.12.6 (via npm:^6.10.0)
│
└─ table@npm:6.8.1
   └─ ajv@npm:8.11.2 (via npm:^8.0.1)

When the resolution is not included in the root package.json the generated out/yarn.lock is correct, and includes an ajv entry.

When the resolution is included in the root package.json then generated out/yarn.lock is incorrect, and does not include any ajv entries.

Expected Behavior

The pruned lockfile should include dependencies that have a corresponding yarn resolution.

To Reproduce

See the README.md in the reproduction repo for more details.

Reproduction Repo

https://github.com/erj826/turbo-prune-bug-repro-11-21

[quinnturner]

This was first mentioned in my ticket in #2049. I will close my version as this ticket is more scoped. CC: @chris-olszewski, since you're assigned to that ticket, perhaps you intend to assign yourself to this ticket?

[alexbchr]

Any update on this? I don't see much activity happening on this issue but I have the exact same problem in our app and this is quite blocking.

[erj826]

Looks like this work might be in progress. https://github.com/vercel/turbo/pull/4437

@chris-olszewski do you still expect this to be fixed in the 1.9 release?

[chris-olszewski]

It should be released as part of a canary next week.

[chris-olszewski]

Hello all, if everyone could test the latest canary (1.9.4-canary.5) I would appreciate it a ton! This issue has been one edge case after another and I think I've covered them all, but testing this against more projects would be great!

[quinnturner]

Hi @chris-olszewski, thank you for the progress on this item! In my large monorepo, there is, unfortunately, still some drift. However, it's gotten much closer than it was before. I will see what I can do to share a private repo.

Also, for others reading this, the release is actually 1.9.4-canary.5 for other viewers, not 1.9.5-canary.5 😄

[quinnturner]

One edge case:

// package/a
"dependencies": {
 "@project/b": "*"
}
// package/c
"dependencies": {
  "@project/b": "workspace:*"
}

with yarn.lock

"@project/b@*, @project/b@workspace:*":

Results in

thread '<unnamed>' panicked at 'Descriptor collision @project/b@workspace:* and *', crates/turborepo-lockfiles/src/berry/mod.rs:124:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
SIGABRT: abort
PC=0x7ff81b59a22a m=16 sigcode=0
signal arrived during cgo execution

while a regular yarn install works.

[erj826]

Running into another edge case when running an install after pruning (yarn 3.5):

Dockerfile

```dockerfile FROM node:16.20.0 AS base RUN mkdir /home/node/app FROM base AS pruner WORKDIR /home/node/app COPY . . RUN yarn --quiet dlx turbo@1.9.4-canary.5 prune --scope= --docker FROM base AS installer WORKDIR /home/node/app COPY --from=pruner /home/node/app/out/json/ . COPY --from=pruner /home/node/app/out/yarn.lock ./yarn.lock COPY --from=pruner /home/node/app/.yarnrc.yml ./.yarnrc.yml COPY --from=pruner /home/node/app/.yarn ./.yarn RUN CYPRESS_INSTALL_BINARY=0 PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 yarn --immutable ```

resolve@patch:resolve@^1.5.0 and resolve@patch:resolve@^1.9.0 are transitive dependencies for another app within our monorepo that are not used by the target app.turbo prune seems to have left them in the lockfile which is leading to the immutable install failing.

I can try to spin up a reproduction repo when I get some time.

[chris-olszewski]

Thanks for testing this out!

@quinnturner

In my large monorepo, there is, unfortunately, still some drift

Are there entire entries missing or is it primarily drift between the keys?

One edge case

I see the issue here, should have a fix up soon.

@erj826 If that's the only difference, then that points to our inference of the compatibility patches being incorrect. Will be taking a look at this today.

[chris-olszewski]

@quinnturner and @erj826 I believe #4755 and #4770 should address the issues you both encountered. If you could both try out 1.9.4-canary.7 hopefully the pruned lockfiles are correct (or at least closer).

[quinnturner]

Hi @chris-olszewski, I just tested. It's closer; amazing job! There is still precisely one issue in my monorepo. It removed a reference to @babel/plugin-proposal-private-property-in-object@npm:^7.16.0 when it should have kept it.

-"@babel/plugin-proposal-private-property-in-object@npm:^7.16.0, @babel/plugin-proposal-private-property-in-object@npm:^7.21.0":
+"@babel/plugin-proposal-private-property-in-object@npm:^7.21.0":
   version: 7.21.0
   resolution: "@babel/plugin-proposal-private-property-in-object@npm:7.21.0"
   dependencies:

I have updated my private repository with the latest canary. You should have access to it; I added you as a contributor last week 😄

What's confusing is that the reference to @babel/plugin-proposal-private-property-in-object@npm:^7.16.0 doesn't exist anywhere in the yarn.lock. It only exists in the yarn-3.5.0.cjs bundled with the repository. If that's the level of complexity you have to deal with, I am sorry... lol.

[quinnturner]

In either case, this issue, as described, seems to be resolved. If there is something to do with yarn-*.cjs that populates the yarn.lock, that should be created in a new GitHub issue

[chris-olszewski]

Sorry, I missed that invite. Thanks for testing this out, I'm so excited that we're this close. I'll try to get that fixed up ASAP, I at least know where we're doing something wrong. (That reference is injected via a default plugin that ships as part of the yarn binary

[chris-olszewski]

@quinnturner A partial work around you can poke around with while I continue to poke at this would be to add "@babel/plugin-proposal-private-property-in-object": "^7.16.0" as a dependency to every workspace that uses babel-preset-react-app.

[erj826]

@chris-olszewski thank you for the quick turnaround! Prune appears to be working without any issues for our monorepo using 1.9.4-canary.7!

[JuanJo4]

Hi, i'm not sure if the following is related to this same issue but seems to me that there are some similarities.

I have to patch a package in my project and i'm using yarn patch to do that (yarn v3), when i do turbo prune the package.json file in the /out folder doesn't have the resolution with the patch. e.g. "package-to-patch@x.x.x": "patch:package-to-patch@x.x.x#.yarn/patches/package-to-patch@x.x.x-0608291d42.patch", it does have other resolutions but not the one that has the patch. The same happens with the .lock file, locally i see the package with the patch in the lock file but then in the /out i'm not seeing that.

is this the same issue or a different one? happy to provide more information if needed or any help to troubleshoot this.

[cimabe]

Only downgrading from 1.9.8 to 1.9.3 worked for me

[Intevel]

Only downgrading from 1.9.8 to 1.9.3 worked for me

Same for me

[alex-dixon]

Not sure if this helps but I ran into this as well on 1.10.0:

 > [stage-0  8/11] RUN --mount=type=secret,mode=0644,id=yarnrc,target=/home/node/.yarnrc.yml yarn install --immutable:                                                                                             
#0 1.005 ➤ YN0000: ┌ Resolution step                                                                                                                                                                               
#0 1.170 ➤ YN0000: └ Completed                                                                                                                                                                                     
#0 1.191                                                                                                                                                                                                           
#0 1.191 ➤ YN0000: ┌ Post-resolution validation                                                                                                                                                                    
#0 1.191 ➤ YN0000: │ @@ -4,19 +4,8 @@
#0 1.191 ➤ YN0000: │  __metadata:
#0 1.191 ➤ YN0000: │    version: 6
#0 1.191 ➤ YN0000: │    cacheKey: 8
#0 1.191 ➤ YN0000: │  
#0 1.191 ➤ YN0028: │ -"@babel/types@npm:^7.8.3":
#0 1.192 ➤ YN0028: │ -  version: 7.22.4
#0 1.192 ➤ YN0028: │ -  resolution: "@babel/types@npm:7.22.4"
#0 1.192 ➤ YN0028: │ -  dependencies:
#0 1.192 ➤ YN0028: │ -    "@babel/helper-string-parser": ^7.21.5
#0 1.193 ➤ YN0028: │ -    "@babel/helper-validator-identifier": ^7.19.1
#0 1.193 ➤ YN0028: │ -    to-fast-properties: ^2.0.0
#0 1.193 ➤ YN0028: │ -  checksum: ffe36bb4f4a99ad13c426a98c3b508d70736036cae4e471d9c862e3a579847ed4f480686af0fce2633f6f7c0f0d3bf02da73da36e7edd3fde0b2061951dcba9a
#0 1.193 ➤ YN0028: │ -  languageName: node
#0 1.193 ➤ YN0028: │ -  linkType: hard
#0 1.193 ➤ YN0028: │ -
#0 1.193 ➤ YN0000: │  "@cspotcode/source-map-support@npm:^0.8.0":
#0 1.193 ➤ YN0000: │    version: 0.8.1
#0 1.193 ➤ YN0000: │    resolution: "@cspotcode/source-map-support@npm:0.8.1"
#0 1.193 ➤ YN0000: │    dependencies:
#0 1.193 ➤ YN0000: │ 
#0 1.194 ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
#0 1.194 ➤ YN0000: └ Completed
#0 1.194 ➤ YN0000: Failed with errors in 0s 192ms
------

When I remove it the install succeeds.

[ydhn]

I have the same problem with alex-dixon in version 1.10.12. After I ran turbo prune --docker and tried to yarn install --immutable, this error occurred:

➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0s 423ms

➤ YN0000: ┌ Post-resolution validation
➤ YN0000: │ @@ -13,19 +13,8 @@
➤ YN0000: │    checksum: 09b7a97a05c80540db6c9e4ddf8c5d2ebb06cae5caf3a87e33c33f27f8c4d49d9c67a2d72f1570e796045288fad569f98a26ceba0c4f5fad2af84b6ad855c4fb
➤ YN0000: │    languageName: node
➤ YN0000: │    linkType: hard
➤ YN0000: │
➤ YN0028: │ -"@babel/types@npm:^7.8.3":
➤ YN0028: │ -  version: 7.20.7
➤ YN0028: │ -  resolution: "@babel/types@npm:7.20.7"
➤ YN0028: │ -  dependencies:
➤ YN0028: │ -    "@babel/helper-string-parser": ^7.19.4
➤ YN0028: │ -    "@babel/helper-validator-identifier": ^7.19.1
➤ YN0028: │ -    to-fast-properties: ^2.0.0
➤ YN0028: │ -  checksum: b39af241f0b72bba67fd6d0d23914f6faec8c0eba8015c181cbd5ea92e59fc91a52a1ab490d3520c7dbd19ddb9ebb76c476308f6388764f16d8201e37fae6811
➤ YN0028: │ -  languageName: node
➤ YN0028: │ -  linkType: hard
➤ YN0028: │ -
➤ YN0000: │  "@eslint/eslintrc@npm:^1.4.1":
➤ YN0000: │    version: 1.4.1
➤ YN0000: │    resolution: "@eslint/eslintrc@npm:1.4.1"
➤ YN0000: │    dependencies:
➤ YN0000: │ @@ -224,8 +213,22 @@
➤ YN0000: │    checksum: e009a2bfb50e90ca9b7c6e8f648f8464067271fd99116f881073fa6fa76dc8d0133181dd65e6614d5fb1220d671d67b0124aef7d97dc02d7e342ab143a47779d
➤ YN0000: │    languageName: node
➤ YN0000: │    linkType: hard
➤ YN0000: │
➤ YN0028: │ +"@types/node@npm:*":
➤ YN0028: │ +  version: 20.5.7
➤ YN0028: │ +  resolution: "@types/node@npm:20.5.7"
➤ YN0028: │ +  languageName: node
➤ YN0028: │ +  linkType: hard
➤ YN0028: │ +
➤ YN0028: │ +"@types/responselike@npm:^1.0.0":
➤ YN0028: │ +  version: 1.0.0
➤ YN0028: │ +  resolution: "@types/responselike@npm:1.0.0"
➤ YN0028: │ +  dependencies:
➤ YN0028: │ +    "@types/node": "*"
➤ YN0028: │ +  languageName: node
➤ YN0028: │ +  linkType: hard
➤ YN0028: │ +
➤ YN0000: │  "@types/semver@npm:^7.3.12":
➤ YN0000: │    version: 7.3.13
➤ YN0000: │    resolution: "@types/semver@npm:7.3.13"
➤ YN0000: │    checksum: 00c0724d54757c2f4bc60b5032fe91cda6410e48689633d5f35ece8a0a66445e3e57fa1d6e07eb780f792e82ac542948ec4d0b76eb3484297b79bd18b8cf1cb0
➤ YN0000: │
➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 442ms

[martin-eq]

Any updates on this issue?

I'm using latest Yarn 4.2.2 and Turborepo 2.0.1, trying to run turbo prune --docker and yarn install --immutable to create a Docker image for my app and I'm getting this error:

 > [deps 6/6] RUN yarn install --immutable:                                                                                                                                                                                 
0.319 ➤ YN0000: · Yarn 4.2.2                                                                                                                                                                                                
0.323 ➤ YN0000: ┌ Resolution step
0.398 ➤ YN0085: │ - @babel/types@npm:7.24.6
0.405 ➤ YN0000: └ Completed
0.405 ➤ YN0000: ┌ Post-resolution validation
0.405 ➤ YN0060: │ @typescript-eslint/eslint-plugin is listed by your project with version 7.12.0, which doesn't satisfy what eslint-plugin-jest (p04867) requests (^5.0.0).
0.406 ➤ YN0002: │ prettier-config@workspace:packages/prettier-config doesn't provide prettier (p81cce), requested by prettier-plugin-tailwindcss.
0.406 ➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
0.429 ➤ YN0000: │ 
0.429 ➤ YN0000: │ @@ -82,19 +82,8 @@
0.429 ➤ YN0000: │    checksum: 10/6c4e12731cd9206a883c19d48fa04f6aaaf7ee83f049b22631e6521b866edc20832b4d5db30aa86d8ae799c4dcf57761fe8a4af2bf7e233245c079c1dafb5668
0.429 ➤ YN0000: │    languageName: node
0.429 ➤ YN0000: │    linkType: hard
0.430 ➤ YN0000: │  
0.430 ➤ YN0028: │ -"@babel/types@npm:^7.8.3":
0.430 ➤ YN0028: │ -  version: 7.24.6
0.430 ➤ YN0028: │ -  resolution: "@babel/types@npm:7.24.6"
0.430 ➤ YN0028: │ -  dependencies:
0.430 ➤ YN0028: │ -    "@babel/helper-string-parser": "npm:^7.24.6"
0.430 ➤ YN0028: │ -    "@babel/helper-validator-identifier": "npm:^7.24.6"
0.430 ➤ YN0028: │ -    to-fast-properties: "npm:^2.0.0"
0.430 ➤ YN0028: │ -  checksum: 10/34552539cdc740513650cb3c7754f77a55cc5253dff9d45afd52292d366eb1c099939d5db066e458abcf4c9a7dedfe43467445f9c2208b3cb64866762dee5e9d
0.430 ➤ YN0028: │ -  languageName: node
0.431 ➤ YN0028: │ -  linkType: hard
0.431 ➤ YN0028: │ -
0.431 ➤ YN0000: │  "@bcoe/v8-coverage@npm:^0.2.3":
0.431 ➤ YN0000: │    version: 0.2.3
0.431 ➤ YN0000: │    resolution: "@bcoe/v8-coverage@npm:0.2.3"
0.431 ➤ YN0000: │    checksum: 10/1a1f0e356a3bb30b5f1ced6f79c413e6ebacf130421f15fac5fcd8be5ddf98aedb4404d7f5624e3285b700e041f9ef938321f3ca4d359d5b716f96afa120d88d
0.431 ➤ YN0000: │ 
0.431 ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
0.431 ➤ YN0000: └ Completed
0.431 ➤ YN0000: · Failed with errors in 0s 113ms