[turborepo] [yarn 3] pruned lockfile missing resolution entries

erj826 commented 2 years ago

What version of Turborepo are you using?

1.6.3

What package manager are you using / does the bug impact?

Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Mac

Describe the Bug

To optimize our Docker builds we are attempting to use turbo prune to create a lighter weight image. After running prune we copy our pruned lockfile and .json files into a directory and then attempt to run an immutable install.

We have observed that when the root package.json includes yarn resolutions we are unable to run yarn --immutable successfully in the pruned monorepo.

For example

In this reproduction ajv is a dependency of eslint so it is a necessary dependency in the pruned monorepo.

ericjacobson@Erics-MacBook-Air pruned % yarn why ajv
├─ @eslint/eslintrc@npm:0.4.3
│  └─ ajv@npm:6.12.6 (via npm:^6.12.4)
│
├─ eslint@npm:7.32.0
│  └─ ajv@npm:6.12.6 (via npm:^6.10.0)
│
└─ table@npm:6.8.1
   └─ ajv@npm:8.11.2 (via npm:^8.0.1)

When the resolution is not included in the root package.json the generated out/yarn.lock is correct, and includes an ajv entry.

When the resolution is included in the root package.json then generated out/yarn.lock is incorrect, and does not include any ajv entries.

Expected Behavior

The pruned lockfile should include dependencies that have a corresponding yarn resolution.

To Reproduce

See the README.md in the reproduction repo for more details.

Reproduction Repo

https://github.com/erj826/turbo-prune-bug-repro-11-21

quinnturner commented 1 year ago

This was first mentioned in my ticket in #2049. I will close my version as this ticket is more scoped. CC: @chris-olszewski, since you're assigned to that ticket, perhaps you intend to assign yourself to this ticket?

alexbchr commented 1 year ago

Any update on this? I don't see much activity happening on this issue but I have the exact same problem in our app and this is quite blocking.

erj826 commented 1 year ago

Looks like this work might be in progress. https://github.com/vercel/turbo/pull/4437

@chris-olszewski do you still expect this to be fixed in the 1.9 release?

chris-olszewski commented 1 year ago

It should be released as part of a canary next week.

chris-olszewski commented 1 year ago

Hello all, if everyone could test the latest canary (1.9.4-canary.5) I would appreciate it a ton! This issue has been one edge case after another and I think I've covered them all, but testing this against more projects would be great!

quinnturner commented 1 year ago

Hi @chris-olszewski, thank you for the progress on this item! In my large monorepo, there is, unfortunately, still some drift. However, it's gotten much closer than it was before. I will see what I can do to share a private repo.

Also, for others reading this, the release is actually 1.9.4-canary.5 for other viewers, not 1.9.5-canary.5 😄

quinnturner commented 1 year ago

One edge case:

// package/a
"dependencies": {
 "@project/b": "*"
}
// package/c
"dependencies": {
  "@project/b": "workspace:*"
}

with yarn.lock

"@project/b@*, @project/b@workspace:*":

Results in

thread '<unnamed>' panicked at 'Descriptor collision @project/b@workspace:* and *', crates/turborepo-lockfiles/src/berry/mod.rs:124:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
SIGABRT: abort
PC=0x7ff81b59a22a m=16 sigcode=0
signal arrived during cgo execution

while a regular yarn install works.

erj826 commented 1 year ago

Running into another edge case when running an install after pruning (yarn 3.5):

Dockerfile

```dockerfile FROM node:16.20.0 AS base RUN mkdir /home/node/app FROM base AS pruner WORKDIR /home/node/app COPY . . RUN yarn --quiet dlx turbo@1.9.4-canary.5 prune --scope= --docker FROM base AS installer WORKDIR /home/node/app COPY --from=pruner /home/node/app/out/json/ . COPY --from=pruner /home/node/app/out/yarn.lock ./yarn.lock COPY --from=pruner /home/node/app/.yarnrc.yml ./.yarnrc.yml COPY --from=pruner /home/node/app/.yarn ./.yarn RUN CYPRESS_INSTALL_BINARY=0 PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 yarn --immutable ```

resolve@patch:resolve@^1.5.0 and resolve@patch:resolve@^1.9.0 are transitive dependencies for another app within our monorepo that are not used by the target app.turbo prune seems to have left them in the lockfile which is leading to the immutable install failing.

I can try to spin up a reproduction repo when I get some time.

chris-olszewski commented 1 year ago

Thanks for testing this out!

@quinnturner

In my large monorepo, there is, unfortunately, still some drift

Are there entire entries missing or is it primarily drift between the keys?

One edge case

I see the issue here, should have a fix up soon.

@erj826 If that's the only difference, then that points to our inference of the compatibility patches being incorrect. Will be taking a look at this today.

chris-olszewski commented 1 year ago

@quinnturner and @erj826 I believe #4755 and #4770 should address the issues you both encountered. If you could both try out 1.9.4-canary.7 hopefully the pruned lockfiles are correct (or at least closer).

quinnturner commented 1 year ago

Hi @chris-olszewski, I just tested. It's closer; amazing job! There is still precisely one issue in my monorepo. It removed a reference to @babel/plugin-proposal-private-property-in-object@npm:^7.16.0 when it should have kept it.

-"@babel/plugin-proposal-private-property-in-object@npm:^7.16.0, @babel/plugin-proposal-private-property-in-object@npm:^7.21.0":
+"@babel/plugin-proposal-private-property-in-object@npm:^7.21.0":
   version: 7.21.0
   resolution: "@babel/plugin-proposal-private-property-in-object@npm:7.21.0"
   dependencies:

I have updated my private repository with the latest canary. You should have access to it; I added you as a contributor last week 😄

What's confusing is that the reference to @babel/plugin-proposal-private-property-in-object@npm:^7.16.0 doesn't exist anywhere in the yarn.lock. It only exists in the yarn-3.5.0.cjs bundled with the repository. If that's the level of complexity you have to deal with, I am sorry... lol.

quinnturner commented 1 year ago

In either case, this issue, as described, seems to be resolved. If there is something to do with yarn-*.cjs that populates the yarn.lock, that should be created in a new GitHub issue

chris-olszewski commented 1 year ago

Sorry, I missed that invite. Thanks for testing this out, I'm so excited that we're this close. I'll try to get that fixed up ASAP, I at least know where we're doing something wrong. (That reference is injected via a default plugin that ships as part of the yarn binary

chris-olszewski commented 1 year ago

@quinnturner A partial work around you can poke around with while I continue to poke at this would be to add "@babel/plugin-proposal-private-property-in-object": "^7.16.0" as a dependency to every workspace that uses babel-preset-react-app.

erj826 commented 1 year ago

@chris-olszewski thank you for the quick turnaround! Prune appears to be working without any issues for our monorepo using 1.9.4-canary.7!

JuanJo4 commented 1 year ago

Hi, i'm not sure if the following is related to this same issue but seems to me that there are some similarities.

I have to patch a package in my project and i'm using yarn patch to do that (yarn v3), when i do turbo prune the package.json file in the /out folder doesn't have the resolution with the patch. e.g. "package-to-patch@x.x.x": "patch:package-to-patch@x.x.x#.yarn/patches/package-to-patch@x.x.x-0608291d42.patch", it does have other resolutions but not the one that has the patch. The same happens with the .lock file, locally i see the package with the patch in the lock file but then in the /out i'm not seeing that.

is this the same issue or a different one? happy to provide more information if needed or any help to troubleshoot this.

cimabe commented 1 year ago

Only downgrading from 1.9.8 to 1.9.3 worked for me

Intevel commented 1 year ago

Only downgrading from 1.9.8 to 1.9.3 worked for me

Same for me

alex-dixon commented 1 year ago

Not sure if this helps but I ran into this as well on 1.10.0:

 > [stage-0  8/11] RUN --mount=type=secret,mode=0644,id=yarnrc,target=/home/node/.yarnrc.yml yarn install --immutable:                                                                                             
#0 1.005 ➤ YN0000: ┌ Resolution step                                                                                                                                                                               
#0 1.170 ➤ YN0000: └ Completed                                                                                                                                                                                     
#0 1.191                                                                                                                                                                                                           
#0 1.191 ➤ YN0000: ┌ Post-resolution validation                                                                                                                                                                    
#0 1.191 ➤ YN0000: │ @@ -4,19 +4,8 @@
#0 1.191 ➤ YN0000: │  __metadata:
#0 1.191 ➤ YN0000: │    version: 6
#0 1.191 ➤ YN0000: │    cacheKey: 8
#0 1.191 ➤ YN0000: │  
#0 1.191 ➤ YN0028: │ -"@babel/types@npm:^7.8.3":
#0 1.192 ➤ YN0028: │ -  version: 7.22.4
#0 1.192 ➤ YN0028: │ -  resolution: "@babel/types@npm:7.22.4"
#0 1.192 ➤ YN0028: │ -  dependencies:
#0 1.192 ➤ YN0028: │ -    "@babel/helper-string-parser": ^7.21.5
#0 1.193 ➤ YN0028: │ -    "@babel/helper-validator-identifier": ^7.19.1
#0 1.193 ➤ YN0028: │ -    to-fast-properties: ^2.0.0
#0 1.193 ➤ YN0028: │ -  checksum: ffe36bb4f4a99ad13c426a98c3b508d70736036cae4e471d9c862e3a579847ed4f480686af0fce2633f6f7c0f0d3bf02da73da36e7edd3fde0b2061951dcba9a
#0 1.193 ➤ YN0028: │ -  languageName: node
#0 1.193 ➤ YN0028: │ -  linkType: hard
#0 1.193 ➤ YN0028: │ -
#0 1.193 ➤ YN0000: │  "@cspotcode/source-map-support@npm:^0.8.0":
#0 1.193 ➤ YN0000: │    version: 0.8.1
#0 1.193 ➤ YN0000: │    resolution: "@cspotcode/source-map-support@npm:0.8.1"
#0 1.193 ➤ YN0000: │    dependencies:
#0 1.193 ➤ YN0000: │ 
#0 1.194 ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
#0 1.194 ➤ YN0000: └ Completed
#0 1.194 ➤ YN0000: Failed with errors in 0s 192ms
------

When I remove it the install succeeds.

ydhn commented 1 year ago

I have the same problem with alex-dixon in version 1.10.12. After I ran turbo prune --docker and tried to yarn install --immutable, this error occurred:

➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0s 423ms

➤ YN0000: ┌ Post-resolution validation
➤ YN0000: │ @@ -13,19 +13,8 @@
➤ YN0000: │    checksum: 09b7a97a05c80540db6c9e4ddf8c5d2ebb06cae5caf3a87e33c33f27f8c4d49d9c67a2d72f1570e796045288fad569f98a26ceba0c4f5fad2af84b6ad855c4fb
➤ YN0000: │    languageName: node
➤ YN0000: │    linkType: hard
➤ YN0000: │
➤ YN0028: │ -"@babel/types@npm:^7.8.3":
➤ YN0028: │ -  version: 7.20.7
➤ YN0028: │ -  resolution: "@babel/types@npm:7.20.7"
➤ YN0028: │ -  dependencies:
➤ YN0028: │ -    "@babel/helper-string-parser": ^7.19.4
➤ YN0028: │ -    "@babel/helper-validator-identifier": ^7.19.1
➤ YN0028: │ -    to-fast-properties: ^2.0.0
➤ YN0028: │ -  checksum: b39af241f0b72bba67fd6d0d23914f6faec8c0eba8015c181cbd5ea92e59fc91a52a1ab490d3520c7dbd19ddb9ebb76c476308f6388764f16d8201e37fae6811
➤ YN0028: │ -  languageName: node
➤ YN0028: │ -  linkType: hard
➤ YN0028: │ -
➤ YN0000: │  "@eslint/eslintrc@npm:^1.4.1":
➤ YN0000: │    version: 1.4.1
➤ YN0000: │    resolution: "@eslint/eslintrc@npm:1.4.1"
➤ YN0000: │    dependencies:
➤ YN0000: │ @@ -224,8 +213,22 @@
➤ YN0000: │    checksum: e009a2bfb50e90ca9b7c6e8f648f8464067271fd99116f881073fa6fa76dc8d0133181dd65e6614d5fb1220d671d67b0124aef7d97dc02d7e342ab143a47779d
➤ YN0000: │    languageName: node
➤ YN0000: │    linkType: hard
➤ YN0000: │
➤ YN0028: │ +"@types/node@npm:*":
➤ YN0028: │ +  version: 20.5.7
➤ YN0028: │ +  resolution: "@types/node@npm:20.5.7"
➤ YN0028: │ +  languageName: node
➤ YN0028: │ +  linkType: hard
➤ YN0028: │ +
➤ YN0028: │ +"@types/responselike@npm:^1.0.0":
➤ YN0028: │ +  version: 1.0.0
➤ YN0028: │ +  resolution: "@types/responselike@npm:1.0.0"
➤ YN0028: │ +  dependencies:
➤ YN0028: │ +    "@types/node": "*"
➤ YN0028: │ +  languageName: node
➤ YN0028: │ +  linkType: hard
➤ YN0028: │ +
➤ YN0000: │  "@types/semver@npm:^7.3.12":
➤ YN0000: │    version: 7.3.13
➤ YN0000: │    resolution: "@types/semver@npm:7.3.13"
➤ YN0000: │    checksum: 00c0724d54757c2f4bc60b5032fe91cda6410e48689633d5f35ece8a0a66445e3e57fa1d6e07eb780f792e82ac542948ec4d0b76eb3484297b79bd18b8cf1cb0
➤ YN0000: │
➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 442ms

martin-eq commented 5 months ago

Any updates on this issue?

I'm using latest Yarn 4.2.2 and Turborepo 2.0.1, trying to run turbo prune --docker and yarn install --immutable to create a Docker image for my app and I'm getting this error:

 > [deps 6/6] RUN yarn install --immutable:                                                                                                                                                                                 
0.319 ➤ YN0000: · Yarn 4.2.2                                                                                                                                                                                                
0.323 ➤ YN0000: ┌ Resolution step
0.398 ➤ YN0085: │ - @babel/types@npm:7.24.6
0.405 ➤ YN0000: └ Completed
0.405 ➤ YN0000: ┌ Post-resolution validation
0.405 ➤ YN0060: │ @typescript-eslint/eslint-plugin is listed by your project with version 7.12.0, which doesn't satisfy what eslint-plugin-jest (p04867) requests (^5.0.0).
0.406 ➤ YN0002: │ prettier-config@workspace:packages/prettier-config doesn't provide prettier (p81cce), requested by prettier-plugin-tailwindcss.
0.406 ➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
0.429 ➤ YN0000: │ 
0.429 ➤ YN0000: │ @@ -82,19 +82,8 @@
0.429 ➤ YN0000: │    checksum: 10/6c4e12731cd9206a883c19d48fa04f6aaaf7ee83f049b22631e6521b866edc20832b4d5db30aa86d8ae799c4dcf57761fe8a4af2bf7e233245c079c1dafb5668
0.429 ➤ YN0000: │    languageName: node
0.429 ➤ YN0000: │    linkType: hard
0.430 ➤ YN0000: │  
0.430 ➤ YN0028: │ -"@babel/types@npm:^7.8.3":
0.430 ➤ YN0028: │ -  version: 7.24.6
0.430 ➤ YN0028: │ -  resolution: "@babel/types@npm:7.24.6"
0.430 ➤ YN0028: │ -  dependencies:
0.430 ➤ YN0028: │ -    "@babel/helper-string-parser": "npm:^7.24.6"
0.430 ➤ YN0028: │ -    "@babel/helper-validator-identifier": "npm:^7.24.6"
0.430 ➤ YN0028: │ -    to-fast-properties: "npm:^2.0.0"
0.430 ➤ YN0028: │ -  checksum: 10/34552539cdc740513650cb3c7754f77a55cc5253dff9d45afd52292d366eb1c099939d5db066e458abcf4c9a7dedfe43467445f9c2208b3cb64866762dee5e9d
0.430 ➤ YN0028: │ -  languageName: node
0.431 ➤ YN0028: │ -  linkType: hard
0.431 ➤ YN0028: │ -
0.431 ➤ YN0000: │  "@bcoe/v8-coverage@npm:^0.2.3":
0.431 ➤ YN0000: │    version: 0.2.3
0.431 ➤ YN0000: │    resolution: "@bcoe/v8-coverage@npm:0.2.3"
0.431 ➤ YN0000: │    checksum: 10/1a1f0e356a3bb30b5f1ced6f79c413e6ebacf130421f15fac5fcd8be5ddf98aedb4404d7f5624e3285b700e041f9ef938321f3ca4d359d5b716f96afa120d88d
0.431 ➤ YN0000: │ 
0.431 ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
0.431 ➤ YN0000: └ Completed
0.431 ➤ YN0000: · Failed with errors in 0s 113ms

AmazingTurtle commented 4 months ago

Still a thing on turbo@2.0.9 unfortunately. Any known workarounds perhaps? Possibly just opting out of turbo for docker builds now. CI servers already on fire lmao

oasisMystre commented 2 months ago

Hey, some packages are missing when pruned for docker. I think this still not fixed lol

vercel / turborepo