[Feature] File protocol should not include gitignored / npmignored files

[borekb]

• [ ] I'd be willing to implement this feature (contributing guide)
• [x] This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

I have an app that depends on lib via the file: protocol:

// app/package.json
{
  "dependencies": {
    "lib": "file:../lib"
  }
}

In app/yarn.lock, I'll get something like this after running yarn:

"lib@file:../lib::locator=root-workspace-0b6124%40workspace%3A.":
  version: 0.0.0
  resolution: "lib@file:../lib#../lib::hash=faeeac&locator=root-workspace-0b6124%40workspace%3A."
  dependencies:
    lodash: 4.17.21
  checksum: 1bb9baca9fcaed71f740e7c5a68af31e2d2e711764cd6c7f374b89f8a8e36ac62ba257caf469b28e09ceab5f9c17afe9d4beee22902cfe6773c6992bbe54e760
  languageName: node
  linkType: hard

The problem is that the hash and a checksum in the lockfile are calculated from the a directory as a whole, including files like .DS_Store, node_modules or any gitignored files. Changing any of those will produce a new "folderHash" and a checksum:

"lib@file:../lib::locator=root-workspace-0b6124%40workspace%3A.":
  version: 0.0.0
- resolution: "lib@file:../lib#../lib::hash=faeeac&locator=root-workspace-0b6124%40workspace%3A."
+ resolution: "lib@file:../lib#../lib::hash=977560&locator=root-workspace-0b6124%40workspace%3A."
  dependencies:
    lodash: 4.17.21
- checksum: 1bb9baca9fcaed71f740e7c5a68af31e2d2e711764cd6c7f374b89f8a8e36ac62ba257caf469b28e09ceab5f9c17afe9d4beee22902cfe6773c6992bbe54e760
+ checksum: db4f858cfc4305daa6174d77b8a59aded15e3959d018c60b7136d3f3f430b50c1a1a51b65db5ae77de124b2a39b4aac04db1c6a5bd3b3097cc7c830273752f74
  languageName: node
  linkType: hard

In practice, we see a changed yarn.lock all the time and need to revert it manually. We wish these changes in gitignored / npmignored files didn't change the lockfile.

Also, I think that copying node_modules from the referenced package might lead to "incorrect installs" – lib might have node_modules installed or not, which might affect which versions of packages app resolves. (In general, I could have also messed with lib/node_modules manually and it feels wrong that this should affect app.)

Describe the solution you'd like

I think that the hash should use the same helper functions as yarn pack (probably functions like getPackList in plugin-pack/sources/packUtils.ts).

Describe the drawbacks of your solution

It's a change in behavior and maybe someone wants their dependencies to be re-fetched into the cache even if files like .DS_Store or something inside node_modules changes? I'm not sure.

Describe alternatives you've considered

Currently, in README, we instruct developers to revert yarn.lock changes that are caused by their local gitignored files.

(Note: this was initially discussed in Discord, see this message and below.)

Search terms: file protocol, gitignored files, gitignore, npmignored files, npmignore, yarn pack

[ds-evo]

We had a similar problem.

The checksum calculation differs between Linux CI/CD and Windows Developer System.

Our Solution: We moved the <pkg>@file:<path> into a seperate private NPM Registry. Now the checksum is consistent.

[bimusiek]

We have run into exactly same issue. After upgrading from classic Yarn (v1) to latest one (3.2.0), our CI is failing with one of the file: dependencies.

In my opinion, the checksum should not be calculated based on whole directory but based on files configuration from package.json. What is the point of saying which files should be included in final lib if checksum is incorrectly calculated.

[omveersing]

We are also facing the very same issue (when upgrading from v1.x to v3.x), is there any resolution for it yet ?

[jackykwandesign]

worse decision in my life to upgrade from yarn1 in corepack to yarn3 this broke my CI

[psychobolt]

Seems like the only workaround is using a portal resolution link e.g. yarn link -r --private ../../packages/lib. This prevents creation of the checksum hash altogether...

"resolutions": {
  "lib": "portal:../../packages/lib"
}

[krassowski]

The portal: workaround does not work with lerna which only supports file: and workspace:. I cannot migrate to workspaces easily yet. Is there another workaround? For example, a way to disable hash calculation in metapackage setting?

[psychobolt]

@krassowski Yeah it will be a issue for monorepo tools that don't support portal:. From my case, it works with my setup using turbo.

[Downchuck]

My use case is bundling some relative workspace dependencies via:

enableGlobalCache: false
nodeLinker: pnp

Works great but I had a big data folder which was meant to be ignored that got bundled in and tripped me up for a bit, leading me to this long lasting bug report.