search

yarnpkg / berry

📦🐈 Active development trunk for Yarn ⚒
https://yarnpkg.com
BSD 2-Clause "Simplified" License
7.43k stars 1.11k forks source link

[Bug?]: Yarn fails to handle when cacheFolder is a symlink farm #3514

Open stevenxu-db opened 3 years ago

stevenxu-db commented 3 years ago

Self-service

Describe the bug

When cacheFolder points to a folder containing symlinks to zips rather than actual zips, Yarn fails to handle it as we expect. The failure happens regardless of whether --immutable-cache is used. Our main use case is with nodeLinker: node-modules, but a similar error also appears to affect PnP. When the cacheFolder is a symlink farm, yarn fails with the following error:

➤ YN0001: │ Error: While persisting /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77720ZcZ4v1xoAlgh/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery/ -> /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77720ZcZ4v1xoAlgh/node_modules/jquery ENOTDIR: not a directory, scandir '/private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77720ZcZ4v1xoAlgh/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery'

Here is the additional failure case affecting PnP. This doesn't affect us right now, but our goal is to migrate to PnP in time, and this would impact our ability to move:

await yarn('init');
await yarn('add', 'jquery@3.6.0');

const { execSync } = require('child_process');
execSync('mkdir symlink-farm');
execSync('find .yarn/cache/* -exec ln -s ../{} symlink-farm/ \\;');
execSync('rm -rf node_modules');

await yarn('config', 'set', 'cacheFolder', 'symlink-farm');
await expect(yarn()).resolves.not.toThrow();
await expect(yarn('node', '-e', 'console.log(require("jquery"));')).resolves.toContain('[Function (anonymous)]');

Reported error:

Error: Required package missing from disk. If you keep your packages inside your repository then restarting the Node process may be enough. Otherwise, try to run an install first.

Missing package: jquery@npm:3.6.0
Expected package location: /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77838wPTUvBCK43Kr/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery/

Require stack:
- /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77838wPTUvBCK43Kr/[eval]
    at Function.external_module_.Module._resolveFilename (/private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77838wPTUvBCK43Kr/.pnp.cjs:9684:55)
    at Function.external_module_.Module._load (/private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/tmp-77838wPTUvBCK43Kr/.pnp.cjs:9483:48)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at [eval]:1:13
    at Script.runInThisContext (vm.js:133:18)
    at Object.runInThisContext (vm.js:310:38)
    at internal/process/execution.js:77:19
    at [eval]-wrapper:6:22
    at evalScript (internal/process/execution.js:76:60)

To reproduce

await yarn('init');
await yarn('config', 'set', 'nodeLinker', 'node-modules');
await yarn('add', 'jquery@3.6.0');

const { execSync } = require('child_process');
execSync('mkdir symlink-farm');
execSync('find .yarn/cache/* -exec ln -s ../{} symlink-farm/ \\;');
execSync('rm -rf node_modules');

await yarn('config', 'set', 'cacheFolder', 'symlink-farm');
await expect(yarn()).resolves.not.toThrow(); // --immutable-cache makes no difference here

Environment

System:
    OS: macOS 11.5.2
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
  Binaries:
    Node: 14.16.1 - /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/xfs-1a58a4dc/node
    Yarn: 3.1.0-rc.8.dev - /private/var/folders/4y/p9httzjd7px11pkmsfvlbdym0000gp/T/xfs-1a58a4dc/yarn
    npm: 6.14.12 - ~/.nvm/version/npm

Additional context

We use Bazel (though not rules_nodejs). For each build action, Bazel naturally provides a sandboxed symlink farm because it doesn't guarantee that the folder the real zips are in is sandboxed. We'd like to point Yarn to this symlink farm instead of the real folder and we'd also like to avoid having to dereference the links and pay the cost of writing it to disk. For us, this only impacts install for now since we're using nodeLinker: node-modules, and after the install, the cacheFolder has no impact. But this will become increasingly important for us in order to migrate to PnP, otherwise we have to dereference the symlink farm at the beginning of every action.

yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.not.toThrow()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js

➤ YN0000: ┌ Resolution step
::group::Resolution step
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
::group::Fetch step
➤ YN0013: │ 3 packages were already cached
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
::group::Link step
➤ YN0001: │ Error: While persisting /tmp/tmp-18lc1Cs88CAhHO/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery/ -> /tmp/tmp-18lc1Cs88CAhHO/node_modules/jquery ENOTDIR: not a directory, scandir '/tmp/tmp-18lc1Cs88CAhHO/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery'
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 76ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:12:7)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:16)
    at async Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)
yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.not.toThrow()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js

➤ YN0000: ┌ Resolution step
::group::Resolution step
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
::group::Fetch step
➤ YN0013: │ 3 packages were already cached
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
::group::Link step
➤ YN0001: │ Error: While persisting /tmp/tmp-19hW663UHaZPYs/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery/ -> /tmp/tmp-19hW663UHaZPYs/node_modules/jquery ENOTDIR: not a directory, scandir '/tmp/tmp-19hW663UHaZPYs/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery'
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 85ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:12:7)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:16)
    at async Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)
yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.not.toThrow()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js

➤ YN0000: ┌ Resolution step
::group::Resolution step
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
::group::Fetch step
➤ YN0013: │ 3 packages were already cached
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
::group::Link step
➤ YN0001: │ Error: While persisting /tmp/tmp-182ZkHHgdoxFCP/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery/ -> /tmp/tmp-182ZkHHgdoxFCP/node_modules/jquery ENOTDIR: not a directory, scandir '/tmp/tmp-182ZkHHgdoxFCP/symlink-farm/jquery-npm-3.6.0-ca7872bdbb-8fd5fef4aa.zip/node_modules/jquery'
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 95ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:12:7)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:16)
    at async Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)
thatsmydoing commented 1 year ago

It seems that this used to be supported but the functionality was removed in https://github.com/yarnpkg/berry/pull/1474/commits/22ef17ae8e65ab80d409a6443ec5ba22e3ff726c for performance reasons. I'm not quite sure removing the symlink resolution actually makes things any faster in the common case (when it's not a symlink) so maybe we can bring it back?