[Bug] Yarn v1 installation delegation of git source packages

[martinjlowm]

• [x] I'd be willing to implement a fix

Describe the bug

Yarn V2 (and V3) both suffer from the original V1 concurrency issue in terms of cache conflicts with git sources as is described here: https://github.com/yarnpkg/yarn/issues/8032 - this effectively causes the installation process to fail.

Installing packages from git sources delegates the work to V1 and there's currently no way of supplying configuration options to its environment unless a V1 .yarnrc-file is provided by the source (I believe that to be the case). The consumer is not always able to push such a file upstream. A workaround to this issue would've been to add a mutex to the installation process with --mutex=network, but there's currently no way of passing that option to the subprocess. The V2 config parser prohibits any legacy configuration options which means neither --mutex=network or YARN_MUTEX=network will work.

The closest solution on the current HEAD I've found is the possibility to inject environment variables through the setupScriptEnvironment-hook. However, as can be seen here: https://github.com/yarnpkg/berry/blob/0d4f86c3ad20d5e4115fa9eae0a35841c34445ed/packages/yarnpkg-core/sources/scriptUtils.ts#L133-L142 this only applies if project is set - and that is not the case for configuring a temporary package environment (called from https://github.com/yarnpkg/berry/blob/0d4f86c3ad20d5e4115fa9eae0a35841c34445ed/packages/yarnpkg-core/sources/scriptUtils.ts#L185) where only the binFolder is passed to the function.

What I'm seeing is a bunch of ENOENT errors with a common cache.

To Reproduce

N/A - this is not a direct problem with Yarn2/3, hence the lack of a reproduction.

Screenshots

N/A

Environment if relevant (please complete the following information):

N/A

Additional context

My current workaround is to patch the Yarn bundle by ~injecting a mutex~ setting a local cache folder to the environment for the V1 interface at https://github.com/yarnpkg/berry/blob/0d4f86c3ad20d5e4115fa9eae0a35841c34445ed/packages/yarnpkg-core/sources/scriptUtils.ts#L206

I.e.:

// env['YARN_MUTEX'] = 'network' // apparently this did not appear to work consistently...
const install = await execUtils.pipevp(`yarn`, [`install`, `--cache-folder`, `./.yarn-cache`], {cwd, env, stdin, stdout, stderr, end: execUtils.EndStrategy.ErrorCode});

[danrha]

@martinjlowm I'm having the same problem. Is it reasonable for us to create an attribute for .yarnrc to specify arbitrary options we want to add to the Yarn Classic executions?

[martinjlowm]

Hmm, I actually ended up forcing Yarn 2 as the packager - it kind of depends on how long Yarn 1 remains the default packager for git sources - I haven't experienced any issues with Yarn 2 so far.

The --cache-folder option did fix this particular issue, but it ended up packing that folder in the archive, producing a new checksum on every new installation.

[danrha]

The --cache-folder option did fix this particular issue, but it ended up packing that folder in the archive, producing a new checksum on every new installation.

@martinjlowm I actually used ./.yarn/cache instead. You can try that, it gets excluded in the .npmignore file generated there so it doesn't end up in the archive.

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟