[Bug?]: Yarn runs multiple classic yarn installs on github deps without mutex. Causes cache issue/error.

[rally25rs]

Self-service

• [ ] I'd be willing to implement a fix

Describe the bug

There seems to be an issue with yarn finding multiple GitHub dependencies that use yarn classic. It tries to yarn install them in parallel, but does not pass any mutex option to yarn install so it ends up running into an error if they conflict on cache usage.

I believe this is the code that runs the parallel installs: https://github.com/yarnpkg/berry/blob/%40yarnpkg/cli/3.5.0/packages/yarnpkg-core/sources/scriptUtils.ts#L290

➤ YN0000: ┌ Fetch step
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT Packing @org/repoOne@git+ssh://git@github.com/org/repoOne#commit=123abc from sources
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT Packing @org/repoTwo@git+ssh://git@github.com/org/repoTwo#commit=234xyz from sources
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT Using Yarn Classic for bootstrap. Reason: "__metadata" key not found in yarn.lock, must be a Yarn classic lockfile
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT 
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT Using Yarn Classic for bootstrap. Reason: "__metadata" key not found in yarn.lock, must be a Yarn classic lockfile
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT 
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT ➤ YN0000: Retrieving https://classic.yarnpkg.com/latest.js
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT ➤ YN0000: Retrieving https://classic.yarnpkg.com/latest.js
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT ➤ YN0000: Saving the new release in .yarn/releases/yarn-1.22.19.cjs
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT ➤ YN0000: Done in 0s 702ms
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT 
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT ➤ YN0000: Saving the new release in .yarn/releases/yarn-1.22.19.cjs
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT ➤ YN0000: Done in 0s 733ms
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT 
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT yarn install v1.22.19
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT [1/4] Resolving packages...
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT yarn install v1.22.19
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT [1/4] Resolving packages...
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT [2/4] Fetching packages...
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT [2/4] Fetching packages...
➤ YN0000: │ /tmp/xfs-71fe8198 STDERR error https://registry.yarnpkg.com/regenerate-unicode-properties/-/regenerate-unicode-properties-10.0.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/circleci/.cache/yarn/v6/npm-regenerate-unicode-properties-10.0.1-7f442732aa7934a3740c779bb9b3340dccc1fb56-integrity/node_modules/regenerate-unicode-properties/Binary_Property/Dash.js'"
➤ YN0000: │ /tmp/xfs-71fe8198 STDOUT info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
➤ YN0000: │ /tmp/xfs-28e89bd5 STDERR error An unexpected error occurred: "https://registry.yarnpkg.com/regenerate-unicode-properties/-/regenerate-unicode-properties-10.0.1.tgz: ENOENT: no such file or directory, open '/home/circleci/.cache/yarn/v6/npm-regenerate-unicode-properties-10.0.1-7f442732aa7934a3740c779bb9b3340dccc1fb56-integrity/node_modules/regenerate-unicode-properties/.yarn-tarball.tgz'".
➤ YN0000: │ /tmp/xfs-28e89bd5 STDOUT info If you think this is a bug, please open a bug report with the information provided in "/tmp/xfs-28e89bd5/yarn-error.log".

To reproduce

Would need to have a yarn modern repo that has a dependency on 2 or more github URL dependencies, each of which are yarn classic repos. Then run a yarn install.

(my only good example is a set of private repos)

Environment

yarn 3.5.0
node 16.14
running in linux/docker with CircleCI's `cimg/node:16.14` image.

Additional context

No response

[arcanis]

Oh interesting - perhaps we should set --mutex network every time, indeed 🤔

[rally25rs]

I think I had wondered about that as well at some point in the past, but then wasn't sure if that would always be available... specifically when running in docker. Not sure if the current user would be able to open network ports.

Maybe using --mutex file with a path to the cwd of the yarn 2 command?

[RDIL]

The yarn 1 instances shouldn’t even be aware of each other, right? So how could they be touching the same files?

[arcanis]

The cache is shared by all instances running yarn install. And unlike Yarn 2+, it's not concurrent-safe by default.

[fmfmartins]

Any plans to move forward with the fix in #5373 ?

[MattFoley]

Is there any chance there is a workaround for this? I think potentially this issue is breaking my build.