Considering Yarn Berry (2.0+)

[zackkrida]

Hi all! Yarn's team lead just released a blog post outlining the first stable release of version 2.

The yarn docs have also switched over to 2.0, with the old site moved to https://legacy.yarnpkg.com.

As far as I can tell, this means a couple of things for Volta:

1. Currently, Yarn 2 cannot be installed via volta. The tag to install 2.0 is yarn@berry and since Yarn 2 is developed in a separate repo on GitHub from 1.x (https://github.com/yarnpkg/berry vs. https://github.com/yarnpkg/yarn), Volta's current list of Yarn releases doesn't support the 2x versions, since it uses the releases list for the yarnpkg/yarn repo:

https://github.com/volta-cli/volta/blob/83578540a04d6db1a503aea9936c56cec5c407df/crates/volta-core/src/tool/yarn/resolve.rs#L30

Depending on if/when Yarn intends on moving the 2.x versions into the yarnpkg/yarn repo, this will fix itself, but I'm not sure if that is planned.

2. The link used to get the latest version of Yarn appears to be deprecated. Currently Volta uses: https://github.com/volta-cli/volta/blob/83578540a04d6db1a503aea9936c56cec5c407df/crates/volta-core/src/tool/yarn/resolve.rs#L34 to get the latest version of yarn. That link now redirects to the legacy site. I do not know if Yarn intends to add this page to the new site.

Both of these issues would be resolved by switching to the parsing the npm registry url for both the latest version and enabling installation of beta releases: https://registry.npmjs.org/yarn

I'm not sure of potential downsides to this approach.

Again, all of these issues may be resolved depending on how Yarn manages the official launch of version 2, but not sure.

[zackkrida]

Got a speedy reply from the yarn team:

Will the yarnpkg/berry repo be moved into the main yarkpkg/yarn repo?

No, I think it's important to keep the history of the project. If we were to move yarnpkg/berry into yarnpkg/yarn, many links would end up broken (unless we purely "take over" the Yarn 1 repo, but I don't want to use the Yarn 1 issue tracker which is in a very sad state).

The previous docs site had a url to return the latest version of yarn. It currently redirects to the legacy docs: yarnpkg.com/latest-version. Will this be maintained on the new site, and should it even be used as a retrieval method of the latest version?

I think it will, yes. Right now we wanted to preserve the behaviour pretty much as it was before the Yarn 2 release so that workflows that expect a 1.x release have time to update their workflows if they need to. In a few weeks we'll start switching the tags and latest-version & friends will be part of it.

So, key takeway—volta's detection of the latest yarn version can stay the same, but detection of available releases should be changed.

As currently setup, when the yarn team officially switches the latest version to 2.x, volta will be unable to resolve the version and fail to install yarn.

Perhaps instaling yarn should just use the existing package resolution?

https://github.com/volta-cli/volta/blob/master/crates/volta-core/src/tool/package/resolve.rs

[charlespierce]

We definitely should look at supporting Yarn 2.0. The resolution issues we'll need to be aware of, if needed we can use the package resolution, though that has its own issues since the package that comes from npm currently doesn't include launchers for Windows the way the 1.x tarball does. We have workarounds for that, but it exposes us to an outstanding issue about tools being able to see each other, which we don't currently have with Yarn.

Ideally, Yarn will start providing a tarball again (their docs say that the "other" installers will be available soon), so if that's the case then we can start using that with whatever index Yarn provides.

Additionally, it looks like we need to support Yarn Plug'n'Play, because it appears that Yarn 2.0 is fully built on that. I think that's a relatively simple change, but we'd need to test it to be sure.

We already had supporting custom npm versions as the highest priority, and I think this fits in fairly well with that as a follow-up.

[charlespierce]

@zackkrida Thanks for compiling all of that documentation in the issue, and for checking with the Yarn team! That makes evaluating the changes a lot easier!

[chriskrycho]

Related: earlier discussion of Yarn Plug'n'Play occurred at #158.

[mbeaudru]

We should at least be able to install and use berry without issues related to volta, since running npm install -g yarn@berry raises an error.

@zackkrida suggestion seems great 👍

Details:

Volta error: Global package installs are not supported.

Use `volta install yarn@berry` to add a package to your toolchain (see `volta help install` for more info).

And the recommended command won't run either:

error: Could not parse version "berry"

Please verify the intended version.

[charlespierce]

Update on this: The core installation support for Yarn 2.0 should be fixed as of Volta 0.8.7. The Plug'n'Play issues outlined in #158 are still outstanding, and will cause globals that are trying to delegate to the local project to fail (since they look in node_modules/.bin which doesn't exist under PnP).

[billyvg]

@charlespierce yarn 2 installation is still not working...

It looks like yarn has not been releasing 2.x to npm: https://www.npmjs.com/package/yarn

[charlespierce]

@billyvg That's a good find, thanks! It looks like Yarn has silently changed their behavior with how they provide their binaries. We'll likely need to investigate deeper if there's a way we can download the packages from the same source that Yarn uses internally in order to provide the appropriate versions.

To use Yarn 2 for the time being, I would recommend using Volta to set a 1.* version of Yarn as your default, and then not pinning Yarn in the package.json but instead using Yarn's approach of yarn set version berry as described here: https://yarnpkg.com/getting-started/install#per-project-install

[alexlafroscia]

I did what @charlespierce suggested on a recent project and it actually worked really well! Volta still manages the Yarn 1 installation but delegates all "real" operations to the local Yarn 2 install. The local experience and using the GitHub Action for Volta both work great this way!

[ishiijp]

Hi, I installed yarn@berry with volta, but it says versino v2.0.0-rc27. Yarn has been already released v2.4.0 How do I install v2.4.0?

I tried to run yarn version set berry, then it created .yarn and .yarnrc.yml folder. it's a little annoying. Please someone help.

[charlespierce]

Hi @ishiijp, as mentioned above, it appears that Yarn has decided to not publish new versions in the npm registry, which is where Volta gets its information. As a result, we don't currently support Yarn 2 within Volta itself. The recommended way forward is to follow Yarn's installation instructions to install Yarn 1 globally (which will be managed by Volta), then use yarn version set berry in a given project to install Yarn 2 for that project.

We may, at some point, support fetching Yarn 2 from whatever source they are using now, however since that method changed at least once without a major version bump, I'm a little wary of tying our implementation to details of Yarn that aren't considered pubic API.

[ishiijp]

@charlespierce Thank you for your explanation. I understand very well.

[arcanis]

Hi @ishiijp, as mentioned above, it appears that Yarn has decided to not publish new versions in the npm registry, which is where Volta gets its information.

Update: I've just published @yarnpkg/cli-dist for the purpose of Volta, and we'll keep it updated after each new release. I'd have made a PR here as well, but I didn't find where to register it inside Volta (and my knowledge of Rust is ... rusty ... 😅)

[charlespierce]

@arcanis Awesome, thank you! Out of curiosity, is there any way to backport release the older Yarn 2 versions through that package as well? It's not a major issue if not, since Volta currently doesn't support directly pinning Yarn 2 at all, so we can say that 2.4.1 is the earliest that we support and it will still be an improvement over the status quo.

[merceyz]

You can also fetch the CLI from GitHub

https://raw.githubusercontent.com/yarnpkg/berry/%40yarnpkg/cli/<VERSION HERE>/packages/yarnpkg-cli/bin/yarn.js

[charlespierce]

@merceyz That's true! The concern we have with using GitHub directly (we actually initially did that for Yarn 1 in the early days of Volta) is the rate-limiting on GitHub API requests. We ran into that a lot, especially in CI / testing environments when we made multiple requests for information in quick succession.

Another concern with fetching only the raw JS file is that we then need to duplicate some of the work that the package manager does to make sure that file is callable on every system (e.g. adding shims / scripts to Windows where the #! operator doesn't work).

Actually looking at it @arcanis it looks like that @yarnpkg/cli-dist package has the same issue: It includes the raw JS file but not any of the helpers that make things executable. Compare with the files in the legacy yarn package, where the bin directory includes scripts named yarn and yarn.cmd which handle some specific details around actually executing the underlying JS file in different environments.

[nemonemi]

@charlespierce, so, since the pull request from @arcanis to yarnpkg/bery for providing dist-scripts has been merged, does this help to progress this issue forward?

[charlespierce]

@nemonemi It looks like I missed that PR, but yes, I think it would help move things forward. From the description, it looks like that will be available starting with Yarn 3.0, so we can probably use that as our cutoff point.

[ondratra]

Yarn 3.0.0 has been released recently. https://github.com/yarnpkg/berry/blob/master/CHANGELOG.md#300 Maybe this issue can finally be resolved?

[MichalBryxi]

Resolved in the sense of opening a ticket for yarn-3.x, right?

[ondratra]

@MichalBryxi Well, :D since v2 was never supported in Volta, there is a little point in supporting it since v3 should be better in every way. I guess this issue can really be closed and superseded by a new ticket for v3, as you suggest.

From the discussion above, I got an impression that after v3 release, the actual problems blocking support of new Yarn versions will be gone, but maybe I misunderstood

[lolmaus]

Yarn has been rewritten from scratch, and this effort was called "Yarn codename Berry". Their repo is called berry.

Both Yarn v2 and v3 belong to the same Berry codebase. They have the same API.

If I'm not mistaken, this issue can be renamed from "Yarn 2.0" to "Yarn Berry" without any changes to its description or conclusions.

[nemonemi]

Just for my information, now with the latest Berry should everything be working, or is there something left to be done here?

[JanJakes]

Not sure what's the status here but it seems to me:

• I can't pin yarn to 3.0.0 with Volta (Volta error: Could not download yarn@3.0.0).
• Tried "@yarnpkg/cli-dist": "3.0.0" but that doesn't seem to work either (Volta error: No Yarn version found in this project.).
• I can keep "yarn": "1.22.11" and it works as Yarn v1 detects and manages Yarn Berry executables by itself.

Then I'm just not sure what this was for:

Update: I've just published @yarnpkg/cli-dist for the purpose of Volta

Is it so that support for this on Volta's side is still missing?

[cspotcode]

As mentioned here https://github.com/volta-cli/volta/issues/651#issuecomment-841668494, the shims are included in @yarnpkg/cli-dist 2.4.2 and up. https://unpkg.com/browse/@yarnpkg/cli-dist@2.4.1/bin/ <-- not included https://unpkg.com/browse/@yarnpkg/cli-dist@2.4.2/bin/ <-- included https://unpkg.com/browse/@yarnpkg/cli-dist@3.0.0/bin/ <-- still included

So volta can support yarn v1.x.x using the package source it uses today, and yarn >= 2.4.2 using npm @yarnpkg/cli-dist

Versions >=2 and <2.4.2 will be unsupported.

Can this be implemented using hooks? I believe it will need to be bin hooks, because they will need to query 2x package sources and merge the results.

https://docs.volta.sh/advanced/hooks

[charlespierce]

@cspotcode Yes, as a short-term solution this can definitely be implemented using bin hooks to direct at the appropriate download location. I'll also try to take a look this week to see if there's a simple way to update Volta to fetch Yarn 2.4.2+ (or possibly 3+ for simplicity, since it looks like there's just the one 2.x version that has the appropriate files).

[taras]

@charlespierce would you be able to provide an example of how to use yarn 3 using hooks?

[ilijapuaca]

@cspotcode Yes, as a short-term solution this can definitely be implemented using bin hooks to direct at the appropriate download location. I'll also try to take a look this week to see if there's a simple way to update Volta to fetch Yarn 2.4.2+ (or possibly 3+ for simplicity, since it looks like there's just the one 2.x version that has the appropriate files).

@charlespierce any updates on whether this is still being explored?

[david2am]

I was trying to use Volta to handle my node versions but till this feature is not resolved I prefer to use nvm

[jmathew]

For the time being, is there a way I can have volta explicitly not manage yarn, but still manage everything else?

I've read through many of these yarn related issues and it seems to me I cannot install yarn 3 alongside Volta (on Windows at least). The migration path for 1 -> 3 involves installing version 2, which can be installed but fails as expected on the symlinks.

[charlespierce]

@jmathew This is actively being worked on right now. That said, in the interim, if you don't configure yarn at all, then Volta should delegate to the rest of the system, so if you have Yarn installed through some other means it would use that installation.

[sargunv]

@charlespierce Could you elaborate on how to not configure yarn with Volta? It seems it doesn't delegate yarn to the rest of the system when node is pinned in the local project.

I just installed Volta v1.0.8, ran volta pin $(cat .nvmrc) in my previously nvm project, and running yarn now gives me a Volta error:

Volta error: No Yarn version found in this project.

Use `volta pin yarn` to select a version (see `volta help pin` for more info).

I understand I can work around the issue with rm ~/.volta/bin/yarn, but I'm not sure if/when I'll need to re-run my workaround.

[dpotter895]

@charlespierce would you be able to provide an example of how to use yarn 3 using hooks?

@charlespierce I would love to use Volta but have a hard requirement for yarn v3. I am fine with a workaround if it exists though. Is yarn v3 in the pipeline to integrate?

[david2am]

I like Volta and all the work the team does but I feel that you have not imagined how important it is for us to have yarn updated to v3+, it's the only reason why I don't use it in work-related projects or personal ones. It has been 2 years since this issue has been open... please do something I'll appreciate it. Regards

[fusion2004]

@JanJakes previously mentioned this, but if you keep yarn pinned to the latest of the 1.x line with volta, and use yarn set version ... to use a repo-local yarn 2/3 then everything should work okay. I've had success doing that across multiple projects over the last few years. @dpotter895 @david2am

[zackkrida]

@david2am I don't work on Volta, just a user like yourself. I also have no need for Yarn 2 support anymore so I'm going to unfollow this issue. Best of luck.

[chriskrycho]

Hey folks, Volta did actually recently implement Yarn 3 support: see volta-cli/rfcs#48 for background and #1193, #1204, and #1230 for implementation. We're in the process of doing a bunch of QA via rolling it out internally at LinkedIn at present.

Also, a friendly reminder for folks like @david2am – we understand quite well and we’ve all wanted this for quite some time. But as with all open source work, it gets done either (a) on people's personal time, which is not great; or (b) when a company is able to fund it because it matters to that company, which is how this is eventually getting across the line. We also welcome contributions and are happy to provide input and guidance on things! That's how the pnpm support is now getting implemented, which is awesome. But it's always worth remembering that everyone's time and ability to go after things like this is limited. 😅

[david2am]

Sorry for the pressure guys, my bad... 😓 @chriskrycho thanks for your comment 👍🏻

[chriskrycho]

No worries! Just figured I'd help everyone understand why these things take so long from the other side—we all get it and experience it ourselves as well!

[chriskrycho]

I think this can be closed; we have had Yarn 3 support for two years now! 🎉