Support for pnpm

[willsoto]

Seems to be a fairly popular alternative to npm and yarn. Apologies if this has already been suggested but I couldn't find an existing issue. It would be great to be able to volta pin pnpm.

https://pnpm.js.org/en/

[charlespierce]

Agreed, this is definitely something we’ve discussed on Discord, I’m surprised there wasn’t an issue already either. Thanks!

[arxpoetica]

What would need to take place for this to happen? Point me to the right place in the code, and I'll see if I can get around to a PR.

[charlespierce]

Hi @arxpoetica, we would definitely welcome that! Adding support for pnpm—or really any unsupported package manager—is a bit involved, though hopefully not too difficult. There are 3 main areas that need to be updated:

• Fetching / installing / pinning the version, which is all contained within the tool module in crates/volta-core (https://github.com/volta-cli/volta/tree/main/crates/volta-core/src/tool)
  • The changes here can mostly be based on what is done for npm, the only caveat I'm aware of is that instead of editing the launcher files, we need to create them (so that the bare command pnpm works as expected).
• Reading / writing pnpm settings, contained in the platform, project, and toolchain modules.
  • The changes here should mostly be adding the pnpm key alongside npm and yarn
• Running pnpm through the shim, contained in the run module.
  • Here the logic should be almost identical to the yarn logic.

That's a very high-level overview of the needed changes, but I'll take a look at write some more this week about any corner cases / difficulties I can predict. I'm also happy to pair on this for a while if that would help.

[arxpoetica]

@charlespierce this ended up being something I just haven't had time to get around to yet. It's definitely still on my radar, but if someone feels like giving it a head start (or even attacking totally)...

...I think you'll find it happening more quickly. Sorry for my lack of involvement (re: time).

[charlespierce]

@arxpoetica No worries at all, thanks for updating!

[willsoto]

@charlespierce slightly related to this, has any thought been given to pinning arbitrary dependencies that would useful to have available on the command-line? For example, it might be useful to have mocha or tsc installed and available directly on the command line at a specific version without telling users to volta install it themselves. Another example is something like lerna or rush which are the primary entry-points for repos that are managed by those tools.

Under the hood, maybe something specific happens when pnpm is installed but in some ways it is just another command line tool being managed by Volta.

Just wondering if you consider this out of scope since the documentation for pin has something specifically about this.

[charlespierce]

Hi @willsoto, are you talking about something like #861? In that issue there's some discussion about the complications, however that is definitely something we'd like to support. Or do you mean a setting in package.json that says "Use Volta to manage running lerna"?

If the latter, there's a couple reasons that's not really in scope for Volta:

1. Technically speaking, Volta operates with shims in a shell-agnostic way, there's no way for us to shim tools that we don't know about ahead of time (if a user runs lerna and the shell says "Command not found", that never goes through anything Volta controls). Individual shells may have a way to control that behavior, but it's unlikely to be universally applicable across OS and shell combinations.
2. Philosophically, we decided against creating automatic shims for everything that had been installed, because it caused issues and raised some security concerns (see #235 for the discussions). We ultimately decided that the end user should have direct control over which tools we do or do not manage, so it would go against that to allow packages to take that control away from the user.

[willsoto]

Of course I didn't see #861 when I went looking for existing issues 🤦

After reading through it a couple of times, it does seem like that would also solve my problem, just in a different way.

Or do you mean a setting in package.json that says "Use Volta to manage running lerna"?

Yeah, something like this is what I had in mind:

// package.json
{
  "volta": {
    "lerna": "2.0.0"
  }
}

I totally understand if this is considered out of scope for Volta, but I thought since pnpm is something that gets installed via npm (I don't think they have an alternative way to download?) then you could really get any tool in this way. I'll subscribe to #861 and see where that goes so I don't clutter this issue up. Thanks

[zkochan]

@muuvmuuv suggested us to document Volta as the preferred tool with pnpm. Unlike Yarn, pnpm currently doesn't have the ability to seamlessly switch between different versions of pnpm. So if Volta was to support pnpm, I think we could just recommend using the pin feature of Volta.

Also, @arcanis might be interested in this discussion, he created corepack

[zkochan]

I think we can offer a $50 bounty from our opencollective funds (if this is enough, I don't know how much work it involves).

[charlespierce]

@zkochan I took a look at it before our 1.0 release late last year, and it actually was somewhat involved, as our models and abstractions around the package managers aren't as clean as they could be (some room for tech debt cleanup). However, it was mostly straightforward. The main sticking point was around the global package directory: As I recall, there isn't a clean way to redirect both the installation and bin directories to a custom location in pnpm using only environment variables.

Volta does that redirection when installing global packages so that we can sandbox them to a specific Node version, and I recall struggling to get that working. I think I could do it with a command-line flag, but that gets a little tricky if there's already one specified and so I didn't dig in too deeply. There also appeared to be a "layout version" type directory that was always appended, which we would need to take into account, but I think I had worked out how we can handle that on our side.

I determined most of that with some experimentation and poking around the source of pnpm, so if there's something I missed about how to define the global directories, that would be much appreciated!

So in all, I think it's probably a larger task than a $50 bounty, however if someone was interested and wanted to collect that on top of the experience, I'd be more than happy to mentor them through it. Working on Volta is also part of my day-to-day, so if I can get it onto the roadmap that would be the other way.

[zkochan]

Is there something we could implement on pnpm's side to simplify the task?

You may set the NPM_CONFIG_GLOBAL_DIR env variable to change the global directory used by pnpm. The global bin directory will currently be selected automatically from one of the directories in PATH. pnpm has to have write access to the directory and the directory should have npm, pnpm, nodejs, or node in the path

https://github.com/pnpm/pnpm/blob/9c41e11b0bfa3f4ece2b7d4d67690496f1c9b9c0/packages/global-bin-dir/src/index.ts#L35-L40

[charlespierce]

Is there something we could implement on pnpm's side to simplify the task?

I don't think so, only because those changes would necessarily be in only newer versions of pnpm, while we would ideally want to support a larger range of versions.

You may set the NPM_CONFIG_GLOBAL_DIR env variable to change the global directory used by pnpm. The global bin directory will currently be selected automatically from one of the directories in PATH. pnpm has to have write access to the directory and the directory should have npm, pnpm, nodejs, or node in the path

https://github.com/pnpm/pnpm/blob/9c41e11b0bfa3f4ece2b7d4d67690496f1c9b9c0/packages/global-bin-dir/src/index.ts#L35-L40

Interesting, that is jogging my memory a bit. I think I may have been able to get that to work by modifying the PATH, but I don't recall the specific issues I was hitting any more (and I appear to have failed to write them down anywhere). I don't think any of them were insurmountable, just that they were decent amount of work to fit into everything, taking into account our current model.

[muuvmuuv]

I would add another 30€.

Just saying, but I'm already using pnpm with Volta since three months without any problems but would also love to see a "clean" way of implementation for future package managers/binaries so Volta will stay on top of the other managers.

[cjk]

I'm using pnpm a lot, so I'll wait until proper pnpm-support is available before switching to Volta

[Kinrany]

I don't think so, only because those changes would necessarily be in only newer versions of pnpm, while we would ideally want to support a larger range of versions.

Is it possible to provide support for newer versions and just warn the user that older versions are not supported?

I think most people using pnpm are OK with upgrading once even if they want to pin their version and not upgrade regularly.

[shiftgeist]

I think most people using pnpm are OK with upgrading once even if they want to pin their version and not upgrade regularly.

pnpm warns about outdated versions whenever you use it. This should reduce the amount of outdated versions. At least for desktop use.

[charlespierce]

@Kinrany @shiftgeist Those are good points, if we need to we can likely work to get changes upstreamed to pnpm and then only support that version or later. Ideally we'd make it work without that, but as a last resort that makes sense, especially if pnpm itself is vocal about pushing you to upgrade.

[kaelonR]

As I would also like to see pnpm officially supported as a pinnable package manager by Volta, I will be adding a $110 bounty for getting this supported. Together with zkochan's $50 bounty offer, this brings the total collectable bounty to $160. (+30EUR/35 USD if muuvmuuv is actually offering another 30EUR, but not sure whether he's offering €30 or was suggesting to raise the bounty by €30).

Hopefully this is enough to stimulate integration of pnpm :)

[arxpoetica]

I'll add $50 USD to that. This is one of my most watched issues.

[muuvmuuv]

Imagine they just pokering to get more people putting $$$ into the issue 😆

[kaelonR]

Imagine they just pokering to get more people putting $$$ into the issue 😆

The goal is to entice someone who's knowledgable enough to pick up this issue and integrate pnpm. The collectable bounty for implementing this is now $210. Surely someone must be interested in making a quick buck?

[muuvmuuv]

This was meant as a joke...

[shiftgeist]

Is there an issuehunt or similar project?

[ekil1100]

Is there an issuehunt or similar project?

I don't think so...can't find the link.

[shiftgeist]

Is there an issuehunt or similar project?

I don't think so...can't find the link.

Might be useful for collecting the funds (doesn't need to be issuehunt tho). I imagine multiple paypal transaction being a little sketchy.

[charlespierce]

Hey everyone! Sorry about the delayed response, I've been out for a week or so. Super exciting to see so much interest for pnpm, I'd definitely be happy to help facilitate the bounty if someone wants to take it on (and help guide people through the code for that section). My time has been pretty limited lately, as I just started a new job, but I'll also try to carve some out to push this forward.

[mikrostew]

I have an RFC up for adding pnpm support: https://github.com/volta-cli/rfcs/pull/46

I think I captured most of the details and requirements from this and other conversations, and welcome your feedback!

[zkochan]

I have lost interest in this feature.

Node.js shipped Corepack, which autoinstalls the right version of pnpm: https://nodejs.org/dist/latest-v16.x/docs/api/corepack.html

Also, pnpm may be used as a Node.js version manager: https://pnpm.io/cli/env

[arxpoetica]

Corepack is still experimental.

[canadaduane]

pnpm (installed via volta) suggested I upgrade today: Update available! 6.18.0 → 6.21.0. and so I did what they suggested, "Run pnpm add -g pnpm to update" and pnpm is now broken:

$ pnpm install
Volta error: 'volta-shim' should not be called directly.

Please use the existing shims provided by Volta (node, yarn, etc.) to run tools.

Just running the executable does the same:

$ pnpm
Volta error: 'volta-shim' should not be called directly.

Please use the existing shims provided by Volta (node, yarn, etc.) to run tools.

I don't know how to get back to a working state, so I might need to remove volta.

[chriskrycho]

A workaround for that last bit while we're working on pnpm support: in a repo which needs pnpm, start by running npm install and then run pnpm install. After that, you'll be in a working state. (It's quite annoying, we all recognize! Hopefully we'll get some motion on merging and implementing volta-cli/rfcs#46 soon!)

[sbonasu]

can you please let me know if there are any updates on this feature request.

[charlespierce]

@sbonasu There aren't any updates at the moment. All of the maintainers are busy with other work. However, in the RFC linked above, there's a potential way forward after some feedback from the maintainer of pnpm, where we can likely start with supporting pnpm 7, which should be more straightforward.

When there are more updates, you should see them here.

[coconilu]

couldn't agree more, please let me know if there are any updates

[Mr-xzq]

Now that pnpm is very popular, expect volta to add support for pnpm.

[chawyehsu]

Hi all participants, I just initialized PR #1273 which is going to take this feature request into reality. I don't know if the bounty is still available but please consider leaving comments when you are available so I can push it forward.

[arxpoetica]

I don't know if the bounty is still available

I'm still good for $50.

[kaelonR]

I'm also still offering a $110 bounty. It's yours if PR 1273 gets merged.

[IT-MikeS]

I will offer up $50 as well if pnpm support is added from your PR @chawyehsu

[ScreamZ]

What about this ? https://pnpm.io/cli/env Seems this is a war between both tools 😅 I prefer volta way to manage things btw

[NatalePorto]

Looking forward to see this issue closed and integrated in a new release.

[ild0tt0re]

please guys, the world is switching to pnpm let us continue to use our favorite tool manager volta ⚡ ❤️

[wagenet]

Is this resolved by #1273 ?

[IT-MikeS]

Is this resolved by #1273 ?

Seems like it. As promised @chawyehsu I sent in a one-time sponsor of 50$. Thank you for your work!

[chawyehsu]

@wagenet The PR was successfully merged, the feature is not released yet though CI builds are available for a try. At present, there is no ETA for it, the team is likely to gather more fixes such as Linux ARM builds and the TLS cert fix apart from this feature for the next release.

@IT-MikeS Much appreciate!

[charlespierce]

I just opened #1394 to add a feature flag environment variable to ensure we maintain backwards compatibility for users who currently have pnpm as a global package. Once that is approved / merged, we should be able to release Volta 1.1.1 with pnpm support included. The feature flag will mean that anyone who wants to try this behavior will need to set VOLTA_FEATURE_PNPM to 1 in their environment variables or profile script, for example:

export VOLTA_HOME="$HOME/.volta"
export VOLTA_FEATURE_PNPM=1
export PATH="$VOLTA_HOME/bin:$PATH"

[haosmos]

I just opened #1394 to add a feature flag environment variable to ensure we maintain backwards compatibility for users who currently have pnpm as a global package. Once that is approved / merged, we should be able to release Volta 1.1.1 with pnpm support included. The feature flag will mean that anyone who wants to try this behavior will need to set VOLTA_FEATURE_PNPM to 1 in their environment variables or profile script, for example:

export VOLTA_HOME="$HOME/.volta"
export VOLTA_FEATURE_PNPM=1
export PATH="$VOLTA_HOME/bin:$PATH"

@charlespierce, hi! 😃

Any news? When should we expect the long-awaited release with pnpm support? Is there any chance it will be released this year?

[charlespierce]

Sorry for the delay on this, I've been swamped with other work! Volta 1.1.1 was just released and it includes the experimental pnpm support (HUGE thanks to @chawyehsu for the implementation)! I also added a section to the advanced docs about the pnpm Support, noting the known limitations and how to activate it!

[arxpoetica]

@chawyehsu sent my sponsorship!