Add nixpkgs tooling for `pnpm-lock.yaml`

[ambroisie]

Issue description

We have buildNpmPackage for package-lock.json, and mkYarnPackage for yarn.lock.

However we currently have no tooling available in nixpkgs to build JS packages that make use of pnpm-lock.yaml.

Proposed solution

Add support to pnpm-lock.yaml through e.g: mkPnpmPackage.

Alternatives

I have packaged pnpm-lock-export for use in packaging the next version of Woodpecker's front-end in my personal configuration with the available tooling. Unfortunately that tools does not support the latest version of the lock file, so it was quickly made useless for my usage once upstream updated their repo...

[figsoda]

cc @NixOS/node

[Lord-Valen]

This would be useful for repackaging, reducing the size of nodePackages and for numerous unmet package requests (I'm guessing nobody is terribly keen on adding to nodePackages and I can't blame them). Just a few examples that I came across: #180968 #207708 #170254 #226535.

[6543]

just as an idea, did someone bring up to upstream https://github.com/cvent/pnpm-lock-export directly into pnpm jet ?

[6543]

they at least have an import ...

[ambroisie]

@6543: I believe this discussion is about a similar subject.

If you need this feature, you need to ask for it in the npm CLI repository. Why would we spend our time on a feature that helps people switch away from pnpm to another package manager? It doesn't make sense.

To be fair, I think their stance is quite rational.

[adamcstephens]

pnpm-lock-export does not support the current pnpm lock file version. The current woodpecker lock file is failing with the below error.

─❯ nix run nixpkgs#pnpm-lock-export -- --schema yarn.lock@v1
Your lockfile version (6.0) is higher than the supported version of pnpm-lock-export (5.4).
Error: /@ampproject/remapping@2.2.0 is an invalid relative dependency path

[adamcstephens]

I'm not proud of it, but I hacked pnpm-lock-export to support converting the newer lock file to a yarn lock: https://github.com/adamcstephens/pnpm-lock-export/commit/d1a0fcfd39099e66c6eecf3896dadf624b6d3da1

I'm a bit over my head here, but I am able to convert the woodpecker lock, which was my goal.

[linsui]

https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/pot/default.nix I added a pnpm package by fetching deps with pnpm. It's pretty simple. What can a mkPnpmPackage bring to us? I thought a fetchPnpmDeps is easier to use.

[ambroisie]

Having a fetchPnpmDeps function would be a necessary first step towards adding mkPnpmPackage, and would indeed be useful for more complicated purposes. For packages that don't need the complexity, having a mkPnpmPackage function makes the easy thing easy.

I think it's good to have mk*Package to keep the API similar to other package managers currently supported by nixpkgs (yarn and npm both have a mk<Name>Package function).

[Lord-Valen]

I think it's good to have mk*Package to keep the API similar to other package managers currently supported by nixpkgs (yarn and npm both have a mk<Name>Package function).

I think build*Package is the more common pattern.

Anyway, having builders for trivial packages makes for less boilerplate and communicates what is going on up front, which is a win. Builders also provide a very clear approach to packaging: They mainly work to smooth out idiosyncrasies e.g. workspaces, package managers. Part of that means developing and using functions like build*Deps.

[winterqt]

master/pkgs/applications/misc/pot/default.nix I added a pnpm package by fetching deps with pnpm. It's pretty simple. What can a mkPnpmPackage bring to us? I thought a fetchPnpmDeps is easier to use.

This will break as soon as anything in pnpm's store format changes.

[linsui]

Yes, fetchers break when format changes.

[winterqt]

@linsui The reason that buildNpmPackage and friends is so complicated is to avoid breaking the store format, so FOD hashes will more or less never have to be updated. We'd need to do the exact same thing for pnpm.

Every other fetcher in Nixpkgs (bar Cargo and Go for exactly this reason) does not cause breakages on random updates.

[MarcelCoding]

Hi, I've discovered a working version of pnpm2nix, and it works really well: Usage example: https://github.com/tlm-solutions/kindergarten/blob/c24b8b3700658dd56ec163941a7e604eae0575c9/derivation.nix Maybe this would be something to integrate into nixpkgs.

[ambroisie]

@MarcelCoding this is using pnpm install --frozen-lockfile --offline which can (and most probably will) change when upgrading pnpm, leading to FOD failures as the hash will suddenly need to be updated (as @winterqt explained a few comments earlier).

[MarcelCoding]

So the solution need to be without any pnpm command execution? Basically rebuilding the node_modules folder in plain nix?

[lilyinstarlight]

So the solution need to be without any pnpm command execution? Basically rebuilding the node_modules folder in plain nix?

If upstream pnpm guarantees that not only their cache format but also cache generation is reproducible long-term then we could potentially use it

But right now we'll otherwise likely do what we do with npm and use our own tooling to generate the pnpm cache (not the node_modules folder) and then let pnpm take care of the rest

[MarcelCoding]

What do you mean by cache? pnpm has a global store with all packages. pnpm2nix fetches all packages mentioned in pnpm-lock.yaml and creates a corresponding entry on /nix/store for every package. After that, another entry in /nix/store is created, which represents the store. The store is generated using pnpm store add. After that, pnpm install is used to generate a node_modules folder using the generated store.

[lilyinstarlight]

What do you mean by cache? pnpm has a global store with all packages. pnpm2nix fetches all packages mentioned in pnpm-lock.yaml and creates a corresponding entry on /nix/store for every package. After that, another entry in /nix/store is created, which represents the store. The store is generated using pnpm store add. After that, pnpm install is used to generate a node_modules folder using the generated store.

The store is pnpm's equivalent to the cache iirc. I assumed by FOD failures @ambroisie meant that it uses pnpm tooling to generate FODs and I'm admittedly on mobile and didn't actually look at the pnpm2nix code and it seems this is an input-addressed output, not a FOD

Ideally pnpm store add is reproducible, but non-reprodudicibility is better in a standard input-addressed derivation than a FOD, so I guess this approach is a little more sound than current pnpm hacks that have been merged with questionably unreproducible FODs

This could be useful at minimum as a reference though, so I'll try to take a thorough look at the pnpm2nix code later and see if we can integrate some of it. Thank you!

[ambroisie]

@lilyinstarlight is right, this does not make use of FOD (don't comment before drinking your morning tea folks!). I think I misread this call and invented a outputHash out of nowhere...

I'm not sure what the stance is on using something like pnpm store add for nixpkgs' purposes then, I'll leave that to the experts. It is exciting to see movement on this issue at least :-).

[MarcelCoding]

pnpm store add allows pnpm to build the node_modules folder (and store) itself while leaving the downloading and hash checking up to nixpkgs. It also still duplicates packages with the same version globally (thanks to nix), like the store of Pnpm is originally intended. Basically pnpm store add just links (idk if it copies, I will research that) the downloaded version by nix to the local Pnpm store and then linkes from the store to node_modules.

[dbaynard]

I had a question about copying, especially where hard links won't work.

idk if it copies, I will research that

I can't speak to the pnpm store add command directly, but afaict pnpm copies when it can't hard link. I have my nix store and the build directory on different partitions (the latter usually a tmpdir — I'm speaking generally to cover both Darwin and Linux). It would be a little frustrating, though understandable, if the packages were stored twice in the nix store: once from nix adding the package, and another for the resulting pnpm store.

a corresponding entry on /nix/store for every package. After that, another entry in /nix/store is created, which represents the store. The store is generated using pnpm store add. After that, pnpm install is used to generate a node_modules folder using the generated store.

So here the store in /nix/store can symlink, or hard link, but the pnpm install will always copy into the node_modules on a separate partition?

It is exciting to see movement on this issue at least

:+1: yes, thank you, all

[ambroisie]

It would be a little frustrating, though understandable, if the packages were stored twice in the nix store: once from nix adding the package, and another for the resulting pnpm store

Presumably this can happen all over the place in nix. Which is why nix store optimize hardlinks duplicate files.

I do agree that it would be unfortunate, but should not be a roadblock IMO.

[MarcelCoding]

pnpm store add allows pnpm to build the node_modules folder (and store) itself while leaving the downloading and hash checking up to nixpkgs. It also still duplicates packages with the same version globally (thanks to nix), like the store of Pnpm is originally intended. Basically pnpm store add just links (idk if it copies, I will research that) the downloaded version by nix to the local Pnpm store and then linkes from the store to node_modules.

My Research: If the store and the node_modules folder are on different partitions - e.g. /nix/store and the working directory of the pnpm install is on different drives (idk how that is with nix) - the required files from the store will be copied https://pnpm.io/faq#does-pnpm-work-across-multiple-drives-or-filesystems

If the store path is specified via the store config, then copying occurs between the store and any projects that are on a different disk.

[dbaynard]

So if a solution uses pnpm it will copy the packages from the store in the /nix/store into the node_modules in a build directory (often a tmpfs), and then the nix tooling will have to recognize that that these are copies and deduplicate afterwards, when putting the node_modules into the/nix/store?

[MarcelCoding]

Something like that, but nix optimize should replace the files with hard links and this copy would only occur during the build stage.

[lilyinstarlight]

Ideally we would symlink until the final derivation (where it would be copied since the data changes when running lifecycle/build scripts). Hard links aren't supported in NARs and store optimization isn't universally enabled. But also it doesn't really matter that much and shouldn't be a blocker

I'll weigh in later more thoroughly, though. I'm on mobile and also at $dayjob right now

[dbaynard]

No need for a more thorough explanation for my benefit: I think you've answered my question. And having run through the code, I see a number of calls to cp (some of which can be configured to ln).

More broadly, having now been through the code, it seems using the hash directly from the pnpm.lock file is 👍 — seems like an elegant implementation. And so it means the dependency deduplication should match the behaviour for pnpm?

It would be cool to be able to use a pnpm store in /nix/store for development with pnpm, too. That seems somewhat tricky, given nix would create a separate pnpm store for each package. But perhaps (again, not blocking!) one could implement a pnpm store server that used /nix/store packages.

Thanks, all.

[Edit: formatting, deduplicate > deduplication]

[MarcelCoding]

The pnpm store server thing sound interesting, I'll play around with it and keep you updated.

[v-morlock]

Great to see progress on that topic! one thing i've noticed when trying the pnpm2nix project linked above is that the don't seem to support configs in .npmrc like additional repos (fontawesome pro in our case) - might be sth to consider in the design of the custom solutions, though i'm sure that's doable :)

[nyabinary]

any updates on this end?

[MarcelCoding]

How about testing pnpm2nix in nixpkgs? There are already many packages linked to this issue which hare using pnpm.

[Scrumplex]

I have just created #290715 which moves the pnpm code from vesktop into a top-level function. It's far from perfect so far, but it should support some use-cases.

[hallettj]

It case it's helpful for anyone, I made a fork of pnpm-export-lock with a bunch of updates to get it into shape for converting the latest pnpm-lock.yaml format to yarn.lock. (I haven't done much testing on updates for generating package-lock.json outputs.) https://github.com/hallettj/pnpm-lock-export

[thenbe]

I'm not sure how relevant this is, but pnpm v9, released yesterday, included changes to the lockfile format:

• Lockfile changes:
  • Lockfile v9 is adopted. This new format has changes for better readability, and better resistence to Git conflicts.
  • Support for lockfile v5 is dropped. Use pnpm v8 to convert lockfile v5 to lockfile v6 https://github.com/pnpm/pnpm/pull/7470.

[Lord-Valen]

This is still an issue. While #290715 does add pnpm.fetchDeps, which is relevant to this issue, it does not add an analogue of buildNpmPackage for pnpm.

[NyCodeGHG]

An install hook like buildNpmPackage has should be pretty easy to do in theory using pnpm deploy, but it seems to have some issues right now (see https://github.com/pnpm/pnpm/issues/5315)

[doronbehar]

To me it is not that clear whether we need such tooling or not, I opened https://github.com/NixOS/nixpkgs/issues/317927 to discuss this. I hope it will be OK with you that I'll close this old issue in favor of https://github.com/NixOS/nixpkgs/issues/317927 , because it's much harder to read this issue and find an actionable thing to focus upon.