Closed wesleytodd closed 6 months ago
(as @GeoffreyBooth said: The most important part of this is that it defines the end goals: no lockfile mismatches, no failed builds.)
To be clear, I don’t think it should be Node’s responsibility to fix bugs in package managers. If pnpm changing lockfile schema in patch versions annoys pnpm users, that should be on pnpm to fix. Reproducible builds are ultimately the responsibility of a package manager: we can pin a package manager version, for example, but if the package manager itself doesn’t support lockfiles or doesn’t use one or the user doesn’t provide one, then there won’t be a reproducible build. npm install
updates the lockfile, for example; it’s not a way to get a reproducible build. But we’re not trying to force that to change somehow; it’s not our job to control how package managers work, or fix the issues that users may have with them.
That isn’t to say that we can’t or shouldn’t ship a version manager for package managers, but the reason to do so is for the more holistic reason of giving the user control over their project’s dependencies, like they already have for the software defined in package.json
dependencies
and devDependencies
. And maybe that control is to avoid bugs in particular versions of those software, and to achieve reproducible builds if the package manager is used correctly, but that’s more the user’s concern than a goal of ours. We would need to ship and control our own package manager if we wanted to ensure that the goals of well-behaved lockfiles and reproducible builds are met; but unless we want to build and ship our own package manager or limit which package managers and versions can be chosen, I think all we can really do is provide the tools for the user to achieve these goals if they choose a package manager capable of achieving them and use that tool correctly.
Closing in favor of this as a PR in #594. Sorry for the last minute thing, I just didn't get a chance until now to clone down the repo on this laptop.
The History
Version Management is something the Node.js project has discussed a few times. Starting in < INSERT HISTORY OF WG HERE >. And more recently with the introduction of
corepack
for managing versions of package managers. While there has been a volume of discussions and code examples I do not believe there has been a cohesive technical document to bring it all together. This is my attempt at such a doc.Node.js and the toolchain it ships with is used throughout the SDLC (software development lifecycle) and has different needs at different stages of work. For example, it is highly uncommon for someone to use a different versions of anything within the same production deployment while is is highly common to do so for local development or CI.
This has led to a the Node.js project focusing on the well understood (but also highly complicated) aspect of simply supporting one locked version of the toolchain across the many supported architectures. This is in fact a required area of investment for the project as no matter the local dev toolchain a team or developer chooses, when in prod it is always necessary to support a secure and stable single version workflow.
This focus has left it to the ecosystem to solve for the local and CI workflows where many versions are commonly needed for a variety of tools. And when the ecosystem is in charge a wonderful variety of solutions emerge.
The Problem
A realistic and productive Node.js development environment cannot be setup with only tools shipped within the current executables. A version manager for both the
node
executable and a package manager is required.The Stakeholders
This proposal requires members of the community to work together. To help enable that, I propose we formalize the Package Maintenance Working Group (PMWG) to own this domain and be a shared space for Node.js contributors, tooling authors, and interested community members to work. That would help ensure that all stakeholders in the ecosystem had a well documented and clear idea of who to work with and how that process proceeds.
Once we have chartered the PMWG to take ownership of this domain we can expand on this proposal and other related document and policies around how the Node.js Project intends to move forward with regards to Package Mangers, version managers, and associated tooling which are vendored or included in some from with the Node.js distribution.
Use Case Scope
We cannot build all things for all people. To help keep focus within this proposal we want to focus on solving for one specific use case:
Users installing Node.js on their local development machine or in CICD process for the purpose of building.
This means we are not intending for this tool to be used to install on production servers. That is not only an already well covered use case, it is also not a place where dynamically installing many runtime or tool versions is a good or recommended practice.
Goals
Prior Art
There is alot here, adding a place holder to remember but not get distracted.
@TODO list out more about the current version managers and their approaches.
A Proposal
Node.js ships or endorses (either builds or adopts an existing project) a version manager with the following qualities:
node
and a small subset of ecosystem tooling for local and CI environmentsAs there is already a lot of prior art in the ecosystem. I will venture to make a clear proposal for an end user experience we hope to achieve without picking an existing tool or set of implementation details (hence the name
nver
as a placeholder):While this is the "bootstrap from scratch UX" there are many other important workflows which led to the development of
corepack
which we should also well support. One such is cloning down an existing project and ensuring that you have a properly setup development environment for working with it. Here is an example for that UX:You may notice in the above that the
npm
executable noticed that the project was not meant for use withnpm
and displayed a message to users. This proposal would include some collaboration by our esteemed colleagues at the various supported packages managers to implement a common stub which would be maintained by the Node.js project which implements a standard check. The inclusion of this stub would be a requirement for being included in the tooling. To achieve this we have been discussing a proposal in the Package Metadata Interop Collab Space to add adevEngines
field. More on that below.And lastly, most developers work on more than one project at a time. It is important for their productivity that it is easy to switch between projects and their associated tooling. Here is an example of working on multiple projects at once:
Declaring development engines
Currently the only widely used way to declare development tooling is the
packageManager
field. This only covers one aspect of the problem (the package manager) and is not designed in a way to allow for a more holistic approach more similar to the pre-existingengines
field. In reality many projects have more broad requirements for development tooling which they document in their project readme files or docs. Additionally, all projects which have native dependencies require an implicit version ofpython
based on theirnode-gyp
version. These things are not covered under the current implementation ofpackageManager
and they typically do not cause as much user issues when discrepancies arise because they are less common and not "the first thing people run" but they are equally important to get correct.To this point, I propose we work toward a slimmed down version of the proposed
devEngines
field which covers explicitly declaring these required development tools. I say slimmed down simply to differentiate it from the version currently documented in the issue (as of April 1 2024). The reason I propose slimming it down is to prevent the initial specification from defining ways tools should fetch these dependencies (aka dropping the download section). Here is what I think we should adopt for a first pass:With this slimmed down version, we can test out approaches with different tooling implementations for how we resolve, fetch, and install the dependency without forcing the hand of tool authors into one single direction. If they wanted to limit that
version
cannot be a git repository, that would be acceptable even thoughnpm
defines that as valid for a package specifier.Technical Approachs
There are three primary approaches which version managers can (and have, see examples) take to solving this problem:
PATH
manipulationWe will describe each in the section below and attempt to well document the pros and cons of each approach.
PATH
manipulationThis approach is the most commonly currently deployed approach to binary version management. It is how
nvm
and other popular tools work.Pros:
Cons:
In-process indirection
This approach uses a stub at the start of the program to load in the concrete implementation of the executable. This approach is the current
corepack
design and also has prior art incoffeescript
(see example).Pros:
Cons:
Links
Jumper Binaries
Jumper binaries are specially crafted executable entry points which find the correct binary to run as a child process. This was the original
corepack
design.Pros:
Cons:
Considerations Needing to be Addressed
npm link
across node versions, etc.NOTES:
equal vs equitable: Equality means each individual or group of people is given the same resources or opportunities. Equity recognizes that each person has different circumstances and allocates the exact resources and opportunities needed to reach an equal outcome.
@darcy: I need to fit this all in here and I am not sure how and it was easier to start just copying it wholesale to start.
@styfle: