[RRFC] mechanism for opting in to experimental changes

[nlf]

Motivation ("The Why")

The npm CLI often has a desire to get new features and/or changes to underlying systems into the hands of users without waiting for a new semver-major to represent the potential for breakage in those changes.

In order to ship these things with minimal disruption, we would like to introduce a means of opting in to experimental features and behaviors. This functionality should work such that when the feature/behavior is no longer experimental, the presence of the flag does not create an error. This is important because our future config changes may lead us to throwing errors for unknown command line flags, which is undesirable here.

Example

As an initial example, we are working toward removing our existing HTTP agent implementation with a new in-house agent that better covers our use cases with regards to socket limitations, proxy usage, and timeouts. In theory this change should not be breaking, and ideally will not be, however due to a lack of visibility into the various configurations of our community we cannot ship it with full confidence that we won't break any workflows.

In order to get this new agent into our community's hands for testing purpose, we would like to gate it behind this new experimental feature flag. This will allow us to test thoroughly, especially with our users who are currently experiencing problems that should be resolved by this change, before it becomes the new default (or in some cases the only option).

How

Current Behaviour

We don't have a means of doing these feature flags today. Some things are configurable, however when we set new defaults or change existing behaviors in ways that we have low confidence will not be breaking we are forced to wait for our next semver-major release to ship the change as breaking and deal with bugs after the fact. This delays our feedback cycle, and leaves our community with potentially broken features for a potentially extended length of time.

Desired Behaviour

Users should be able to opt-in to these experimental changes with a command line flag. Current thinking is something like the following:

> npm install --experimental       # all experiments enabled

Known Risks

Changes to the set of experiments could potentially be considered breaking. We have some options for dealing with this in the future, however. The simplest solution would be changing the experiments in a new semver-major release of npm.

Alternatively, and likely more effectively, we can consider changes to the experiments to not be breaking and make it as clear as possible to the user that when running in experimental mode they may experience issues and if they do they should try the command again with --no-experimental. This flag is meant to opt in to changes that are not yet considered stable after all. The suggested communication here can be implemented via a notice log informing the user experimental mode is on and a verbose or silly log that tells us what features are associated with the flag in the current release.

[ljharb]

Presumably --experimental= would take a comma-separated list of experiment names?

Would it be useful if experimental was not provideable via config (env var or npmrc or package.json), only by CLI flag?

[wraithgar]

based on conversations at the last round table I think allowing it in the npmrc is exactly what some folks are going to want, so they can opt-into the mode always.

[ljharb]

As long as breaking changes introduced to the experiment are semver-major, i suppose that's probably fine - the risks will depend on what the experiment is, i think.

[nlf]

the consensus we reached internally and within the open office hours call where we discussed this flag was that it's unlikely a user will want to selectively enable experiments, the more common thing would be opting in to all experiments. our initial implementation will be a boolean only --experimental flag.

this does come with a risk that if we change the set of experiments that means a user could be unaware of what experiments are actually enabled. to mitigate this, i believe we should log a notice that experimental mode is enabled and a verbose log could also specify what those experiments are to help the user understand what may be going on. in addition we may need to treat changes to the set of experiments as breaking, as in new experiments will only be added in a new semver-major. since these types of changes should hopefully be somewhat rare i think this is an acceptable risk at this time.