[RFC] Moving to trunk based development and introducing feature flags

[marcins]

💬 RFC

We have a lot of complex features / pieces of work planned for Parcel over the foreseeable future. In order to be able to meet our goals of: being able to work on features in parallel, deliver value incrementally, and provide the ability to dogfood new / risky / complex features at scale before making them generally available, we need to shift the approach to how we do feature development and releases in Parcel.

The summary of this proposal is:

• we shift to a "trunk based development" approach - where changes are merged early and often, rather than living in long running feature branches - note that the current ParcelDB work is out of scope for this, as it is too far along to spend the effort to separate it out now, but future work of the scope / scale of PDB would definitely be in scope.
• we implement a feature flag system that allows control of the state of these features, and allows for a combination of configuration and CLI / API overrides for them to be controlled by external systems for internal use cases.
• we shift from a "nightly" build to a "snapshot" build that's driven off merges to the main (v2) branch - this means we can "immediately" (once a build runs) consume a release rather than waiting for a nightly to run.

Trunk based development

The main philosophy shift here is that rather than working on a feature on a branch until it is done / tested / ready for release, features are able to be released piecemeal onto the main branch and are protected by feature flags (see below for feature flags description).

This means that where changes are isolated in scope within a module, they are conditionally executed based on feature flag outcomes. Where changes are more broad and would require peppering conditionals throughout the code then a parallel implementation needs to be created and conditionally included / abstracted at a higher level. How this is achieved will be specific to particular use cases so is not prescribed by this RFC.

(Open question - what about changes that require new dependencies? Are there any issues with this? This doesn't apply to dependency bumps, those would not be possible to feature flag easily)

The advantages of this approach are that integration costs are lower - you're not needing to constantly keep a branch in sync with what's happening on the main branch. Parallel work from a baseline piece of work on the main branch is easier to achieve.

The disadvantage is additional overhead for complex changes of maintaining an abstraction or conditional code.

We believe the advantages for parallelising work and shipping changes sooner for internal usage outweigh the disadvantages.

Feature flags

We will introduce the concept of feature flags in Parcel. This will allow us to ship inert code in open source, that we are actively testing and stabilising in our internal large scale use cases. Without the ability to do this, in combination with the improvements to the development model above, it may become untenable for us to work in the open source repository as the source of truth.

Note for the purposes of this proposal feature flags will not be used to produce different binary output from Rust builds - at this stage if some new Rust feature is being developed that has a dependency on a new crate, that crate will need to be included in the build even if unused.

The main approach to feature flagging is largely inspired by other open source projects that take this approach such as React (https://legacy.reactjs.org/docs/how-to-contribute.html#feature-flags) and Relay (https://github.com/facebook/relay/blob/main/packages/relay-runtime/util/RelayFeatureFlags.js, https://github.com/facebook/relay/blob/main/compiler/crates/common/src/feature_flags.rs)

The proposed approach is:

• provide a strongly typed feature flag struct (as opposed to untyped arbitrary key/value pairs) - this will form part of the ParcelOptions on the JavaScript side.
• ensure feature flag state is available to both Rust and JavaScript code, including JavaScript plugins
• feature flags can be controlled via .parcelrc - with their defaults being set in @parcel/config-default, with a new top level featureFlags key.
• feature flags can be overridden with CLI arguments / options passed to the API - this allows integration with external systems for controlling rollouts (e.g. you could use a system like Launch Darkly to provide a feature flag value based on username, and use that to progressively rollout a feature by setting it's state when invoking Parcel)
• feature flags will need to contribute to cache state - changing feature flags should invalidate the cache (it would be too complex to try and scope feature flag cache to specific cache entries that are affected by that feature flag)

There is a spike implementation here: https://github.com/parcel-bundler/parcel/compare/v2...spike-feature-flags

The main considerations when trying out the spike:

• getting feature flags into ParcelOptions / PluginOptions - with support for defaults in config, with overrides in InitialParcelOptions - is relatively straightforward though has to be done in a few places to ensure types flow properly.
• we can pass feature flags into Rust code - though in some cases like for transform this currently receives it's options as an untyped JsObject.
• I tried a couple of different approaches for integrating into the Parcel CLI, and found that the best ergonomically was to specify feature flag overrides with a single CLI option --feature-flag.. e.g. parcel build --feature-flag myFeature=true --feature-flag someOtherFeature=false. The feature flag definitions are in an object exported from @parcel/feature-flags and can be validated that they exist in the CLI via a custom option processing function.

Next steps

• [ ] for the trunk based development part - mostly need agreement between the core team that this is how we'll work. The contribution guide will be updated to reflect the agreed changes.
• [x] implement "snapshot" builds on merge to the main branch
• [x] .. as part of the above, remove "nightly" builds
• [ ] implement the feature flags - initially they'll just be empty but we'll ensure all places options are used have this available, tests are updated, etc. Similarly to above - we'll add usage instructions / expectations to the contribution guide.

[jondlm]

Any thoughts about how we'll manage the lifecycle of feature flags? AFAIK there's no perfect solution to the problem of feature flag atrophy but it's nice to at least consider the problem up front.

Seems like a solid proposal to me 👍

[marcins]

Any thoughts about how we'll manage the lifecycle of feature flags? AFAIK there's no perfect solution to the problem of feature flag atrophy but it's nice to at least consider the problem up front.

Fair point. There's definitely a risk that we might end up with features behind a flag that for whatever reason don't "graduate" to real features for a long time. I suppose it's one of those things that we'll need to address as it comes up.

We've kind of already got this to some extent with some of the unstable_ features we added during our long migration. The main one I can think of is the "async runtime" stuff. Whether that was behind a feature flag or unstable_ prefix it's still just sitting there with no current plan for what needs to be done to "graduate" it.