Adding Feature Flags

[HalosGhost]

Background

Right now, we have two architectures (2PC and Atomizer), so when a new feature gets proposed, there is a high-probability that implementation will require touching two separate locations in the code. As we explore more architectures, or solutions which involve changes to the two we have, this problem is likely going to be more pronounced. It results in the following draw-backs:

• The possibility for architectures to get out-of-sync in feature-support.
  • Accepting this leads to an extra documentation burden (documenting the matrix of which architecture supports which features).
  • Avoiding this leads to an ever-increasing implementation burden (a new feature will need to intentionally interoperate will all previously-merged features).
• Testing with a new feature (say, B) but without an old one (say, A) is tedious.
  • Accepting this would require the following overhead: checkout a new branch, reset to a commit before A, cherry-pick the relevant commits for B, and fix conflicts.
  • Avoiding this leads to treating any sufficiently-fundamental change to an architecture being treated as a new architecture.
• The mental burden necessary to get involved grows dramatically with every new feature (as even knowing which architecture you want to use becomes much less intuitive).
  • Accepting this leads to fewer contributions.
  • (Even partially) avoiding this leads to dramatically increased documentation burden (trying to explain not just which architectures support which features, but why and which you might want to use for a given experiment).
• Supplemental tooling (e.g., docker compose configurations) is subject to more and more significant duplication/complexity/churn.
  • Accepting this may mean that we end up with a very large library of docker compose configurations (one for each of the now-many architectures), which also implies frequent additions of new tooling solutions for each new architecture/feature.
  • Avoiding this either results in a much more complex setup for the supporting tooling such that it can handle the complexity above, or simply not supporting some architectures with tooling as well as others (additional maintenance or documentation burden, or both)

In short, we know there are a lot more architectures and features worth exploring; to ensure project sustainability, we need to develop patterns and conventions for adding new features in such a way that we can minimize the potential combinatorial expansion shown above.

A Solution

Feature flags offer a potential avenue to refactor the code such that new features are gated behind a flag to change the system's behavior at run-time. This would allow, for example, many different implementations of a sentinel to coexist and for which implementation gets used to be selected at run-time (rather than build- or deploy-time).

Taking this possibility to its logical conclusion, an architecture becomes a particular set of components being run with a particular set of feature flags (e.g., 2PC and Atomizer could be partly merged and we could have a flat component list).

Pros

Assuming we could come up with a perfect feature flags system, it would enable the following benefits:

• Changes to current architectures (even deeply fundamental ones) only require touching the specific, affected locations.
• New architectures can be added by adding necessary flag-gated features to current components and adding any new components.
• Testing with a combination of features that hasn't been considered before becomes much more tractable (just set the feature flags and run the needed components).
• Contributors can care only about which other features a new feature/architecture needs to interact with (rather than with all of them).
• New contributors see the flat list of components and a small document detailing known configurations which result in particular outcomes is much more approachable.
• Supplemental tooling (as long as it can be taught to interpret/leverage the feature flags) can be much more ever-green (so long as the feature-flags system is capable of containing a new change, tooling may not need to change at all).
• May increase potential velocity of new contributions by minimizing additional technical debt associated with adding new features/components.

Cons

Still assuming that the system we adopt/implement is ideal for our use-case, there are some draw-backs:

• It becomes possible to run the system in multiple configurations that won't work (potentially, more options than ones that would work even).
  • E.g., not running an archiver if you want to run a variant of the Atomizer architecture will result in several failure modes.
  • This is specifically because feature flag solutions make most sense at the "how the program works" level (not the "what program gets run" level). It is possible we could additionally add a notion of dependencies between components (perhaps influenced by feature flags) to mitigate this, but such a solution isn't clear.
• Adapting to such a system requires significant changes to our current layout and tooling.
• The clarity and meaning of something being "an architecture" is muddied (this leads to benefits mentioned above, but does make architecture discussions/research more nuanced and as a result more difficult).
• Some changes might be too fundamental for the feature flags system to handle.
  • The obvious solution would be to "fork" the changed component (note: not a github fork, but rather a new component initially copied from the old one), but drawing that line becomes a nuanced decision to make.

Practical Issues

Beyond the costs and benefits mentioned above, there's a much more practical problem: feature flags aren't well standardized (because they operate inside the application-layer's logic, implementations tend to be very domain-specific). Additionally, using feature-flags may result in control flow in hot-path code (significant performance degradation).

Pre-made

There are off-the-shelf solutions we could try to adopt, but they each have their draw-backs; below is a list of known solutions we could adopt and known draw-backs (more will be added as they are suggested and evaluated):

• Backtrace's dynamic_flag: it's x86_64-specific (leverages inline assembly), and only supports boolean flags
• A great many command-line option parsing implementations: unifies settings (knobs to turn to change specific bits of behavior) and arguments (e.g., the client's newaddress command) with feature-flags (potentially signifcant behavior changes)
• Some work is underway to create a unified specification for feature flags in OpenFeature; it's still early days, but this may become the ideal option in the future.

Custom-built

This is the most sensible option in a lot of ways (we build it for exactly what we need and want to support), but has its own issues. Namely, we actually need to design and implement a feature flags system (which, though enabling future work, is orthogonal to our research purpose).

Conclusion

Ultimately, some version of organizing the code to handle the combinatorial expansion of features/architectures is probably necessary, but I do not know if feature-flags is really the correct solution (another might be to maintain separate branches for each architecture, though that also has obvious drawbacks). Comments, questions, clarifications, and suggestions are all welcome!

[davebryson]

@HalosGhost Why not simplify it by refactoring architectures into their own separate repositories? utils and 3rdparty would be a shared dependency. This could reduce the complexity, isolate testcases per architecture, reduce build time, and potentially simplify benchmarking. It also lays the structure for any future architectures.

[HalosGhost]

@davebryson this is a reasonable idea. It does somewhat guarantee that the architectures will not be in-sync with one another (perhaps that's ideal in this case because it means that interest will drive contributions to each architecture—rather Darwinian). It does mean that if a given change is useful for multiple architectures, it's quite a bit more work for contributors to ensure the change is ported to each one (a PR per architecture, at least; though, each PR is likely to be more self-contained and clear).

The only big drawback I can see is a documentation burden for tracking what architectures exist (for that decoupling, we'd probably want to make a documentation-only repo to track all the repos that exist). And, that's a drawback already present in some of the paths I laid out in the initial proposal. The slightly more contained version of this would be to have a branch per-architecture (which has a lot of the same benefits and a couple fewer drawbacks, but adds a discoverability problem that's a little more work to mitigate).

I also think the big win of feature flags would be that if you wanted to test the system, you'd only clone once, and just configure it to run as you'd like (regardless of what you'd be testing)—though, this assumes that it's actually possible to build the code in such a way that that level of configuration is reasonable.

@metalicjames, @anders94, @narula, I'd be interested in hearing all of your thoughts on this.

[HalosGhost]

Some more recent, relevant art: https://www.libelektra.org/home

Personally, this is one of the more compelling ones I've seen so far. Not quite sure it covers all our cases, but I'm exploring it.