Implement database feature flags

[jnaviask]

Description

Currently we use environment variables + webpack define plugin to handle feature flags. This is a fairly slow + inflexible system, and introduces gaps, e.g. requiring a rebuild to set a client-based feature flag, which can be out of sync if the same flag is used on the backend, makes it difficult to turn on and off, etc.

In #5217 @kurtisassad introduced Unleash as a feature flag service -- we may want to proceed with that as our primary options, but we should evaluate it in comparison with the project scoped out here, which avoids external dependencies but still meets our general requirements on the engineering side.

Engineering Requirements

Briefly, the work involved consists of the following steps, which can be broken out into separate tickets as needed.

1. Add table FeatureFlags consisting of key/value and associated sequelize models + migration
2. Load feature flags from database into memory at app startup
3. Add backend utility exposing a featureFlag() function that takes a key and returns a value if exists
4. Add section in /status to return currently loaded feature flags to client (in form of utility function featureFlags() that accepts the request object and for now just returns the flags in memory -- later we can filter by user, ip, headers, etc if needed)
5. Add feature flags store to frontend that exposes a useFeatureFlag() hook for components
6. Add script yarn set-feature-flag that takes a key and value argument and sets (or unsets) the provided flag in the database
7. Add migration to insert existing feature flag configuration into database + replace all usages + remove env vars in app environments

[Rotorsoft]

I've only used LaunchDarkly to manage feature flags, but in general we should prioritize:

1. Dynamic Configuration - So we can toggle flags in real-time without requiring code changes or redeployments. This way it's easier to experiment and rollback.
2. Integrated with Logging/Monitoring - So we can track the usage and performance

Not sure why we want to integrate FF with our db. Storage (loading) should be part of the functionality provided by the FF utility (Unleash?), and we can expose it to our app via a FF interface.

• We can use hexagonal ports and adapters to create a clean abstraction and leverage the provider SDKs to build the adapters (with the ability to change providers)
• We can use several strategies to react to FF changes in real time:
  • Client polling or FF SDKs that signals a change in the configuration. The client app should be able to refresh accordingly.
  • Server side polling can work in a similar fashion
  • We have to consider cache invalidations
  • Attach feature flags to user sessions in case we need to test different configurations (by user)

[jnaviask]

@Rotorsoft my question is: why do we need a separate, external utility when we can create a simple version that meets our needs without involving new services?

I think hexagonal architecture is a good fit here. If you want to take this on, let me know and I'll assign you this ticket, which you can edit to describe how we should best proceed. You can also work off @kurtisassad's #5217 if needed.

[Rotorsoft]

This is another generic problem we can try to solve (reinvent)... or immediately leverage a lot of features from existing solutions. We can start with our own (mocked) adapter to see how far we can go with it, just concerned about diverting resources to building something that's not adding value to our customers.