Epic: Support writes in the UI for declarative backends (Git)

[GeorgeMac]

This is more of an epic issue detailing the challenges and the ideas we've had internally around supporting writes in the UI for the declarative backends for Flipt. We can spin more concrete items off of this as and when we settle on a more complete vision.

I will treat it like a living / WIP issue and update it as we go. This is a big piece of work and will require changes in all the places. We put this out here to get the communities thoughts on the idea and your help solving the problem. Please do come and share if you have thoughts as soon as. Don't wait for it to be marked non-WIP.

There will be lots of interesting technical problems that fall out of this. Let us know if you want to take a stab at them.

Some TL;DR questions and problems areas we need to tackle:

• How can we support both old and new UI flows simultaneously?

e.g. can we leverage Redux to switch out the persistence changes?

• How valuable would a first-class Proposal concept be to you?

e.g. something which tracks associated GH PRs and raise proposals in the UI

• Should we eventually support this new UI flow with the relational backends?

or does it supercede them and make having a relational DB just an additional, unnecessary burden?

• How should we address conflict resolution concerns?

automatically re-apply? reject via base revision? see below

The Problem

To date we have four declarative backends:

• Local
• Git
• Object (s3)
• OCI

The original goal of these backends was to enable GitOps workflows for feature flags. When either of these is configured as a backend for Flipt, the UI enters into a read-only mode. In fact, the implementations of Flipts storage APIs return "not implemented" on all mutating methods.

Ultimately, we would like to create a unified UI experience across these backends. One where you can make edits in the UI and them be realized (in some manner) in these other backends. To start with, we likely might benefit most from supporting the Git backend. The reason being that backends such as Object and OCI will benefit from this indirectly through the Git backend. These backends can be contributed too via CI, meaning writing directly to Git from the UI will feed the state to these backends. We may never have to implement them directly, it depends on what the end experience turns out like and folks desires.

This issue tracks some thoughts on what needs to change, some assumptions and invariants we would like to maintain, as well as some ideas around approaches to tackle it.

Existing API Limitations

Flipt's existing management APIs were designed around the relational backends and the UI to management API can be considered an imperative contract. Whenever an operation occurs in the UI (create flag, update variant, re-order rules) they take effect immediately.

While this is technically achieveable, it doesn't map effectively onto these newer backends. A one to one mapping with these operations and, for example, Git commits would have technical issues, undesirable performance and an incompatible experience with GitOps practices.

For example, mapping an single operation into a commit puts a responsibility on the server to have to handle automatic conflict management. While there may be a suitable / sensible approach, during concurrenct contributions, the server will have to handle replaying changes until it succeeds. This will create some unpredictable write performance characteristics and be cumbersome to management on the server side. Additionally, this would require contributing directly to a trunk branch. This seems at odds with GitOps practices as it completely omits the opportunity for review process.

Additionally, a single one of these imperative operations (e.g. add rule) while technically valid, can and will leave your flag state in an not-yet desirable state. Taking rules as an example, you first have to create a rule, which appends it to the rule set. Then you have to re-order the rules to get them in the effective order that makes sense for your usecase. The time between creating and re-ordering is up to the user. During this time, the flag might produce unexpected results due to the currently undesirable rule order.

One of the benefits of the declarative style is that you can commit the entire flag state atomically. There exist relationships between resources in Flipt, and this style of state management has much better atomicity characteristics for downstream consumers. The new declarative style backends, would be (unsurprisingly) more appropriately managed via a declarative API.

Ideas

1. Commits

sequenceDiagram
    actor C as Client
    participant F as Flipt
    participant L as Git (local)
    participant R as Git (remote)
    C ->>+ F: PUT { changes: [] }
    activate F
    activate L
    F ->> L: Checkout trunk
    activate L
    F ->> L: Add changes
    F ->> L: Commit changes
    deactivate L
    F ->> L: Push trunk
    L ->>+ R: Push
    R -->>- L: success
    L -->> F: new revision
    F -->> C: updated state
    deactivate L
    deactivate F

One thought we have often thrown around is the idea of the existing operations in the UI becomming uncommitted / staged changes instead.

In this world, the UI would present some base committed revision state merged with any uncommitted, queued up changes. Each operation would append to the set of uncommitted changes. Then we would require the user to have to commit their changes explicitly.

We would need to add a new endpoint in this scenario. The endpoint would take this batch of operations and turn them into an atomic contribution to the respective backend.

The following are some ways we could structure this new endpoint:

a. Batch of existing (imperative) request operation types

In this scenario we leverage all the request payloads and operations we have today. We take these types and create a batch out of them directly.

This has some potential simplicity benefits for the UI. Existing operations mostly just become an append operation. Though the UI still needs to materialize the base state + apply each operation to it to make up the viewed state.

This also has benefits in the form of these operations being something we can enumerate easily in some additional staged changes UI component. Imagine a sidebar with the list of uncommitted operations. For example:

• create flag
• add variant
• add rule
• add rule
• order rules

PUT /commit/v1/namespaces/{namespace}

{
  base: "<revision>",
  changes: [
    { action: "createFlag", flag: { key: "a" } },
    { action: "addVariant", flag: "a", variant: {} },
    { action: "addVariant", flag: "a", variant: {} },
    { action: "addRule", flag: "a", rule: {} },
    { action: "addRule", flag: "a", rule: {} },
    { action: "orderRules", rules: [2, 1] }
  ]
}

b. Batch of desired (declarative) states

In this scenario we could instead just expect a set of desired states (e.g. PUT/DELETE flag and segment). The UI could still operate as it does in (a), by tracking operations to be applied. This might have benefits to show e.g. the staged changes UI component. Also, might help if we want to support undo operations for uncommitted changes.

However, instead of submitted the operations themselves, it would instead translate these into a set of desired states. For example, entire snapshots of the desired flag or segment state.

This could have benefits for the simplicity of the server implementation. Particularly around automatic conflict resolution via replaying batches over later revisions.

As well as open the door for this operation to simplify supporting capabilities like server-side import. (related issue, we discussed that server-side import could support this kind of capability https://github.com/flipt-io/flipt-python/issues/2)

One outstanding thought here is how deletes work in this mode. This would likely required the verb to be part of the payload, to support the batch containing both puts and deletes. Would this be at odds with a declarative API? (maybe not).

In both approaches, it may or may not be useful to also require some revision identifier (e.g. base SHA). This would match the revision serving as the base of the UI changes. The purpose of this may be to support atomic commit operations that could either be rejected if the target reference does not match the base revision on commit. This could be one strategy for conflict resolution (see below) (or rather conflict avoidance).

PUT /commit/v1/namespaces/{namespace}

{
  base: "<revision>",
  state: [
    { verb: 'PUT', flag: {} },
    { verb: 'PUT', segment: {} },
    { verb: 'DELETE', flag: "a" }
  ]
}

2. Proposals

sequenceDiagram
    actor C as Client
    participant F as Flipt
    participant L as Git (local)
    participant R as Git (remote)
    participant S as SCM (e.g. GitHub)
    C ->>+ F: PUT { changes: [] }
    activate F
    activate L
    F ->> L: Checkout branch
    activate L
    F ->> L: Add changes
    F ->> L: Commit changes
    deactivate L
    F ->> L: Push branch
    L ->>+ R: Push
    R -->>- L: success
    L -->> F: new revision
    deactivate L
    F ->>+ S: Open PR for branch
    S -->>- F: PR
    F -->> C: Proposal
    deactivate F

Another consideration is how to support the concept of changes being proposed instead of directly committed. Imagine, for example, changes being proposed via a new branch and a PR being automatically opened. This would be instead of e.g. directly committing to some trunk branch like main.

This has benefits for GitOps workflows, where the UI becomes a tool to improve the contribution experience. However, changes still flow through the internal PR process.

The new endpoint in this scenario would be functionality the same as in the committed section. However, it would return some reference (PR link) or perhaps we would have some new entity (e.g. Proposals). This entity would have its own page, so you could track the proposals status directly in Flipt. The proposal would track the state of the PR in the backing SCM (e.g. GitHub / Gitlab).

3. Conflict Resolution

With the relational backend concurrent updates and atomicity is managed via the RDMS guarantees. Given we support proposing and comitting changes, this will likely have consequences w.r.t resolving conflicts.

With committing directly (1) to some trunk branch in Git, there will come a time when the upstream rejects a push for new commits. This is because we will only perform fast-forward only pushes (no --force). Otherwise, we risk corrupting the upstream repository of changes both relevant or not to Flipts only purposes (note that Flipt can operate over a repository containing other app contents).

In this scenario we could do a few things:

(a) Reject the operation.

Push back on the client to update their state and try again. We could achieve this by required the base revision identifier be provided with the operation. We would base the new changes on this reference and reject the operation if we can't fast-forward on the remote.

(b) Rebase the physical commits.

We could simply attempt the push, if it fails, rebase and try again. Rinse, repeat until we succeed (or some operational timeout / retry limit). Potential for corruption here due to bad rebase? Poor rebase could easily lead to semantically or syntactically invalid changes to be committed. Also, some complexity here dealing with rebase conflicts in general (we will need a strategy).

(c) Replay the changes automatically.

We could simply always play the operations from the current HEAD of the target reference. This creates a last-write win approach to updating state. However, the atomic changes are still preserved as linear history. Previous commits will be restorable.

We would only reject if the operations end up non-commutable (e.g. update rule after delete rule leads to rule not found). This could be mitigated against by embracing [1 (b)] and making the unit of change the entire declarative state.

(d) Embrace proposals and leave it to the consumer

If we embrace a proposal model, we can push this reponsibility onto e.g. the SCM and their concept of PRs or MRs. The use then handles conflicts in this external system.

[GeorgeMac]

cc @jalaziz as I know we spoke about this in the past. Might be of interest to you 👍

[jalaziz]

Super excited for this.

Some thoughts on conflict resolution:

• I'm personally a fan of letting that be handled by git / PR processes + checks for a valid feature flag config.
• In the UI, I think you could start simple and simply show a basic diff if there's a conflict and give the user the option to pick one or manually edit.

Another thing that I'm not sure if you're thinking about is the ability to incrementally update the file. In particular, if I have comments and a specific ordering of flags in my committed feature flag config, Flipt shouldn't completely override the file and drop comments or re-order things. I'm not sure how easy this would be using the libraries that are currently used, but would be nice for UI operations not to be destructive of manual formatting changing.

From a UI perspective, I really like the idea of building up a set of changes and then opening a PR and/or committing directly to a branch.

[GeorgeMac]

Keeping it simple is a great shout for first pass. Might even be that we don't even have conflict support, maybe we just choose the PR route and eventually bring more of that up into the UI.

Really good shout on the comments preservation. That needs some thought, but would love to make that work.