Architecture: Determine the best/most secure way to handle secure credentials in the Flow SaaS model

[snowzach]

We need to be able to provide Secure Configuration Storage of User's configurations.

We believe the most ideal tool to leverage is Hashicorp Vault. It's somewhat the standard in secure credential storage. This is attempting to outline the workflow and process for getting Secure Credentials in and out of Vault as well as how the Secure Configuration will be managed.

Definitions/Assumptions/Statements

• We will be storing Secure Configurations (SC).
• SC will likely be credentials but could be other sensitive configuration information.
• SC will be stored in Secure Configuration Storage (SCS) outside of the normal management database.
• The management database will store the existence of SC in the SCS along with some metadata but have no visibility into it.
• A Security Group is a logical container that holds Secure Configurations.
• A Security Group construct should exist in both the Management Database and the Secure Configuration Storage.
• A user is granted Read or ReadWrite access to a Security Group.
• Read access grants a user permission to leverage a SC.
• ReadWrite access grants a user permission to Update/Remove an SC. (User cannot view)
• A Security Group may be the same as a Tenant. If it's not the same, it must only belong to one Tenant.
• Management and visibility of SC in the SCS should be maintained on a per Security Group basis

Secure Credentials Creation Workflow

• Users are able to provide static SC with sensitive information via the management API.
  • Users assign one of their available ReadWrite Security Groups to the SC during creation.
• Management API stores Information, encrypted at rest, in the SCS
• Management API makes note the SC exists along with any metadata in it's database.

Secure Credentials View Workflow

• The management API enforces credential visibility for users based on security groups.
• The user is only allowed to see the SC metadata (not the actual SC data)

Secure Credential Usage Workflow

• When the Flow runtime is ready to leverage Secure Configuration it obtains a ReadOnly token from Secure Configuration Storage that is applicable to the Secure Configurations's Security Group.
• The Flow Runtime leverages the ReadOnly token to provide the means of fetching the Secure Configuration to the Flow Connector.
• The method for providing the Secure Credentials to the Connector is TBD

Day One Features

For initial development we'll start with Static Credentials.

Day N Features

Vault supports a myriad of plugins for obtaining temporary credentials that we can add support for in the future. These present some challenges though as it's plausible connectors could run un-interrupted for months and thus some credentials will expire. We need to manage these challenges and ensure that credential expiration and rotations are handled appropriately by the connectors.

[jgraettinger]

Other braindumps / notes:

For user ergonomics, the separation of secure vs unsecure connector configuration should be as automated as possible. For example, rather than having two places to manage secure vs unsecure config, we should understand what parts of a connector's config are secure vs which are insecure, and separate the two at build time -- leaving a placeholder for secure portions of the config in the unsecured version that tell us how to resolve it later.

Which raises the question "How do you know what parts of the config are secure"? Ask the connector! The spec can be marked up with an annotation that distinguishes fields which are secure. This is also useful for UI form presentation (starring out secret fields). Airbyte's already done this with an airbyte_secret annotation.

There's a related question to this design which is "how should I manage credentials in my GitOps catalog" ? Solution sketches today include some existing tooling, like git-secret & others. Or, the user can use a UI workflow to create the connector. I don't think we've "solved" this issue either, but I do think we should consider it as a separate problem.

[snowzach]

@jgraettinger good thoughts! At this point in the process, the biggest question in my mind is still "How will the secure credentials be provided to the connector". I've come up with 4 options that each has it's own advantages and drawbacks. I wanted to write them all down with pros and cons before documenting here or bringing to the team. I am working on that now.

Basically the challenge (in my mind) is balancing:

1. Simplicity.
2. The Flow Runtime having to touch Secure Credentials. (In that ideally it doesn't need to)
3. Credential renewal/cycling.
4. Compatibility with the orchestration platform.
5. Needing to make changes to the current adapters.

[snowzach]

Use Cases

These are the different things I believe will require being stored in the Secure Credential Storage.

• Connector Configurations - The secure portions of the connector configuration

  • Database Credentials / API Credentials / Service Accounts / etc
  • Hostnames - It's plausible this could be considered sensitive as well

• Internal Orchestration Management Credentials - We will likely also want to secure platform internal secrets as well. Possibly not all of them but certainly some.

  • Google Service Accounts
  • AWS Access/Secret Access IDs
  • Kubernetes Services Accounts (if not automatically delivered)

NOTE: We have pretty much already decided to use Hashicorp Vault at this point, I will just refer to it as Vault from here on out.

[snowzach]

Credential Handling Options

For storing credentials, in pretty much all cases I believe the Management API has the job of saving Secure Credentials into the Vault. It should also make note of the credentials metadata (created/updated/name/security group/etc) in it's database.

The method for fetching and using credentials could be one of the following:

Flow Runtime Managed

In this scenario the Flow Runtime will communicate directly with Vault having full access to the credentials inside. When running a connector it will inject/merge the secure configuration with the relevant calls to the connector. If the credentials change, the Flow Runtime can restart the connector with the new credentials using the appropriate lifecycle. Flow also uses Vault credentials to launch connector runtimes.

Pros

• Explicit control of connectors and secure config from Flow Runtime
• No change required to connectors

Cons

• Flow runtime might need powerful credentials able to read across many secrets
• Flow Runtime is managing credentials
• Added complexity to Flow Runtime logic

Secure Credentials Injected at Connector Runtime

In this scenario the credentials are injected out-of-band to the connector runtimes via either environment variable or configuration files. Something like Vault Agent is used to fetch credentials and provide it to the connector.

Pros

• The Flow Runtime never needs to manage credentials other than ensuring the Agent runs as part of runtime.
• Credentials are fetched and provided very close to where they are being used.

Cons

• Requires modification of the connectors to get configuration from file/environment
• Requires connector runtime supports running another process like the Vault Agent adjacent to the connector runtime.

Secure Runtime Connector

Another idea is to create a new connector type which I'll just call the Secure Runtime Connector = SRC. I was thinking this connector could be similar to the FlowSink connector but serve several purposes, the key ones being managing connector runtimes, securely proxing flow connector communication and fetching/providing Secure Credentials.

• Flow Runtime communicates with SRC via network gRPC.
• SRC runs actual collector/materialization connectors using a variety of built in drivers such as:
  • Kubernetes - For trusted connectors
  • Firecracker / K8S+Kata - For untrusted connectors
  • Docker/ContainerD - For bare metal / VMs / Customer Prem installations
  • Future...?
• SRC securely fetches credentials from Vault and Injects them into the relevant RPC calls.
• If credentials change, attempts to terminate/restart in process RPC's at appropriate point in lifecycle.

Pros

• Very little change to Flow Runtime and the runtime does not need to handle credentials at all.
• Can be provided specific access tokens to only fetch the credentials it requires.
• Secure Credentials are fetched as close to connector as possible (without being in them which is better but harder).
• Each instance could possibly handle multiple streams and also be horizontally scaled.
• Could allow deploying connector runtimes elsewhere and only exposing the SRC's port as a secure gateway.
• Upgradable outside of Flow Runtime (Add a new runtime type with no change to Flow)
• Could facilitate suspending/sleeping connectors for runtimes that support it while the Flow Runtime/SRC gRPC stream remains active.

Cons

• Adds another layer between Flow Runtime and Connector
• Should Flow Runtime be doing this job?

Example Architecture Options

• Cloud Native Connections
• Cross Cloud Connections
• Customer Prem Connections

[snowzach]

Questions

• When pulling Secure Configuration from Vault, what token(s) are used to fetch data. Can it be limited in scope to just the credentials needed or just the Security Group?

• Can credentials be further wrapped by vault and only unwrapped by the last hop of trust?

• Can the Vault credentials be provided by the orchestration platform (Kubernetes Service Accounts maybe?)

• Other Features/Options

• Tokens can be locked to CIDR ranges

• Multiple Vault Plugins for Temporary Credentials

[jgraettinger]

We had an in-person conversation where we walked through various options. My understanding of the consensus:

We'll use Hashicorp's managed Vault service to power storing, retrieving, and auditing of credentials used within Flow. We'll seek to altogether avoid storing any credentials on our infrastructure by 1) separating credentials from configuration early in the catalog build process, 2) delegating credential storage to Vault, and 3) retrieving credentials and re-hydrating full connectors configurations at the very last mile: immediately before invoking the connector and only within the connector's isolation boundary.

Secure Connector Proxy

We'll develop a secure connector proxy (working name still TBD): a static binary which is placed inside the isolation boundary of the connector. Today, this means it runs within the Docker container of the connector. In the future, it means inside a Firecracker VM dedicated to the connector. It's most similar in spirit to the secure connector runtime discussed above.

This proxy will directly execute the connector as a subordinate process. The proxy will interface with the delegate connector using the delegate's chosen protocol (Airbyte spec over stdin/out, or Flow's native capture or materialization protocol, or anything else we use in the future).

It will similarly interface with the Flow runtime using Flow's native capture / materialization protocol (only). We'll consider the means of integration with the runtime to be a change-able implementation detail. Today it would speak the native protocol over stdin/out, but in a Firecracker future it might be simpler to run gRPC over an exposed port.

The secure connector proxy will perform last-mile re-hydration of secure credentials which are separated from a connector configuration at build time.

Build-Time Credential Separation

The control plane's build API accepts HTTPS POST'd catalog specifications which include connector configuration. That configuration includes a mix of less sensitive configuration as well as sensitive credentials. The build API will utilize schema annotations returned by the connector to identify locations of the configuration which represent secure credentials. It will extract credentials and push to Vault for storage, and then re-writes the configuration to replace credentials with references to a credential now stored in Vault.

Runtime Credential Injection

Immediately prior to starting a connector, the Flow runtime will obtain a wrapped token from Vault. The wrapped token will be passed to the secure connector proxy during initialization of the stream (as a field of the CaptureRequest of a capture, or TransactionRequest.Open of a materialization). Already today, these messages include the connector configuration, but this configuration will not yet include actual credentials.

The proxy will directly interact with Vault using the wrapped token, will fetch credentials, will merge them back into the connector configuration, and will then pass this restored configuration to the exec'd connector delegate.

(@snowzach you had a nice diagram of this ⬆️ workflow; mind attaching here?)

Limitation of Scope

At this time, we are not trying to design a general credential management feature which our users would directly interact with. Our use of Vault would be limited to a) writing credentials during a catalog build, and b) reading credentials for purposes of running a connector. There would not be a capability to fetch & expose a credential to a user, for example.

We are not trying to provide an opinion about how users should manage credentials within a Flow GitOps workflow -- there are a number of options for this, and we're still evaluating them ourselves.

We may introduce some form of user-facing credential management in the future, but expect it would be a separable concern from this secure credential delivery design. That's because a credential management feature would ultimately still need to produce and present fully-merged connector configurations to the build API, which would then separate and isolate credentials of that configuration using this design.

[snowzach]

Here's the current thought process around how the Secure Connector Proxy would work

[jgraettinger]

Closing because we've achieved the goal of having an architectural consensus. Still more to figure out, but next steps are to start prototyping.