Vault deprivilege, allow bring-your-own Vault

[bnevis-i]

🚀 Feature Request

Relevant Package

Affects security-secretstore-setup and HashiCorp Vault components.

Description

EdgeX currently uses HashiCorp Vault as its secret store and is moving in the direction of using Vault as the EdgeX instance's root of trust. Besides storing secrets such as database and MQTT credentials, EdgeX is looking to Vault to also provide service identity, and user identity and authentication for external API callers.

In order to provide this functionality, security-secretstore-setup is the one-stop shop for:

• Initializing and unsealing Vault
• Encrypting the Vault master key to a hardware root of trust (optional)
• Installing and configuring the various Vault secrets engines
• Seeding database passwords and MQTT credentials
• Creating and distributing Vault access tokens to the EdgeX microservices
• Assigning microservice and external user identity (future)

The EdgeX project is increasingly running into adopters that have their own Vault instances that they are running either on the edge or in their cloud, making the EdgeX Vault instance potentially redundant.

Describe the solution you'd like

While it is possible to have an edge-based Vault store credentials to an upstream Vault, a better solution would be for EdgeX to simply leverage an upstream Vault. The key to using an upstream Vault is to "de-privilege" EdgeX's use of Vault and modularize security-secretstore-setup to make many of its tasks optional.

The following is a list of current operations that security-secretstore-setup performs and the required changes to "de-privilege" these operations.

Initialize and Unseal Vault

If someone is bringing their own Vault, EdgeX obviously cannot perform this operation. This functionality should be separated into an initialize-or-unseal daemon, that only runs when EdgeX is managing the Vault instance itself.

Install Vault Secret Engines

Installing Vault secret engines is a one-time configuration step that should be separated out into its own script. Adopters who bring their own Vault instance will be responsible for running the script on their own to prepare the Vault instance for EdgeX use, or perform their own compatible configuration. If EdgeX is managing its own Vault instance, it needs to perform this step immediately after the initialization phase.

Configure Secret Engines

Several Vault secret engines require a separate configuration step. For example, the Consul secrets engine is configured with a Consul management token that allows for creation of Consul access tokens in a given Consul instance. Configuration of a secret engine is a one-time activity that can be done with special-use Vault access token with appropriate permissions. EdgeX needs to be able to perform such configuration whenever EdgeX owns the dependent service. For example, if EdgeX initializes Consul, EdgeX also needs limited permission to initialize the Consul secrets engine.

Configure Identities and Access Policies for EdgeX Services

Every EdgeX microservice must be made known to Vault and a service-specific access poicy attached to that service. Creating these identities and policies is a limited-privilege operation that could be offloaded to an administrator for manual provisioning, or delegated to EdgeX instances with an appropriately-scoped access token, depending on adopters' needs.

Credential Seeding

Credential seeding is a one-time limited-privilege operation that could be offloaded to an administrator for manual provisioning, or delegated to EdgeX instances with an appropriately-scoped access token,

Creating and Distributing Vault Access Tokens

In a fully-deprivileged scenario, EdgeX microservices authenticate themselves to Vault and receive an scoped access token in response. This is done by enabling one or more Vault authentication engines, such as AppRole authentication, Username/Password authentication, Kubernetes authentication, or other supported mechanism.

For a bring-your-own Vault implementation to be meaningful, EdgeX must no longer be able to arbitrarily generate Vault access tokens with arbitrarily attached access policies.

It may still be useful to delegate this task to a central component that can impersonate different EdgeX services in order to minimize code changes. SPIFFE workload attestation might provide a simpler alteranative to seeding a Vault authentication secret into each EdgeX microservice, or having a central component to pre-fetch Vault tokens for EdgeX microservices, as is done today.

Conclusion

In summary, security-secretstore-setup is a big monolith, and it needs to be broken up into a series of submodules that can be enabled or disabled as needed depending on how much Vault the customer wants to bring.

[bnevis-i]

Consider adding restart-on-failure to compose files with limited restarts. Right now if Vault fails, it won't be restarted.

[jpwhitemn]

FYI in support of this issue - we have now received a couple of adopter requests for this, but without all the needed details to know which way to fall on some of the requirements.