[Platforms] Managing service bindings and secrets

[joshgav]

:exclamation: If you're interested in supporting and driving this work please respond to this issue. Once we have enough interest we'll schedule an initial meeting to review current ideas and proposals.

---

Background

The Platforms group is going deep into capability domains for platforms to seek ways to reduce complexity and increase synergies and standardization across related projects, vendors and users.

One of the domains which many find hard to rationalize today is how to seamlessly bind workloads to the capabilities or services they rely on. That is, workloads must be able to authenticate and connect to databases, object stores, message brokers, observability systems and more and there are many different ways to accomplish this.

At a high level this binding is typically accomplished by 1) getting a credential and URL from the service; 2) securely storing that info in a secure store like Hashicorp Vault; and 3) injecting that info into workloads in specific formats and locations. However, it seems every service, every platform, every secure store and every workload type has a different way to accomplish these steps.

This leads to difficulty, confusion and delay for cloud developers and lots of discomfort and concern to cloud security practitioners about how secrets are managed. For example, many folks may not realize that some projects write secret data in Kubernetes secrets as it's transferred to the workload, a practice some enterprises don't allow.

We've jumpstarted discussion about projects and ideas to address this in this Google doc. I'll copy the two main sections here to continue discussion.

Please share other projects and ideas with us and if/how you can contribute. Thanks!

Related Projects and Services

• External Secrets Operator (ESO) - synchronizes secrets from an external store into Kubernetes' secret store (i.e., etcd)
• SOPS ("Secrets Ops") - secrets are encrypted and the encrypted strings are stored in plaintext. A master secret is maintained elsewhere and decrypts secrets before use.
• Service binding runtime and spec - a relationship between a service and a workload is declared; on reconciliation credential is retrieved from service and injected into workload
• Hashicorp Vault - static secrets are stored in an encrypted database; and/or dynamic secrets are generated on demand for specific service types. Secrets are retrieved by direct API call or injected into workloads by a sidecar.
• Azure Passwordless - automatically generate secrets for services and inject into workloads (need verification as it's closed source)
• Dapr secrets management - provides a standard API for retrieving secrets from any store, bypassing the Kubernetes store
• SPIFFE/SPIRE - inject a workload identity
• GCP Cloud SQL Auth Proxy
• vault-config-operator
• Tekton auth: https://github.com/tektoncd/pipeline/blob/main/docs/auth.md
• Cloud providers: Azure Key Vault, AWS KMS, GCP Key Management
• ArgoCD’s doc on secrets integration: https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/
  • How Argo manages secrets in its rendering engine: https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/#mitigating-risks-of-secret-injection-plugins
• kube-bind - binding CRD based APIs from providers to consumer clusters, with the ability to exchange secrets and other auxiliary objects, e.g. services.
• Vault Secrets Operator - syncs secrets from Hashicorp Vault into Kubernetes secrets

Ideas

• Focus on dynamic generation of short-lived credentials and transparent injection of those credentials into workloads (pods). Contrast with static secrets, i.e., simple key/value pairs that must be kept secret, like an API key from a service provider.
• Explore conventions for requesting short-lived credentials and locators from services on demand
• Could we have a standard API that given a CR* returns fresh credentials?
  • Compare Vault dynamic secrets engines
  • Compare knative SinkBinding and Binding duck type
  • Compare service binding "Provisioned Service"
  • How to avoid transiting a K8s secret? (some enterprises insist on this)
• Explore conventions for injecting credentials and locators into workloads in specific locations, e.g. as files at a specific path
  • Hypothesis: this will enable library authors to seamlessly consume secrets, reducing load on app developers to template secrets in specific files at specific locations.
  • Establish a convention on where to put binding info in workloads, e.g. as files in a well-known folder. Could be used in Vault agent and SBO.
• Establish a library of dynamic credential generators for different service types, a la Hashi Vault's dynamic secrets engines.
  • Use these in service binding runtime and other secret generators.
• Encourage user community to treat all secrets as short-lived and ephemeral. Encourage folks to avoid static secrets persisted in a store. If we get a library going, ask service and cloud providers to contribute "generators" for their services.
• Add SecretClass to Kubernetes secrets and enable alternate implementations, e.g. served from an external store rather than etcd. Compare to ClusterClass and StorageClass.

cc @monadic @raffaelespazzoli @sttts @sabre1041 @hiddeco @cmoulliard

[jkleinlercher]

What about bitnami sealed-secret?

[joshgav]

This is still a domain we're looking to simplify. Our next step is to set initial goals and activities and gather stakeholders to discuss. Let's keep working on this Google doc: https://docs.google.com/document/d/1-9ZU7yRB3oVKozIDO5FkkZQc1svvBj4ME8pUOqN9Mjw/

Wanted to add a couple other items that have come up recently:

• @cmoulliard has published an example of a binding broker and claim system as part of project Primaza, check it out: https://github.com/halkyonio/primaza-poc
• Hashicorp released a first version of github.com/hashicorp/vault-secrets-operator which has CRDs for several secret types that it syncs to the K8s secret store (i.e., etcd), almost using K8s as a cache
  • This ^ makes me think - perhaps we should have a SecretClass in K8s secrets and allow them to be backed by any of several implementations 🤔

[cmoulliard]

Many thanks @joshgav to take the time to grab projects/ideas and to highlight many of the points that we should improve around the binding which is a simple concept that all of us we understand but which is complex to implement and support as many systems/actors are involved :-)

The action to bind requires under the hood to perform the following actions that we are trying to support part of the Primaza project:

• Discover from a ServiceCatalog managed by a platform engineer a Service running in a cluster or to be deployed automatically using crossplane providers)
• Populate from the Service discovered the URL to access the service (IP or hostname + port, etc)
• Get the credentials associated to the service from a secure storage engine: vault kv2, etc
• Generate according to a contract definition the list of the key/value to be returned to a claimer (= actor issuing a claim to bind a service with a workload)
• Provide the information to be consumed to a workload (= pod) in a secure way to let a runtime to instantiate the proper object (datasource, jms broker, etc) able to access a service.

Remark: Different strategies should be reviewed in order to find the best approach(es) to pass the information in a more flexible way or less-locked model:

• Volume mounted using an ephemeral secret,
• Runtime lib able to get the information using a Claim REST service
• Support to encrypt the data transported from platform to the workload client
• Rotate and renew the tokens to informa the workload about changes regarding how to access the services
• etc

[szh]

I'm interested in potentially helping out with this. I'm a member of TAG Security and a maintainer of Conjur Open Source.

[abangser]

In trying to get alignment on current WG priorities, I am going to suggest we close this topic. But before doing so, I want to give 1 week for anyone who is interested to revive the focus on this, else we can reopen in the future when we have the right people/energies to commit to this topic.

Thanks!

[abangser]

Per my last message, I am closing for now but am happy to reopen this conversation if it becomes higher priority / has a sponsor to drive it!

[cmoulliard]

Instead of closing it, it should be better to move the request/issue to a future milestone