Feature Request: Secret as Code

[howieyuen]

Feature Request

Let me introduces 2 scenes here:

1. some apps store ak/sk in there env variables
2. some apps need cert file which is stored in k8s secret

Since Konfig is a public repository, it is obviously inappropriate to store the sensitive information in the above scenarios in a large repository. Therefore, this issue explores solutions for maintaining sensitive information in a large repository.

[howieyuen]

SealedSecrets

SealSecrets is consist of:

• cluster controller: sealed-secrets-controller
• user CLI: kubeseal

SealedSecret object has a template section which encodes all the fields you want the controller to put in the unsealed Secret. This includes metadata such as labels or annotations, but also things like the type of the secret.

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: mynamespace
  annotations:
    "kubectl.kubernetes.io/last-applied-configuration": ....
spec:
  encryptedData:
    .dockerconfigjson: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
  template:
    type: kubernetes.io/dockerconfigjson
    # this is an example of labels and annotations that will be added to the output secret
    metadata:
      labels:
        "jenkins.io/credentials-type": usernamePassword
      annotations:
        "jenkins.io/credentials-description": credentials from Kubernetes

After decrypting, generate a raw k8s secret:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: mynamespace
  labels:
    "jenkins.io/credentials-type": usernamePassword
  annotations:
    "jenkins.io/credentials-description": credentials from Kubernetes
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: mysecret
    uid: 5caff6a0-c9ac-11e9-881e-42010aac003e
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewogICJjcmVk...

As you can see, the generated Secret resource is a "dependent object" of the SealedSecret and as such it will be updated and deleted whenever the SealedSecret object gets updated or deleted.

Asymmetric Encryption

kubeseal uses an asymmetric encryption algorithm, and the encrypted result can only be decrypted by sealed-secrets-controller. kubeseal will fetch the certificate from the controller at runtime (requires secure access to the Kubernetes API server); or use kubeseal --fetch-cert > mycert.pem to store the certificate somewhere (e.g. local disk) and use kubeseal --cert mycert.pem is available offline. The certificate is also printed to the controller log on startup.

Encrypt/Decrypt Principle

Summary

• ✅ Easy to set up and use
• ✅ Unified management of Secret files
• ❎ Manual management of public keys and public disclosure of encrypted files
• ❎ Only used on Secret (eg ConfigMap)

References

• https://github.com/bitnami-labs/sealed-secrets
• https://aws.amazon.com/cn/blogs/china/managing-secrets-deployment-in-kubernetes-using-sealed-secrets/
• https://learnk8s.io/kubernetes-secrets-in-git

[howieyuen]

Helm Secrets

Helm secrets is capable of leveraging Helm to template secrets resources.

If you work in a large team with several namespaces and you use Helm already, you might find Helm secrets more convenient than Sealed secrets. If you work as part of a small team this could be a minor issue.

Helm secret has another advantage over Sealed Secrets - it's using the popular open-source project SOPS (developed by Mozilla) for encrypting secrets. SOPS supports external key management systems, like AWS KMS, making it more secure as it's a lot harder to compromise the keys.

With that said, Helm Secrets and Sealed Secrets share the same issues - to use them, you must have permissions to decrypt the secrets. However, if you want to reduce your blast radius, you might not want to hand over the keys to your secrets to every DevOps and Developer in your team.

Also, Helm Secrets is a Helm plugin, and it is strongly coupled to Helm, making it harder to change to other templating mechanisms such as kustomize.

Encrypt/Decrypt Principle

When encrypting a secrets file located in the helm_vars directory using helm secerts enc, the helm secrets plugin uses the public key to encrypt the content. When decrypting, use helm secrets dec to decrypt the encrypted content and add it to the values.yaml file. Subsequent uses can directly access the values in this file.

Summary

• ✅ Supports multiple backends and encryption methods
• ✅ Unified management of Secret files
• ❎ Complex configuration
• ❎ Manual management of public keys and public disclosure of encrypted files

References

• https://github.com/jkroepke/helm-secrets
• https://github.com/jkroepke/helm-secrets/blob/main/docs/Usage.md
• https://github.com/mozilla/sops
• https://github.com/variantdev/vals

[howieyuen]

External Secrets Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

The goal of External Secrets Operator is to synchronize secrets from external APIs into Kubernetes. ESO is a collection of custom API resources:

• ExternalSecret describes what data should be fetched, how the data should be transformed and saved as a Kind=Secret
• SecretStore specifies how to access the external API. The SecretStore maps to exactly one instance of an external API.
• ClusterSecretStore is a cluster scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces

They provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.

SecretStore

The following code example uses Vault as the backend and uses a static token:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      # vault kv put secret/foo my-value=s3cr3t
      server: "http://vault.default.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
data:
  token: cm9vdA== # "root"

ExternalSecret

The following code example creates an ExternalSecret and references the SecretStore created in the previous step:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: example-sync
  data:
  - secretKey: foobar
    remoteRef:
      key: secret/foo
      property: my-value
---
# will create a secret with:
kind: Secret
metadata:
  name: example-sync
data:
  foobar: czNjcjN0

Encrypt/Decrypt Principle

ESO synchronizes ExternalSecrets in the following ways:

1. ESO uses spec.secretStoreRefto find the appropriate SecretStore. If it does not exist or the spec.controller field does not match, it will be discarded and not processed.
2. The ESO instantiates the external API client using the credentials specified in SecretStore.spec.
3. The ESO obtains the secret value upon request from ExternalSecret and decodes the secret if necessary.
4. ESO creates Kind=Secret based on the template provided by ExternalSecret.target.template. Secret.data can be templated with secret values from external APIs.
5. The ESO ensures that the secret value is kept in sync with the external API.

Summary

• ✅ Supports various external Secret management systems
• ✅ Secret centralized management, flexible access
• ❎ Authentication exposure
• ❎ For Secret resources only

References

• https://github.com/external-secrets/external-secrets
• https://external-secrets.io/v0.6.1/
• https://www.infoq.com/articles/k8s-external-secrets-operator/

[howieyuen]

Secrets Store CSI Driver

Secrets Store CSI Driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

SecretProviderClass

The SecretProviderClass is a namespaced resource in Secrets Store CSI Driver that is used to provide driver configurations and provider-specific parameters to the CSI driver.

Here is an example of a SecretProviderClass resource:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.default:8200"
    roleName: "database"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"

Reference the SecretProviderClass in the pod volumes when using the CSI driver:

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault-database

Encrypt/Decrypt Principle

Summary

• ✅ Supports multiple providers
• ✅ Secret centralized management, flexible access
• ❎ Authentication exposure
• ❎ Only mount volumes can be used, not friendly to single-value encryption

Reference

• https://github.com/kubernetes-sigs/secrets-store-csi-driver
• https://secrets-store-csi-driver.sigs.k8s.io/
• https://learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver

[howieyuen]

Conclusion

According to the analysis and summary of the above solutions, the current solutions for sensitive information are roughly divided into two categories:

• Use cipher text of secret
• Use references to plain text of secret

Use cipher text of secret

If we use cipher text of secret, there are 3 questions to think about:

1. how to manage key pair?
2. who will decrypt cipher text?
3. when to decrypt cipher text?

Taking SealSecrets as an example to answer these 3 question:

1. kubeseal is responsible for cert rotation.
2. Cluster controller, sealed-secrets-controller will do the decryption.
3. After creating an SealedSecret object

In this regard, KusionStack needs to build a complete set of encryption and decryption suites. It can be placed in Runtime or used as an independent encryption and decryption component, providing a set of APIs to be integrated with kusion.

Use references to plain text of secret

If we use references, there are 3 questions to think about:

1. how to authenticate and authorize user identity?
2. how to reference a secret?
3. who will fetch the secret by reference?

Taking ESO as an example to answer these 3 question:

1. The SecretStore contains references to secrets which hold credentials to access the external API.
2. The ExternalSecret describes what data should be fetched, how the data should be transformed and saved as a Kind=Secret
3. Each supported provider, eg, AWS, Azure,Google, IBM, etc will fetched the secret and generate a k8s secret.

In this regard, KusionStack does not need to do too much extra work, especially the most critical encryption and decryption actions are handed over to the corresponding Secret Management to complete. KusionStack needs to provide schemas for connecting to various platforms in the Konfig library.

Summary

KusionStack needs the both 2 plans, for individual usage, user can is responsible for key pair management; for enterprise usage, we need an external secret management or even an identity service to control authentication and authorization.

[howieyuen]

How to do with crossing config management structure?

[ldxdl]

Comprehensive research and thought.

From my point of view, the important things are :

• Tool chain, workflow for both individuals and teams
• 'encode on client-side, auto-decode and install on server-side' is the most general use cases for all users
• Easy to install and start with
• For individuals
  • A out-of-box, K8s based and Kusion built-in solution is where to start from
  • Better local debugging support
• For teams
  • Multi-backends of secret manager/store should be supported, plugable and configurable as needed with Kusion
  • Provides better security through server-side auto-decryption. It depends on access control to prevent security issue with decode in client-side with multi-roles.
• A 'secrets' subcommand may be a good choice
• Reuse secret manager/store framework as possible

[howieyuen]

Use Ciphertexts: Secret Management Based on Go CDK

The Go Cloud Development Kit (Go CDK) allows Go application developers to seamlessly deploy cloud applications on any combination of cloud providers.

Usage: Vault Transit

# install
brew install vault

# run in dev mode
vault server -dev

# vault server
export VAULT_ADDR='http://127.0.0.1:8200'

# vault token
export VAULT_TOKEN='hvs.F9A6wK2FaaJbHkbw3nJ8sbuC'

# enable the Transit secrets engine
vault secrets enable transit

# create a named encryption key
vault write -f transit/keys/my-key

# encrypt some plaintext data using the /encrypt endpoint with a named key
vault write transit/encrypt/my-key plaintext=$(echo "my secret data" | base64)

# decrypt a piece of data using the /decrypt endpoint with a named key
vault write -field=plaintext transit/decrypt/my-key ciphertext=vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w== |base64 -d

TODO

Module 1: default secrets manager

It's a symmetric encryption algorithm, an out-of-box and built-in solution.

func defaultSecretsManager(str) *Manager

Module 2: cloud secrets managers

go-cloud/secrets support "awskms", "azurekeyvault", "gcpkms" and "hashivault".

func cloudSecretsManager(secretsProvider, encryptedKey str) *Manager

Module 3: unify default and cloud secrets managers

// Manager provides the interface for providing stack encryption.
type Manager interface {
    Type() string
    // An opaque state, which can be JSON serialized and used later to reconstruct the provider when deserializing
    // the deployment into a snapshot.
    State() interface{}
    // Encrypter returns a `config.Encrypter` that can be used to encrypt values when serializing a snapshot into a
    // deployment, or an error if one can not be constructed.
    Encrypter() (config.Encrypter, error)
    // Decrypter returns a `config.Decrypter` that can be used to decrypt values when deserializing a snapshot from a
    // deployment, or an error if one can not be constructed.
    Decrypter() (config.Decrypter, error)
}

Module 4: merge into the apply workflow

New steps:

1. new flag secrets-provider, URL parsed
2. createSecretsManager(secretsProvider str)
3. decrypt cipher texts recursively

Depends on:

• encrypt command: enc --secrets-provider="hashivault://my-key"
• decrypt command: dec --secrets-provider="hashivault://my-key"

Module 5: save secrets provider info to Project&Stack structure

# cloud secret manager
secretsProvider: hashivault://my-key
# defualt secret manager
encryptedKey: dmF1bHQ6djE6bWptUEpGNXg4ZVlDYTl1SVJqU2kwNGZucjArT2M3NG82MDhPZWVvajFLcVZOTmhmNUM2Zld5V1g4b0wxZ2Rsa3ZsRCt6UWRydGt6bkY5bG4=

References

• https://gocloud.dev/howto/secrets/
• https://developer.hashicorp.com/vault/docs/secrets/transit

[howieyuen]

Use References: Secret Management Based on Vals

The "vals" is a tool for managing configuration values and secrets. Supported backends:

Echo

ref+echo://KEY1/KEY2/VALUE[#/path/to/the/value]

File

ref+file://relative/path/to/file[#/path/to/the/value]

ref+file://absolute/path/to/file[#/path/to/the/value]

EnvSubst

ref+envsubst://$VAR1

Terraform (tfstate)

ref+tfstate://relative/path/to/some.tfstate/RESOURCE_NAME

ref+tfstate:///absolute/path/to/some.tfstate/RESOURCE_NAME

SOPS powered by sops

ref+sops://base64_data_or_path_to_file?key_type=[filepath|base64]&format=[binary|dotenv|yaml]

ref+sops://base64_data_or_path_to_file#/json_or_yaml_key/in/the_encrypted_doc

Vault

ref+vault://PATH/TO/KVBACKEND[?address=VAULT_ADDR:PORT&token_file=PATH/TO/FILE&token_env=VAULT_TOKEN&namespace=VAULT_NAMESPACE]#/fieldkey

ref+vault://PATH/TO/KVBACKEND[?address=VAULT_ADDR:PORT&auth_method=approle&role_id=ce5e571a-f7d4-4c73-93dd-fd6922119839&secret_id=5c9194b9-585e-4539-a865-f45604bd6f56]#/fieldkey

ref+vault://PATH/TO/KVBACKEND[?address=VAULT_ADDR:PORT&auth_method=kubernetes&role_id=K8S-ROLE

AWS SSM Parameter Store

ref+awsssm://PATH/TO/PARAM[?region=REGION]

ref+awsssm://PREFIX/TO/PARAMS[?region=REGION&mode=MODE&version=VERSION]#/PATH/TO/PARAM

AWS Secrets Manager

ref+awssecrets://PATH/TO/SECRET[?region=REGION&version_stage=STAGE&version_id=ID]

ref+awssecrets://PATH/TO/SECRET[?region=REGION&version_stage=STAGE&version_id=ID]#/yaml_or_json_key/in/secret

ref+awssecrets://ACCOUNT:ARN:secret:/PATH/TO/PARAM[?region=REGION]

AWS S3

ref+s3://BUCKET/KEY/OF/OBJECT[?region=REGION&profile=AWS_PROFILE&version_id=ID]

ref+s3://BUCKET/KEY/OF/OBJECT[?region=REGION&profile=AWS_PROFILE&version_id=ID]#/yaml_or_json_key/in/secret

AWS KMS

ref+awskms://BASE64CIPHERTEXT[?region=REGION&profile=AWS_PROFILE&alg=ENCRYPTION_ALGORITHM&key=KEY_ID&context=URL_ENCODED_JSON]

ref+awskms://BASE64CIPHERTEXT[?region=REGION&profile=AWS_PROFILE&alg=ENCRYPTION_ALGORITHM&key=KEY_ID&context=URL_ENCODED_JSON]#/yaml_or_json_key/in/secret

Google GCS

ref+gcs://BUCKET/KEY/OF/OBJECT[?generation=ID]

ref+gcs://BUCKET/KEY/OF/OBJECT[?generation=ID]#/yaml_or_json_key/in/secret

GCP Secrets Manager

ref+gcpsecrets://PROJECT/SECRET[?version=VERSION]

ref+gcpsecrets://PROJECT/SECRET[?version=VERSION]#/yaml_or_json_key/in/secret

Azure Key Vault

ref+azurekeyvault://VAULT-NAME/SECRET-NAME[/VERSION]

GitLab Secrets

ref+gitlab://my-gitlab-server.com/project_id/secret_name?[ssl_verify=false&scheme=https&api_version=v4]

Usage

CLI

# default config
# VAULT_ADDR: 127.0.0.1:8200
# VAULR_TOKEN: ~/.vault-token
echo "foo: ref+vault://secret/foo#/foo" | vals eval

# specify host and proto
echo "foo: ref+vault://secret/foo?proto=http&&host=127.0.0.1:8200#/foo" | vals eval

# specify address
echo "foo: ref+vault://secret/bar?address=http://127.0.0.1:8200#/bar" | vals eval

SDK

import "github.com/variantdev/vals"

runtime, err := vals.New(vals.Options{})
if err != nil {
    panic(err)
}

valsRendered, err := runtime.Eval(map[string]interface{}{
    "inline": map[string]interface{}{
        "foo": "ref+vault://secret/foo?proto=http&&host=127.0.0.1:8200#/foo",
        "bar": map[string]interface{}{
            "baz": "ref+vault://secret/bar?address=http://127.0.0.1:8200#/bar",
        },
    },
})

TODO

Step 1: label securing stack.yaml

name: dev
labels:
  kusionstack.io/secure: true

Step 2: replace before preview

// Replace ref secrets if needed
secure, ok := stack.Labels[projectstack.LabelSecure].(bool)
if ok && secure {
    if err := vals.ReplaceRefs(planResources); err != nil {
        return err
    }
}

Step3: helper command: evaluate reference secret

kusion eval ref+vault://secret/foo#/foo

References

• https://github.com/variantdev/vals

[howieyuen]

Common Sense: DONOT save cipher text or sensitive data in git repo.