Support storing of secrets safely in Zarf package

[runyontr]

Is your feature request related to a problem? Please describe.

The installation and configuration of applications at runtime can/will need to use secret/sensitive values that should not be publicly distributed. e.g. a License file to support Enterprise Grafana/Loki in the BigBang deployment.

Describe the solution you'd like

I think 1. (from below) sounds like a better end-user experience, but adds significant complexity and adds complexity to building/defining the package. It would also allow tools that inspect Zarf packages to potentially function on the object that's shipped, rather than needing to decrypt to inspect the package

Describe alternatives you've considered

1. Encrypted Values

Allow the inclusion of an encrypted values file in the Zarf package. This would require decryption at install time and support of potentially multiple encryption/decryption paradigms depending on the use case (e.g. gpg, passphase, x509, etc).

Pro:

• The Zarf package looks like a Zarf package as its shipped and could be inspected/viewed

Cons:

• Would need to support multiple tools for encryption/decryption as end-user requirements change for supported file encryption

2. Sensitive Package

Treat the Zarf package as sensitive, and use other tools to manage the encryption/decryption process outside of the Zarf lifecycle. This would also allow support from any end user's encryption/file sharing platform without needing to support plugins/extensions/etc.

Pro:

• No changes required to the Zarf package. Would leverage existing (end-user/enterprise) encryption processes for end users to distribute the Zarf Package as a secret

Cons

• The object that's shipped doesn't look like a Zarf package and would require decryption to inspect/view the object and its non-sensitive contents

Additional context

• The parallels the use of SOPs in GitOps installations.

[anoncam]

@danfedick

[ntwkninja]

Storing secrets safely boils down to key management (bottom turtle). Where are the keys used to encrypt / decrypt stored, how are they protected and when are they rotated? We should have a discussion around key management since it would arguably be the most opinionated and critical design choice.

How we get secrets into k8s has a lot of options. Many enterprise models are shifting a refs to secrets model (vs. encrypted secrets stored in git) which generally supports more robust / scalable secrets providers that handle keys rotations, secure storage, etc.

I propose we create a similar model for portable / non-enterprise use cases to:

• establish a consistent user experience for both enterprise and non-enterprise configuration of secrets
• establish a common pattern for interacting with secrets
• allow operators (IT admins) to focus on key management

This does not solve the problem of key management for air gap / non-enterprise but rather standardizes how secrets are handled in deployment manifests regardless of the backend chosen.

As a use case for air gap / non-enterprise, helm-secrets using the vals backend with SOPS/AGE could do the trick. This would make helm secrets easier to work with (i.e. I don't have to encrypt an entire stanza) and support gitops-based secrets management. This also makes the configuration more discoverable (i.e. an entire stanza of config isn't obfuscated by encryption). To use this in a declarative approach, the backend should be configured to write once-read only which would would force an update to the secrets ref anytime a new secret was created in the backend (i.e. don't allow updates to secrets since this would obfuscate changes in git).

Note: Need to investigate performance of helm-secrets if that is a desired model.

Refs https://cloud.redhat.com/blog/a-guide-to-secrets-management-with-gitops-and-kubernetes https://fluxcd.io/docs/guides/mozilla-sops/#encrypting-secrets-using-age https://major.io/2022/04/19/encrypted-gitops-secrets-with-flux-and-age/ https://github.com/jkroepke/helm-secrets/wiki/Secret-Backends https://github.com/variantdev/vals/ https://secrets-store-csi-driver.sigs.k8s.io/introduction.html (in alpha)

[RothAndrew]

I'm seeing this issue as motivation for documentation and/or examples of how to do secrets when using Zarf, vice development of features in Zarf around this area. We can already:

• Pass in secrets at deploy-time, with further enhancement coming soon to be able to pass them from a file vice just on the command line/interactive (#675)
• Use encrypted secrets via Flux+SOPS, or SealedSecrets, or any of the handful of other approaches that involve adding encrypted secrets to the system that get decrypted when they land inside the cluster

Is there more enhancement that is needed to complete a particular use case that we care about, or to make things easier to use? More investigation is needed here based around where the pain points are for users of Zarf who are trying to do secrets.

I think the potential resolution of this issue could be that the user can now keep secrets out of their package completely by using deploy-time variables, with further enhancement coming soon to let them come from a file.

[RothAndrew]

Reading closer into @ntwkninja's comment I think I'm hearing support for having a secret store as part of Zarf deployments, something like Hashicorp's Vault.

If that's the case, my question is, is that something that Zarf should own? Or would more investment in other open source projects that we closely align with (like Big Bang) be a better use of that time?

[JasonvanBrackel]

This is tough use case over airgap and will require more discussion. Feels like a lot of work either in HowTos for Zarf + (SOPS | Sealed Secrets | Vault), or adding functionality to Zarf to ship packages with secrets.

Marking as epic

[Noxsios]

There has not been a lot of drive for native support for encrypted contents in Zarf. My opinion leaning more towards that secrets should all be ephemeral and runtime generated.

Closing for now.