gitops secrets

[gregdurham]

I asked a previous question, in relation to helm plugins, and having more experience with ArgoCD, I determined it made more sense for our environment to utilize helm template to render out k8s manifests and store those in git, rather than using tiller.

The issue now is, how to handle secrets in a safe manner without committing creds in the clear in git. The solution I am using now is to use sealed secrets, and it works...just not really easy for handling multiple environments and clusters.

I would really like a solution integrated into argocd for git-crypt or SOPS which allows tighter control of secrets, with sops, we can utilize a cloud providers KMS, and tie a user's IAM creds to that.

Is there any feature on the roadmap to support such functionality?

Thanks!

[jessesuen]

Argo CD needs to remain un-opinionated about how secrets are managed in git, simply because there's just so many ways to do it, and there's no one-size-fits-all solution that satisfies everyone's use case, especially when a tight integration with a cloud provider's KMS is desired, or some other central secret store (e.g. S3, Vault). Off the top of my head, here are some ways people are doing secrets with GitOps:

• sealed secrets - https://github.com/bitnami-labs/sealed-secrets
• helm secrets - https://github.com/futuresimple/helm-secrets
• kustomize secret generator plugins - https://groups.google.com/forum/#!msg/kubernetes-sig-cli/MY6hjg2-J-0/bDfwF89SAwAJ

This needed flexibility is precisely why repo-server customization and config management plugins were built.

One solution that I've been considering that might work well with AWS, is:

1. bind an IAM role to the argocd-repo-server pods (e.g. using KIAM)
2. this IAM role should have privileges to either: (a) access the KMS key to decrypt an encrypted secret stored in git, or (b) retrieve the plain-text value of the secret from a source other than git (e.g. s3 bucket, vault)
3. if using kustomize, customize the repo-server with a kustomize secret generator plugin, which would use the mechanism in step 2 to return the secret.

With the above approach, because a single IAM role is bound to the single argocd-repo-server, the tenants of an Argo CD instance would need to be trusted, or at least the kustomize secret generator plugin would need to be designed in such a way that it would only retrieve secrets associated with the repo and not return secrets across git repos.

In any case, it's unlikely that Argo CD will form an opinion on this, at least until clear patterns form on how GitOps and secrets.

[larslevie]

Using the new secret generator plugin requires using a unreleased version and calling kustomize with the --enable_alpha_goplugins_accept_panic_risk flag.

Using a custom version of kustomize is documented but it doesn't look like there is a way to invoke it with the enable_alpha_goplugins_accept_panic_risk flag. Is that correct, or am I not looking in the right place?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexec]

Re-opening for discussion

[jcooklin]

@alexec We'd like to the sops kustomize secret generator plugin with argocd. Is or will this be supported? Thanks.

[yujunz]

You can always build your own image for argocd-repo-server.

See https://argoproj.github.io/argo-cd/operator-manual/custom_tools/

[alexec]

See https://argoproj.github.io/argo-cd/operator-manual/secret-management/.

[devstein]

@gregdurham @jcooklin

We switched to Argo CD + SOPS to manage our K8s secrets the same way we manage the rest of our K8 resources. To do this, we create a kustomize plugin for SOPS encoded resources and a commit hook to prevent people from committing unencrypted secrets.

https://github.com/viaduct-ai/kustomize-sops#argo-cd-integration

[imranismail]

I built this to use secretsmanager with argocd and kustomize: https://github.com/imranismail/external-secret

[omerlh]

Maybe we can integrate Kamus easily - it has built-in support for encrypting secret for a specific application, so it fit the terms used by argocd. The encryption is done using external KMS, so multiple environments should be no-burner.

(full disclosure: I'm the maintainer of Kamus)

[rchenzheng]

What's the recommended way to fetch Vault secrets without redeploying?

[MattLud]

Is there any guidance on setting up stuff like SOP or helm plugins - helm init happens on each application if I recall correctly.

[devstein]

@MattLud If you are referring to SOPS, we made a plugin, called KSOPS for kustomize and Argo CD.

I don't have any advice on helm plugins 😞

[MattLud]

@devstein - Yeah - I was hoping specifically for guidance on helm plugins since that recommendation is there. Specifically, I'm curious how ${HELM_HOME} should be setup for the plugin directory - should I use a custom workflow to install the plugin every time ala https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ ?

[devstein]

@MattLud With my limited knowledge of helm plugins, that makes sense to me. You either would inject the plugins via an initContainer and override the HELM_HOME or create a custom docker image with whatever you need.

If you are concerned with the configure of HELM_HOME in the Argo CD image, I recommend kubectl exec into to explore the directory structure or run the docker image itself and override the entrypoint

[MattLud]

Ok - got it helm secrets working. It is not nearly as straightforward as overriding helm secrets as Argo reprovisions a new helm home every run it seems. But I did get it working. Where is the best place for me to dump the doc on doing it?

[nemo83]

@MattLud I'd be interested in knowing how you did that. We use helm secrets with Sops and I would love to integrate it in ArgoCD. Can you create a 1-pager github repo in your account and link here? Much appreciated.

[MattLud]

Hi there - Yes I can put something together on what I did. I don't know how much more refining it will need, but some direction on this would have been better than nothing.

On Fri, Mar 6, 2020 at 7:43 AM Giovanni Gargiulo notifications@github.com wrote:

@MattLud https://github.com/MattLud I'd be interested in knowing how you did that. We use helm secrets with Sops and I would love to integrate it in ArgoCD. Can you create a 1-pager github repo in your account and link here? Much appreciated.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-cd/issues/1364?email_source=notifications&email_token=AACTA7NRP2Q32LYHRRSO4O3RGD4WNA5CNFSM4HDLUNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOBMNYA#issuecomment-595773152, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACTA7OSSIGZZF2HIMFWBM3RGD4WNANCNFSM4HDLUNQA .

[yann-soubeyrand]

@nemo83 here are instructions on how we use Sops with Helm: https://github.com/camptocamp/docker-argocd. We don't use Helm Secrets plugin but I think it should be compatible or at least fairly easy to adapt.

[nhuray]

Just a quick comment about what I did for my company. I took a different approach using a Sops Kubernetes Controller: https://github.com/isindir/sops-secrets-operator/

I'm using Helm to create SopsSecret object (CRD) and the controller use the KMS key (in my case GCP) to decrypt the secret and to create a regular Secret object.

That approach really looks like what's doing Sealed Secret but the encryption/decryption key is outside the Kubernetes cluster (in Cloud KMS) which reduce maintenance in case of DR (recreate the entire cluster).

[xsreality]

Hi there - Yes I can put something together on what I did. I don't know how much more refining it will need, but some direction on this would have been better than nothing. … On Fri, Mar 6, 2020 at 7:43 AM Giovanni Gargiulo @.***> wrote: @MattLud https://github.com/MattLud I'd be interested in knowing how you did that. We use helm secrets with Sops and I would love to integrate it in ArgoCD. Can you create a 1-pager github repo in your account and link here? Much appreciated. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1364?email_source=notifications&email_token=AACTA7NRP2Q32LYHRRSO4O3RGD4WNA5CNFSM4HDLUNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOBMNYA#issuecomment-595773152>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACTA7OSSIGZZF2HIMFWBM3RGD4WNANCNFSM4HDLUNQA .

Curious to know how you integrated helm secrets and sops with argocd. Can you share some info on this please?

[ibmdcb]

@nemo83 here are instructions on how we use Sops with Helm: https://github.com/camptocamp/docker-argocd. We don't use Helm Secrets plugin but I think it should be compatible or at least fairly easy to adapt.

I tried this out and works like a charm. very simple to use and easy to follow. Much appreciated.

[kvaps]

Well. I was trying to implement workaround for git-crypt support, but this is far from ideal: https://github.com/kvaps/argocd-git-crypt

I would prefer to have native hooks on git clone

[kvaps]

Hi folks, I found better workaround to implement git-crypt for ArgoCD, thus now it is working much better.

I even wrote small article about my advantages: https://itnext.io/configure-custom-tooling-in-argo-cd-a4948d95626e#567a

However It would be nice to add native support for persistent gpg-keys from the ArgoCD side (see: https://github.com/argoproj/argo-cd/issues/4316)

[arthurk]

@kvaps I'm interested in reading your article but medium wants me to create an account to do so. Is it available on another platform to read?

[kvaps]

@arthurk what's a pity, sure, I copied its content direct on GitHub project https://github.com/kvaps/argocd-git-crypt#configure-custom-tooling-in-argo-cd

[ittennull]

Hi guys, I'd like to add another solution to the Argocd's pool of secret management tools. I created a project that lets Argocd use sops-encrypted secrets in Helm applications. https://gitlab.com/ittennull/sopshelm It's easy to install and works great!

[jribmartins]

I'm using two different approaches to solve the GitOps problem with secrets within ArgoCD:

1. Wrapping argocd helm with a helm secrets installation, similar to this article
2. Using sealed-secrets, with this you only need to replace secrets with sealed-secrets in your helm chart, see the project page - requires sealed secrets installed on the k8s cluster.

I hope it helps, I'm happy with it 👍

[Hariharasuthan99]

Has anyone attempted to use CyberArk or Azure Vault to pass secrets to ArgoCD?

[jkroepke]

@Hariharasuthan99

We tried Azure Vault using https://github.com/Azure/secrets-store-csi-driver-provider-azure

It works, but since its way more complex then Sealed Secret, we are going to use SealedSecrets.

[jkroepke]

For reference I wrote a documentation to integrate helm-secrets into ArgoCD. It's not longer necessary to seed the ArgoCD managed gpg agent.

Checkout: https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration

[rfashwall]

For reference I wrote a documentation to integrate helm-secrets into ArgoCD. It's not longer necessary to seed the ArgoCD managed gpg agent.

Checkout: https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration

hey @jkroepke thanks for sharing that but I guess this is not possible any more since mounting value files outside the repo boundaries is not possible any more due to security issue update https://github.com/argoproj/argo-cd/security/advisories/GHSA-63qx-x74g-jcr7

[jkroepke]

I know. see #8397

[rossigee]

Not sure if there's a better place to suggest it, but here's another K8S secret management tool for the list:

https://piggysec.com/

[crenshaw-dev]

I think the previous close was accidental.

[crenshaw-dev]

I think Jesse's comment makes a lot of sense. I also think his caveat that Argo CD should remain un-opinionated "until clear patterns form on how GitOps and secrets" work leaves us plenty of room to consider whether those patterns have formed enough for us to start looking at first-class support for some of them.

[darox]

I don't know whether this is the right place to ask, but let's go. What's the recommended way to apply plain Kubernetes secrets encrypted with SOPS? For Kustomize we will use https://github.com/viaduct-ai/kustomize-sops but what if we have plain YAML files?

[johgoe]

Azure Key Vault with aksv2k8s (https://github.com/SparebankenVest/azure-key-vault-to-kubernetes) is also working well with Argo CD.

[mmalyska]

Hi, I've created simple plugin wrapper for sops that replaces secrets inside yamls argocd-secret-replacer. I'm using it personally for quite some time and it performs quite well. Also I've prepared PR to add it to docs https://github.com/argoproj/argo-cd/pull/10494

[christianh814]

Closing this since there have been multiple solutions posted here, there is a secret management solution in Argo Project Labs, and the conversation has been marked as "answered"

[MohammedNoureldin]

Is it just me, or does anyone else feel that all solutions are unsatisfying for the particular case of using Helm encrypted values? Which has to be huge part of Argo CD users, because of being GitOps tool.

I mean, in some cases you might want also to inject secrets as manifest, but many users would still want to install Helm charts out of the box and pass some secrets in a multiple values.yaml files. These values.yaml has to be encrypted and uploaded to a secure place, and Argo CD has to be able out-of-the-box to decrypt them without any further manipulation (except maybe enabling the decryption functionality).

These encrypted secrets can be uploaded to a private Git repo and monitored by Argo CD. Using them encrypted by Helm-Secrets is something different and needs some further work, if it is possible at all because of some limitations mentioned by Helm-Secrets plugin. I am not sure what other options we have for the values.yaml to keep them secret but also monitored by Argo CD? I am interested here only in self-hosted open-source solutions.

Being able to reference the values from Kubernetes secrets would be one nice possiblity, as would be able to deploy a helm of all secret values.yaml wrapped inside K8s secrets, then these could be referenced by Argo CD as Helm values.yaml. Would this approach be somehow possible? Or is there any better or easier approach?

At the moment using encrypted values is relatively kind of pain. Most plugins or suggestions talk about encrypting the secrets as manifest, which does not cover the "main" use-case for GitOps. I would continue the discussion and focus on the cases I mentioned (particularly encrypted Helm values.)

[victorwon]

check https://github.com/hashicorp/vault-secrets-operator if you are using Vault as secret manager.

[tomkukral]

I agree with @MohammedNoureldin. There are two ways how most of the people send secrets to the applications.

A: use sealedsecrets/sops/vault-secrets-operator to provide secrets encrypted and decrypt it in the cluster

B: send secrets as values to helm (or argocd application) and let helm to render it

Option B is much more common (and yes, you can configure some helm charts to use external secret) so having possibility to encrypt data in Argocd application object seems to be essential.

[ramizpolic]

Just another idea for anyone trying to resolve these issues:

At Cisco, we use Vault alongside https://github.com/bank-vaults/vault-secrets-webhook to enable direct injection of secrets from Vault into k8s resources. With this approach, you can specify an environment variable in a pod like

        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY

and it will be replaced using webhook with the value of AWS_SECRET_ACCESS_KEY secret found in secret/accounts/aws Vault store. Injection is also possible into secrets, config maps, and custom resources. Feel free to check docs and use-cases more in depth at https://bank-vaults.dev/docs/mutating-webhook/.

[crenshaw-dev]

Option B is much more common

In my opinion, we should move towards a future where injecting secrets (or any runtime state for that matter) into manifests is avoided. The manifests should be an expression of the developer's intent, and the precise value of a secret is not part of their intent.

Secret injection in manifests also makes the emerging "rendered manifests" pattern much more difficult to adopt. You can't present the manifests to the users in a PR or commit them to a "rendered manifest branch" if they contain secrets.

In my opinion, management of runtime state is the job of operators. If the runtime state doesn't change often, a mutating webhook is probably fine.

[rossigee]

At Cisco, we use Vault alongside https://github.com/bank-vaults/vault-secrets-webhook to enable direct injection of secrets from Vault into k8s resources.

A tip for AWS users: We have had great success using Piggysec to inject AWS Secrets and Parameter Store values into our clusters.

[MohammedNoureldin]

I agree that injecting the secrets using some kind of operators seems the way to go, probably also is the most beautiful and clean way, and provides the easiest experience in managing and rotating the secrets. However at the moment we do not have any real open-source vendor-agnostic solution. I mean a really open-source one with no hidden pay-to-get features. A few core-only open-source are out there, but IMHO they cannot be used on production without upgrading to the closed-source plan.

I like the idea of Vault, and I have been planning to use it. However, my issue with it, that arose a few month ago, is the new not open-source license Hashi-corp started using. And with such software you cannot just freeze the version on an old one because of potential security issues.

For me such vault-similar software that is really fully open-source and self-hosted for my secrets is the only option to use, I believe quite a lot of users will be in the situation now after Hashicorp started using BSL.

That leaves the option to host our secrets (values.yaml) encrypted on some private Git repo. Not because this is the most beautiful option, but because it is the only self-hosted under-our-control option.

[rounakdatta]

I see the Infisical OSS framework wasn't talked about here. It has a Kubernetes operator component, which can take care of delivering the secrets as Secret objects. It is more intuitive to use than Vault, however of course has some security trade-offs compared to Vault.

[MohammedNoureldin]

Well TBO Infisical is sold as OSS, but most of it is closed-source, and you need the closed features to be able to use it reasonably in a a serious production environment.