argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.63k stars 5.37k forks source link

[Feature] Implement option to fetch repository credentials at runtime #10218

Open joshmue opened 2 years ago

joshmue commented 2 years ago

I could not find an existing GH issue covering that, so here we go. If you want, I could take a look on how to implement this.

Summary

Implement option to fetch repository credentials at runtime.

Motivation

For cluster access, ArgoCD already offers setting ExecProviderConfig to fetch cluster credentials dynamically using a Kubernetes credential plugin.

This feature makes it possible to rely on a cloud's IAM system instead of relying on manually maintained secrets.
That is great, as it has big advantages in terms of security as well as in terms of maintainability.

E.g. one may use... (I did not use the latter two yet)

However, for (private) repositories, it is still required to manage secrets manually.

Proposal

Implement support for fetching repository credentials dynamically - akin to the ExecProviderConfig functionality for clusters.

This most likely has to be implemented separately for Git repositories and Helm repositories.

For Helm repositories

Use case: Serve (OCI) Helm Charts from a managed container registry.

Add configuration section (akin to ExecProviderConfig for clusters) allowing to specify a command/script returning username, password/token and optionally time of expiry for a specific container registry.

As far as I can tell, most of the heavy lifting is already done by most big cloud providers (I did not use the latter two yet):

While the configuration and input/output of Kubernetes credential plugins are well defined using K8s-style API groups and versions, there is no such thing for registries, AFAIK.
So it may be necessary to define some sort of such contract and wrap execution of the platform tools with some shell script harmonizing input/output. For thoughts on how such contract may look like, see below.

For Git repositories

Unfortunately, I have no idea how to implement this in a good-enough fashion. It's not as easy like it is for Helm repositories, as Git providers usually do not accept tokens issued by a cloud IAM system.

Appendix

Example API's

In its simplest form:

Configuration:

apiVersion: v1
kind: Secret
metadata:
  name: argo-helm
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: argo
  url: my-private-registry.domain.invalid/argo-helm
  type: helm

  # Really not sure about the name "providerExec".
  # I just used YAML for readability of example implementation for Azure;
  # Most probably actual implementation would use JSON.
  providerExec: |
    - bash
    - -c
    - |
        set -e
        az login --identity --allow-no-subscriptions > /dev/null
        az acr login -n my-private-registry.domain.invalid --expose-token -o json \
          | jq '{"username": "00000000-0000-0000-0000-000000000000", "password": .accessToken, "expiresAt": .accessToken | split(".")[1] | @base64d | fromjson | .exp}'

Input via environment variable: None

Output via stdout:

{"username": "some-username", "password": "some-token", "expiresAt": "... unix-timstamp ..."}

Mimic cluster ExecProviderConfig style:

Configuration:

apiVersion: v1
kind: Secret
metadata:
  name: argo-helm
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: argo
  url: my-private-registry.domain.invalid/argo-helm
  type: helm
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "helm-auth.argoproj.io/v1alpha1",
        "command": "bash",
        "args": ["-c", "..."]
      }
    }

Input: Like K8s via environment variable.

Output: Like K8s via stdout.

blakepettersson commented 1 year ago

I think there's an upcoming feature in external-secrets that could be of interest here; external-secrets/external-secrets#1539.

If I understand it correctly, we'd be able to do something like this

apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken # (or ACRAccessToken)
metadata:
  name: "my-ecr"
spec:
  # specify aws region (mandatory)
  region: eu-west-1

  # choose an authentication strategy
  # if no auth strategy is defined it falls back to using
  # credentials from the environment of the controller.
  auth:

    # option 2: IAM Roles for Service Accounts
    # point to a service account that should be used
    # that is configured for IAM Roles for Service Accounts (IRSA)
    jwt:
      serviceAccountRef:
        name: "oci-token-sync"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "argo-helm"
spec:
  refreshInterval: "30m"
  target:
    name: argo-helm
    # Specify a blueprint for the resulting Kind=Secret
    template:
      metadata:
        labels: 
           argocd.argoproj.io/secret-type: repository
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: "my-ecr"
alexef commented 1 year ago

@blakepettersson Agree we shouldn't reinvent the wheel, just want to check how ESO would work.

For a Helm OCI ECR repository, we have a secret like this:

...
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  url: {{ .awsAccount }}.dkr.ecr.{{ .awsRegion }}.amazonaws.com
  name: ecr
  type: helm
  enableOCI: "true"
  username: AWS
  password: {{ $ecrPasswordValue }}

While username and password would be mapped from the AWS ECR generator output, I wonder how would the secret get to fill url, name, type and enableOCI?

I checked the external-secrets.io docs, and couldn't find a way to set values in clear (maybe it's obvious, but I fail to find it)

alexef commented 1 year ago

Nevermind, I guess this can be achieved with templating: https://external-secrets.io/main/guides/templating/

blakepettersson commented 1 year ago

@alexef exactly, in theory the ExternalSecret would look like this:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "argo-helm"
spec:
  refreshInterval: "30m"
  target:
    name: argo-helm
    # Specify a blueprint for the resulting Kind=Secret
    template:
      type: Opaque
      stringData:
        url: {{ .proxy_endpoint }}
        name: ecr
        type: helm
        enableOCI: "true"
        username: {{ .username }}
        password: {{ .password }}        
      metadata:
        labels: 
          argocd.argoproj.io/secret-type: repository
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: "my-ecr"

We'll see how this actually pans out once this goes GA though 😄

blakepettersson commented 1 year ago

So.. I managed to give this a bit of a spin with the newest ESO version, 0.7.0. This snippet below creates a secret which gets updated every 30 mins via IRSA.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secret-ecr
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${myAccountId}:role/my-special-role
---
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-eu-north-1
spec:
  region: eu-north-1
  auth:
    jwt:
      serviceAccountRef:
        name: external-secret-ecr
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argocd-ecr-helm-credentials
spec:
  refreshInterval: "30m"
  target:
    creationPolicy: Owner
    name: argocd-ecr-helm-credentials
    template:
      type: Opaque
      data:
        # The url needs to be explicitly stated since `proxy_endpoint` returns a url prefixed with `https://`
        url: ${myAccountId}.dkr.ecr.eu-north-1.amazonaws.com/${myRepository}
        type: helm
        enableOCI: "true"
        username: "{{ .username }}"
        password: "{{ .password }}"
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: ECRAuthorizationToken
          name: ecr-eu-north-1

The prerequisite IAM permissions for the role itself; I tried to scope this down to not be * but ESO did not accept that:

resource "aws_iam_policy" "get-test-oci-helm-charts" {
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "ecr:BatchGetImage",
          "ecr:GetAuthorizationToken",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchCheckLayerAvailability"
        ],
        "Resource" : "*"
      }
    ]
  })
}

When using IRSA the prerequisite trust relationships needs to be set (as shown below with this TF module)

module "get-oci-helm-charts" {
  source                          = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                         = "4.2.0"
  create_role                   = true
  role_name                    = "my-special-role"
  provider_url                  = module.cluster.oidc_issuer_url
  role_policy_arns              = [aws_iam_policy.get-test-oci-helm-charts.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:argocd:external-secret-ecr"]
}

This propagates the secret to Argo CD, but I had problems actually using a private OCI repo dependency, which hopefully might be resolved with #11327. I'll do a bit more testing next week once 2.5.5 is out to see how that works...

blakepettersson commented 1 year ago

I tried to upgrade to 2.5.5 but I still can't get Helm OCI dependencies to work. I'm probably missing something obvious here. I'm trying to use repo credentials with a Git repository which is a Helm chart. If I understand correctly it should be enough to specify a prefix with repo credentials, like below

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: ecr-eu-north-1
  refreshInterval: 30m
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: argocd-ecr-helm-credentials
    template:
      data:
        enableOCI: "true"
        password: '{{ .password }}'
        type: helm
        url: my-account-id.dkr.ecr.eu-north-1.amazonaws.com
        username: '{{ .username }}'
      engineVersion: v2
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repo-creds
      type: Opaque

This will create a secret, which is indeed present:

$ kubectl get secret argocd-ecr-helm-credentials -o json | jq '.data | map_values(@base64d)'
{
  "enableOCI": "true",
  "password": "some really long password",
  "type": "helm",
  "url": "my-account-id.dkr.ecr.eu-north-1.amazonaws.com",
  "username": "AWS"
}

I then have an application, which in this case is a Git repo hosting a Helm chart

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: whatevs
  namespace: argocd
spec:
  # Everything else redacted
  source:
    helm:
      passCredentials: true
      releaseName: some-release
      valueFiles:
        - values.yaml
    path: .
    repoURL: https://some-git-repo.git
    targetRevision: blake-test-2

In the Chart.yaml itself:

apiVersion: v2
name: some-chart
version: 3.2.2

dependencies:
  - name: some-dependency
    version: v0.18.1
    repository: oci://my-account-repo.dkr.ecr.eu-north-1.amazonaws.com/some-repo

But I still get

rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://my-account-id.dkr.ecr.eu-north-1.amazonaws.com/some-repo pulling from host my-account-id.dkr.ecr.eu-north-1.amazonaws.com failed with status code [manifests v0.18.1]: 401 Unauthorized

Any ideas @alexef?

alexef commented 1 year ago

At first sight, everything is legit. I'll try to reproduce the issue locally.

alexef commented 1 year ago

@blakepettersson so I made it work like this:

apiVersion: v2
name: some-chart
version: 3.2.2

dependencies:
  - name: <repo_name>
    version: 3503.0.0
    repository: oci://redacted.dkr.ecr.eu-central-1.amazonaws.com
  1. note repository is the same as the one in secret (no sub-path)
  2. note the dependency name is exactly the same as the repo name
  3. version is a tag in ECR

hope this helps. I tried with latest master, but I can also try out with 2.5.5. LE: works the same on 2.5.5 tag

blakepettersson commented 1 year ago

After a lot of trial and error I finally made it work. I was bashing my head over the helm dependency build error all day, but after I copied over the exact same chart and application code to a new repository + app it worked!

There seems to be an issue with how Argo CD is caching Helm (OCI only? Not sure) dependencies. I managed to reproduce this by doing the following:

1) Create a suitable secret 2) Create a new Helm chart in a Git repository with a Helm dependency which makes use of said secret 3) Sync, verify all is good 4) Delete the secret 5) Do a hard refresh 6) The app will now error, and no amount of hard refreshes or recreations of the application will allow the application to sync its manifests, which is fine for now 7) Recreate the same secret 8) Try to do another hard refresh 9) Although the secret is back, no amounts of hard refreshes or recreations of the application will ever get it back to a good state

blakepettersson commented 1 year ago

~Seems like the issue above is related to #7673. In any case using ESO works, but I suspect we need to address #7673 to ensure that Argo CD's secret cache gets properly updated whenever ESO updates the repo creds.~ Scratch that, I cannot repro this locally, works as intended there.

jacek-jablonski commented 1 year ago

I'm trying to do the same thing for Google Artifact Registry. However, no success so far, refreshing my application causes error:

rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not retrieve list of tags for repository oci://europe-west4-docker.pkg.dev: GET "https://europe-west4-docker.pkg.dev/v2/xxx/helm/app/tags/list": GET "https://europe-west4-docker.pkg.dev/v2/token?scope=repository%3Axxx%2Fhelm%2Fapp%3Apull&service=europe-west4-docker.pkg.dev": unexpected status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/xxx/locations/europe-west4/repositories/helm" (or it may not exist)

Secret:

kubectl get secret -n argo-cd argocd-acr-helm-credentials -o json | jq '.data | map_values(@base64d)'
{
  "enableOCI": "true",
  "password": "xxx",
  "type": "helm",
  "url": "europe-west4-docker.pkg.dev",
  "username": "oauth2accesstoken"
}

Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    argocd.argoproj.io/instance: app-of-apps
  name: docs
  namespace: argo-cd
spec:
  destination:
    name: dev
    namespace: default
  project: devtools
  source:
    path: docs/production
    repoURL: 'https://github.com/xxx/infra-devtool.git'
    targetRevision: HEAD
  syncPolicy:
    automated: {}

Chart.yaml of docs application:

apiVersion: v2
name: docs

type: application
version: 0.1.0

dependencies:
  - name: xxx/helm/app
    version: "^0.1.0"
    repository: oci://europe-west4-docker.pkg.dev

Any ideas of what am I missing here?

jacek-jablonski commented 1 year ago

It seems that Argo CD, doesn't use credentials provided in secret. Setting helm dependency version to static: 0.1.19 results in error:

rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://europe-west4-docker.pkg.dev/xxx/helm/app: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

I'm on ArgoCD 2.5.5

nahum-landa commented 1 year ago

got this fully working on latest argocd with latest ESO @blakepettersson you are the real MVP @jacek-jablonski notice the exact syntax of the URL

module "get-oci-helm-charts" {
  source                      = "github.com/aws-ia/terraform-aws-eks-blueprints/modules/irsa"
  kubernetes_namespace        = "argocd"
  create_kubernetes_namespace = false
  kubernetes_service_account  = local.k8s_sa
  irsa_iam_policies           = [aws_iam_policy.get-test-oci-helm-charts.arn]
  eks_cluster_id              = module.eks_blueprints.eks_cluster_id
  eks_oidc_provider_arn       = module.eks_blueprints.eks_oidc_provider_arn
  depends_on                  = [module.kubernetes_addons]
}

resource "aws_iam_policy" "get-test-oci-helm-charts" {
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "ecr:BatchGetImage",
          "ecr:GetAuthorizationToken",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchCheckLayerAvailability"
        ],
        "Resource" : "*"
      }
    ]
  })
}

resource "kubectl_manifest" "ecr_auth_token" {
  yaml_body  = <<YAML
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-${data.aws_region.current.name}
  namespace: argocd
spec:
  region: ${data.aws_region.current.name}
  auth:
    jwt:
      serviceAccountRef:
        name: ${local.k8s_sa}
YAML
  depends_on = [module.kubernetes_addons]
}

resource "kubectl_manifest" "ecr_auth_token_external_secret" {
  yaml_body  = <<YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ecr-auth-token-external-secret
  namespace: argocd
spec:
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: ecr-${data.aws_region.current.name}
  refreshInterval: 30m
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: argocd-ecr-helm-credentials
    template:
      data:
        enableOCI: "true"
        password: '{{ .password }}'
        type: helm
        url: ${data.aws_caller_identity.current.account_id}.dkr.ecr.${data.aws_region.current.name}.amazonaws.com/
        username: '{{ .username }}'
      engineVersion: v2
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repo-creds
      type: Opaque
YAML
  depends_on = [module.kubernetes_addons]
}
ahaw023 commented 1 year ago

@blakepettersson @nahum-landa - thanks for all your inputs.

The only other thing that could make this even more useful would be to do this a credential template so when we add repos we don't have to worry about templating each specific repo with External Secrets.

blakepettersson commented 1 year ago

The only other thing that could make this even more useful would be to do this a credential template so when we add repos we don't have to worry about templating each specific repo with External Secrets.

It would be interesting to have something like that, how do you envision that would look like?

hyelavarthi commented 1 year ago

@nahum-landa Thank you. Your solution saved hours of my time. I faced an error with the url and as per the argocd, url should be just repo name. Removing oci:// worked for me.

nahum-landa commented 1 year ago

@nahum-landa Thank you. Your solution saved hours of my time. I faced an error with the url and as per the argocd, url should be just repo name. Removing oci:// worked for me.

damit! fixed on my machine and forgot to update. very sorry! fixed the example

rohit-dimagi commented 1 year ago

@nahum-landa thanks for this solution. i've tried using this which seems to be working for few days and now i'm getting this error

rpc error: code = Unknown desc = `helm registry login <redacted>.dkr.ecr.eu-central-1.amazonaws.com --username ****** --password ******` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin. Error: login attempt to https://<redacted>.dkr.ecr.eu-central-1.amazonaws.com/v2/ failed with status: 401 Unauthorized

However i'm able login to helm registry using bash with same credentials just fine. any ideas on how to correct this ? i'm using ArgoCD 2.7.1

abatesbot commented 1 year ago

I see that this Feature mentions gke-gcloud-auth-plugin for Google GKE. The gke-gcloud-auth-plugin plugin will be required for GKE 1.26+ per https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

Starting with each release schedule https://cloud.google.com/kubernetes-engine/docs/release-schedule

Is ArgoCD 2.7+ set to handle this requirement? Is there anything needed to be changed to the deployments?

burakovsky commented 1 year ago

For someone who uses ESO solution and still experiencing the auth issue when using OCI Helm dependencies - in my case, the issue was that I did not add OCI repository to the list of source repositories for my ArgoCD project. Also, for me it works only with repo-creds secret type, not repository

konsloiz commented 11 months ago

I don't know if this is still of any interest but we managed to solve the problem by following three steps:

  1. Add the Helm repo via the UI (enable OCI) or via the argocd cli. This should result in a Secret (argocd namespace) which looks like the following:
apiVersion: v1
data:
  enableOCI: true
  name: helm-ecr
  type: helm
  url: <redacted>.dkr.ecr.eu-central-1.amazonaws.com
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repo-123456
  namespace: argocd
type: Opaque
  1. Create ECRAuthorizationToken as described above

  2. Create External Secret with exactly the same name as the repo secret name (in the example repo-123456):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "repo-123456"
spec:
  refreshInterval: 30m
  target:
    name: repo-123456
    creationPolicy: Merge
    template:
      data:
        password: "{{ .password  }}"
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: ecr-eu-central-1

We got the idea from this guide and for us it did the trick!

joshmue commented 11 months ago

Glad to see that ESO works out for many people! :+1:

However, using ESO means that an user has to maintain ESO alongside ArgoCD, of course. That means that ESO cannot initially be deployed via ArgoCD like just another workload, but as prerequisite (either manually, by other GitOps tooling or custom automation). That increases operational overhead significantly.

Agree we shouldn't reinvent the wheel

I understand that implementing native support would incur coding and maintenance efforts, while creating some sort of non-standardized interface, while other tooling like ESO already solved very similar problems in a generic fashion (with code that maybe would end up 80% identical).

On the other hand, from user perspective, using native support of cloud-IAM-based authentication would probably keep operational overhead much slimmer [^1]. IMHO/AFAICT, authentication via an IAM system supersedes static/password authentication slowly everywhere, making it the norm rather than a special case.

E.g. flux also implements native support for AWS/Azure/GCP.

It is of course up to you maintainers (e.g. @alexef @blakepettersson) to decide on the trade-off between slim code maintenance and operational experience; Just wanted to make above points in favor of implementing native (generic or provider specific, like flux did it) support.

[^1]: IMHO, using ESO is also rather a workaround than a solution, as opposed to e.g. ExecProviderConfig for K8s access, where ArgoCD already has great native support for non-static credentials.