Open joshmue opened 2 years ago
I think there's an upcoming feature in external-secrets that could be of interest here; external-secrets/external-secrets#1539.
If I understand it correctly, we'd be able to do something like this
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken # (or ACRAccessToken)
metadata:
name: "my-ecr"
spec:
# specify aws region (mandatory)
region: eu-west-1
# choose an authentication strategy
# if no auth strategy is defined it falls back to using
# credentials from the environment of the controller.
auth:
# option 2: IAM Roles for Service Accounts
# point to a service account that should be used
# that is configured for IAM Roles for Service Accounts (IRSA)
jwt:
serviceAccountRef:
name: "oci-token-sync"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: "argo-helm"
spec:
refreshInterval: "30m"
target:
name: argo-helm
# Specify a blueprint for the resulting Kind=Secret
template:
metadata:
labels:
argocd.argoproj.io/secret-type: repository
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: "my-ecr"
@blakepettersson Agree we shouldn't reinvent the wheel, just want to check how ESO would work.
For a Helm OCI ECR repository, we have a secret like this:
...
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
url: {{ .awsAccount }}.dkr.ecr.{{ .awsRegion }}.amazonaws.com
name: ecr
type: helm
enableOCI: "true"
username: AWS
password: {{ $ecrPasswordValue }}
While username and password would be mapped from the AWS ECR generator output, I wonder how would the secret get to fill url
, name
, type
and enableOCI
?
I checked the external-secrets.io docs, and couldn't find a way to set values in clear (maybe it's obvious, but I fail to find it)
Nevermind, I guess this can be achieved with templating: https://external-secrets.io/main/guides/templating/
@alexef exactly, in theory the ExternalSecret would look like this:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: "argo-helm"
spec:
refreshInterval: "30m"
target:
name: argo-helm
# Specify a blueprint for the resulting Kind=Secret
template:
type: Opaque
stringData:
url: {{ .proxy_endpoint }}
name: ecr
type: helm
enableOCI: "true"
username: {{ .username }}
password: {{ .password }}
metadata:
labels:
argocd.argoproj.io/secret-type: repository
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: "my-ecr"
We'll see how this actually pans out once this goes GA though 😄
So.. I managed to give this a bit of a spin with the newest ESO version, 0.7.0
. This snippet below creates a secret which gets updated every 30 mins via IRSA.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secret-ecr
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::${myAccountId}:role/my-special-role
---
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
name: ecr-eu-north-1
spec:
region: eu-north-1
auth:
jwt:
serviceAccountRef:
name: external-secret-ecr
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: argocd-ecr-helm-credentials
spec:
refreshInterval: "30m"
target:
creationPolicy: Owner
name: argocd-ecr-helm-credentials
template:
type: Opaque
data:
# The url needs to be explicitly stated since `proxy_endpoint` returns a url prefixed with `https://`
url: ${myAccountId}.dkr.ecr.eu-north-1.amazonaws.com/${myRepository}
type: helm
enableOCI: "true"
username: "{{ .username }}"
password: "{{ .password }}"
metadata:
labels:
argocd.argoproj.io/secret-type: repository
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: ecr-eu-north-1
The prerequisite IAM permissions for the role itself; I tried to scope this down to not be *
but ESO did not accept that:
resource "aws_iam_policy" "get-test-oci-helm-charts" {
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"ecr:BatchGetImage",
"ecr:GetAuthorizationToken",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability"
],
"Resource" : "*"
}
]
})
}
When using IRSA the prerequisite trust relationships needs to be set (as shown below with this TF module)
module "get-oci-helm-charts" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "4.2.0"
create_role = true
role_name = "my-special-role"
provider_url = module.cluster.oidc_issuer_url
role_policy_arns = [aws_iam_policy.get-test-oci-helm-charts.arn]
oidc_fully_qualified_subjects = ["system:serviceaccount:argocd:external-secret-ecr"]
}
This propagates the secret to Argo CD, but I had problems actually using a private OCI repo dependency, which hopefully might be resolved with #11327. I'll do a bit more testing next week once 2.5.5 is out to see how that works...
I tried to upgrade to 2.5.5 but I still can't get Helm OCI dependencies to work. I'm probably missing something obvious here. I'm trying to use repo credentials with a Git repository which is a Helm chart. If I understand correctly it should be enough to specify a prefix with repo credentials, like below
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: ecr-eu-north-1
refreshInterval: 30m
target:
creationPolicy: Owner
deletionPolicy: Retain
name: argocd-ecr-helm-credentials
template:
data:
enableOCI: "true"
password: '{{ .password }}'
type: helm
url: my-account-id.dkr.ecr.eu-north-1.amazonaws.com
username: '{{ .username }}'
engineVersion: v2
metadata:
labels:
argocd.argoproj.io/secret-type: repo-creds
type: Opaque
This will create a secret, which is indeed present:
$ kubectl get secret argocd-ecr-helm-credentials -o json | jq '.data | map_values(@base64d)'
{
"enableOCI": "true",
"password": "some really long password",
"type": "helm",
"url": "my-account-id.dkr.ecr.eu-north-1.amazonaws.com",
"username": "AWS"
}
I then have an application, which in this case is a Git repo hosting a Helm chart
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: whatevs
namespace: argocd
spec:
# Everything else redacted
source:
helm:
passCredentials: true
releaseName: some-release
valueFiles:
- values.yaml
path: .
repoURL: https://some-git-repo.git
targetRevision: blake-test-2
In the Chart.yaml
itself:
apiVersion: v2
name: some-chart
version: 3.2.2
dependencies:
- name: some-dependency
version: v0.18.1
repository: oci://my-account-repo.dkr.ecr.eu-north-1.amazonaws.com/some-repo
But I still get
rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://my-account-id.dkr.ecr.eu-north-1.amazonaws.com/some-repo pulling from host my-account-id.dkr.ecr.eu-north-1.amazonaws.com failed with status code [manifests v0.18.1]: 401 Unauthorized
Any ideas @alexef?
At first sight, everything is legit. I'll try to reproduce the issue locally.
@blakepettersson so I made it work like this:
apiVersion: v2
name: some-chart
version: 3.2.2
dependencies:
- name: <repo_name>
version: 3503.0.0
repository: oci://redacted.dkr.ecr.eu-central-1.amazonaws.com
hope this helps. I tried with latest master, but I can also try out with 2.5.5. LE: works the same on 2.5.5 tag
After a lot of trial and error I finally made it work. I was bashing my head over the helm dependency build
error all day, but after I copied over the exact same chart and application code to a new repository + app it worked!
There seems to be an issue with how Argo CD is caching Helm (OCI only? Not sure) dependencies. I managed to reproduce this by doing the following:
1) Create a suitable secret 2) Create a new Helm chart in a Git repository with a Helm dependency which makes use of said secret 3) Sync, verify all is good 4) Delete the secret 5) Do a hard refresh 6) The app will now error, and no amount of hard refreshes or recreations of the application will allow the application to sync its manifests, which is fine for now 7) Recreate the same secret 8) Try to do another hard refresh 9) Although the secret is back, no amounts of hard refreshes or recreations of the application will ever get it back to a good state
~Seems like the issue above is related to #7673. In any case using ESO works, but I suspect we need to address #7673 to ensure that Argo CD's secret cache gets properly updated whenever ESO updates the repo creds.~ Scratch that, I cannot repro this locally, works as intended there.
I'm trying to do the same thing for Google Artifact Registry. However, no success so far, refreshing my application causes error:
rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not retrieve list of tags for repository oci://europe-west4-docker.pkg.dev: GET "https://europe-west4-docker.pkg.dev/v2/xxx/helm/app/tags/list": GET "https://europe-west4-docker.pkg.dev/v2/token?scope=repository%3Axxx%2Fhelm%2Fapp%3Apull&service=europe-west4-docker.pkg.dev": unexpected status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/xxx/locations/europe-west4/repositories/helm" (or it may not exist)
Secret:
kubectl get secret -n argo-cd argocd-acr-helm-credentials -o json | jq '.data | map_values(@base64d)'
{
"enableOCI": "true",
"password": "xxx",
"type": "helm",
"url": "europe-west4-docker.pkg.dev",
"username": "oauth2accesstoken"
}
Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
labels:
argocd.argoproj.io/instance: app-of-apps
name: docs
namespace: argo-cd
spec:
destination:
name: dev
namespace: default
project: devtools
source:
path: docs/production
repoURL: 'https://github.com/xxx/infra-devtool.git'
targetRevision: HEAD
syncPolicy:
automated: {}
Chart.yaml of docs application:
apiVersion: v2
name: docs
type: application
version: 0.1.0
dependencies:
- name: xxx/helm/app
version: "^0.1.0"
repository: oci://europe-west4-docker.pkg.dev
Any ideas of what am I missing here?
It seems that Argo CD, doesn't use credentials provided in secret. Setting helm dependency version to static: 0.1.19
results in error:
rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://europe-west4-docker.pkg.dev/xxx/helm/app: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
I'm on ArgoCD 2.5.5
got this fully working on latest argocd with latest ESO @blakepettersson you are the real MVP @jacek-jablonski notice the exact syntax of the URL
module "get-oci-helm-charts" {
source = "github.com/aws-ia/terraform-aws-eks-blueprints/modules/irsa"
kubernetes_namespace = "argocd"
create_kubernetes_namespace = false
kubernetes_service_account = local.k8s_sa
irsa_iam_policies = [aws_iam_policy.get-test-oci-helm-charts.arn]
eks_cluster_id = module.eks_blueprints.eks_cluster_id
eks_oidc_provider_arn = module.eks_blueprints.eks_oidc_provider_arn
depends_on = [module.kubernetes_addons]
}
resource "aws_iam_policy" "get-test-oci-helm-charts" {
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"ecr:BatchGetImage",
"ecr:GetAuthorizationToken",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability"
],
"Resource" : "*"
}
]
})
}
resource "kubectl_manifest" "ecr_auth_token" {
yaml_body = <<YAML
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
name: ecr-${data.aws_region.current.name}
namespace: argocd
spec:
region: ${data.aws_region.current.name}
auth:
jwt:
serviceAccountRef:
name: ${local.k8s_sa}
YAML
depends_on = [module.kubernetes_addons]
}
resource "kubectl_manifest" "ecr_auth_token_external_secret" {
yaml_body = <<YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ecr-auth-token-external-secret
namespace: argocd
spec:
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: ecr-${data.aws_region.current.name}
refreshInterval: 30m
target:
creationPolicy: Owner
deletionPolicy: Retain
name: argocd-ecr-helm-credentials
template:
data:
enableOCI: "true"
password: '{{ .password }}'
type: helm
url: ${data.aws_caller_identity.current.account_id}.dkr.ecr.${data.aws_region.current.name}.amazonaws.com/
username: '{{ .username }}'
engineVersion: v2
metadata:
labels:
argocd.argoproj.io/secret-type: repo-creds
type: Opaque
YAML
depends_on = [module.kubernetes_addons]
}
@blakepettersson @nahum-landa - thanks for all your inputs.
The only other thing that could make this even more useful would be to do this a credential template so when we add repos we don't have to worry about templating each specific repo with External Secrets.
The only other thing that could make this even more useful would be to do this a credential template so when we add repos we don't have to worry about templating each specific repo with External Secrets.
It would be interesting to have something like that, how do you envision that would look like?
@nahum-landa Thank you. Your solution saved hours of my time. I faced an error with the url
and as per the argocd, url
should be just repo name. Removing oci://
worked for me.
@nahum-landa Thank you. Your solution saved hours of my time. I faced an error with the
url
and as per the argocd,url
should be just repo name. Removingoci://
worked for me.
damit! fixed on my machine and forgot to update. very sorry! fixed the example
@nahum-landa thanks for this solution. i've tried using this which seems to be working for few days and now i'm getting this error
rpc error: code = Unknown desc = `helm registry login <redacted>.dkr.ecr.eu-central-1.amazonaws.com --username ****** --password ******` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin. Error: login attempt to https://<redacted>.dkr.ecr.eu-central-1.amazonaws.com/v2/ failed with status: 401 Unauthorized
However i'm able login to helm registry using bash with same credentials just fine. any ideas on how to correct this ? i'm using ArgoCD 2.7.1
I see that this Feature mentions gke-gcloud-auth-plugin
for Google GKE. The gke-gcloud-auth-plugin
plugin will be required for GKE 1.26+ per https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Starting with each release schedule https://cloud.google.com/kubernetes-engine/docs/release-schedule
Is ArgoCD 2.7+ set to handle this requirement? Is there anything needed to be changed to the deployments?
For someone who uses ESO solution and still experiencing the auth issue when using OCI Helm dependencies - in my case, the issue was that I did not add OCI repository to the list of source repositories for my ArgoCD project. Also, for me it works only with repo-creds
secret type, not repository
I don't know if this is still of any interest but we managed to solve the problem by following three steps:
apiVersion: v1
data:
enableOCI: true
name: helm-ecr
type: helm
url: <redacted>.dkr.ecr.eu-central-1.amazonaws.com
kind: Secret
metadata:
annotations:
managed-by: argocd.argoproj.io
labels:
argocd.argoproj.io/secret-type: repository
name: repo-123456
namespace: argocd
type: Opaque
Create ECRAuthorizationToken as described above
Create External Secret with exactly the same name as the repo secret name (in the example repo-123456):
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: "repo-123456"
spec:
refreshInterval: 30m
target:
name: repo-123456
creationPolicy: Merge
template:
data:
password: "{{ .password }}"
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
name: ecr-eu-central-1
We got the idea from this guide and for us it did the trick!
Glad to see that ESO works out for many people! :+1:
However, using ESO means that an user has to maintain ESO alongside ArgoCD, of course. That means that ESO cannot initially be deployed via ArgoCD like just another workload, but as prerequisite (either manually, by other GitOps tooling or custom automation). That increases operational overhead significantly.
Agree we shouldn't reinvent the wheel
I understand that implementing native support would incur coding and maintenance efforts, while creating some sort of non-standardized interface, while other tooling like ESO already solved very similar problems in a generic fashion (with code that maybe would end up 80% identical).
On the other hand, from user perspective, using native support of cloud-IAM-based authentication would probably keep operational overhead much slimmer [^1]. IMHO/AFAICT, authentication via an IAM system supersedes static/password authentication slowly everywhere, making it the norm rather than a special case.
E.g. flux also implements native support for AWS/Azure/GCP.
It is of course up to you maintainers (e.g. @alexef @blakepettersson) to decide on the trade-off between slim code maintenance and operational experience; Just wanted to make above points in favor of implementing native (generic or provider specific, like flux did it) support.
[^1]: IMHO, using ESO is also rather a workaround than a solution, as opposed to e.g. ExecProviderConfig
for K8s access, where ArgoCD already has great native support for non-static credentials.
I could not find an existing GH issue covering that, so here we go. If you want, I could take a look on how to implement this.
Summary
Implement option to fetch repository credentials at runtime.
Motivation
For cluster access, ArgoCD already offers setting
ExecProviderConfig
to fetch cluster credentials dynamically using a Kubernetes credential plugin.This feature makes it possible to rely on a cloud's IAM system instead of relying on manually maintained secrets.
That is great, as it has big advantages in terms of security as well as in terms of maintainability.
E.g. one may use... (I did not use the latter two yet)
aws eks get-token
for Amazon EKSgke-gcloud-auth-plugin
for Google GKEHowever, for (private) repositories, it is still required to manage secrets manually.
Proposal
Implement support for fetching repository credentials dynamically - akin to the
ExecProviderConfig
functionality for clusters.This most likely has to be implemented separately for Git repositories and Helm repositories.
For Helm repositories
Use case: Serve (OCI) Helm Charts from a managed container registry.
Add configuration section (akin to
ExecProviderConfig
for clusters) allowing to specify a command/script returning username, password/token and optionally time of expiry for a specific container registry.As far as I can tell, most of the heavy lifting is already done by most big cloud providers (I did not use the latter two yet):
az acr login --expose-token
aws ecr get-authorization-token
gcloud auth print-access-token
While the configuration and input/output of Kubernetes credential plugins are well defined using K8s-style API groups and versions, there is no such thing for registries, AFAIK.
So it may be necessary to define some sort of such contract and wrap execution of the platform tools with some shell script harmonizing input/output. For thoughts on how such contract may look like, see below.
For Git repositories
Unfortunately, I have no idea how to implement this in a good-enough fashion. It's not as easy like it is for Helm repositories, as Git providers usually do not accept tokens issued by a cloud IAM system.
Appendix
Example API's
In its simplest form:
Configuration:
Input via environment variable: None
Output via stdout:
Mimic cluster ExecProviderConfig style:
Configuration:
Input: Like K8s via environment variable.
Output: Like K8s via stdout.