argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.95k stars 5.46k forks source link

OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

Closed pthornton closed 1 year ago

pthornton commented 2 years ago

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

Describe the bug Application Chart:

apiVersion: v3
name: react-frontend-service
version: 0.2.7-latest-a6f42056
dependencies:
  - name: react-frontend-service
    repository: oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service
    version: 0.2.7-latest-a6f42056
$ argocd --server argocd-test.falkor.rocks --grpc-web repo list
TYPE  NAME           REPO                                                              INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
helm  us-west-2-oci  712053168757.dkr.ecr.us-west-2.amazonaws.com                      false     true   false  true   Successful 

All applications referencing AWS oci charts fail after upgrade. It appears "helm registry login" is only called when add the helm repo using "helm repo add" from cronjob. The "helm dependency update" fails and no login messages appear in repoServer logs.

Error message from UI:

rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized

Repo server log:

time="2022-01-18T22:25:22Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-18T22:30:22Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-18T22:25:22Z" grpc.time_ms=2.77 span.kind=server system=grpc

If I bash into the repo server and do "helm registry login" and then "help dependency build" in the /tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base folder, it successfully pulls the OCI helm chart. So no permission issues. I'm using username/password/url from the argocd cli helm repo add created kubernetes secret repo-4159968007.

This also happens if I upgrade to 2.1.7 instead of 2.2.2. Post in argocd slack also: slack

To Reproduce

Add Application using chart with reference to AWS OCI helm chart.

Expected behavior

Helm charts should pull from AWS OCI after successful "helm registry login" executed before running "helm dependency update".

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v2.2.2+03b17e0.dirty
  BuildDate: 2022-01-01T16:53:02Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: dirty
  GoVersion: go1.17.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.2.2+03b17e0
  BuildDate: 2022-01-01T06:27:52Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Logs

time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=1232.725993
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git checkout --force 4bf8341c9c979b45c443870bfff1328a49536dd1]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=4.994643
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=3.643158
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git rev-parse HEAD]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=2.335853
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm template . --name-template react-frontend-service-a0000000027 --namespace example-test --kube-version 1.21 --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml --api-versions acme.cert-manager.io/v1 --api-versions acme.cert-manager.io/v1/Challenge --api-versions acme.cert-manager.io/v1/Order --api-versions acme.cert-manager.io/v1alpha2 --api-versions acme.cert-manager.io/v1alpha2/Challenge --api-versions acme.cert-manager.io/v1alpha2/Order --api-versions acme.cert-manager.io/v1alpha3 --api-versions acme.cert-manager.io/v1alpha3/Challenge --api-versions acme.cert-manager.io/v1alpha3/Order --api-versions acme.cert-manager.io/v1beta1 --api-versions acme.cert-manager.io/v1beta1/Challenge --api-versions acme.cert-manager.io/v1beta1/Order --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1 --api-versions admissionregistration.k8s.io/v1beta1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiextensions.k8s.io/v1beta1 --api-versions apiextensions.k8s.io/v1beta1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apiregistration.k8s.io/v1beta1 --api-versions apiregistration.k8s.io/v1beta1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions batch/v1beta1 --api-versions batch/v1beta1/CronJob --api-versions bitnami.com/v1alpha1 --api-versions bitnami.com/v1alpha1/SealedSecret --api-versions cert-manager.io/v1 --api-versions cert-manager.io/v1/Certificate --api-versions cert-manager.io/v1/CertificateRequest --api-versions cert-manager.io/v1/ClusterIssuer --api-versions cert-manager.io/v1/Issuer --api-versions cert-manager.io/v1alpha2 --api-versions cert-manager.io/v1alpha2/Certificate --api-versions cert-manager.io/v1alpha2/CertificateRequest --api-versions cert-manager.io/v1alpha2/ClusterIssuer --api-versions cert-manager.io/v1alpha2/Issuer --api-versions cert-manager.io/v1alpha3 --api-versions cert-manager.io/v1alpha3/Certificate --api-versions cert-manager.io/v1alpha3/CertificateRequest --api-versions cert-manager.io/v1alpha3/ClusterIssuer --api-versions cert-manager.io/v1alpha3/Issuer --api-versions cert-manager.io/v1beta1 --api-versions cert-manager.io/v1beta1/Certificate --api-versions cert-manager.io/v1beta1/CertificateRequest --api-versions cert-manager.io/v1beta1/ClusterIssuer --api-versions cert-manager.io/v1beta1/Issuer --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions certificates.k8s.io/v1beta1 --api-versions certificates.k8s.io/v1beta1/CertificateSigningRequest --api-versions config.gatekeeper.sh/v1alpha1 --api-versions config.gatekeeper.sh/v1alpha1/Config --api-versions constraints.gatekeeper.sh/v1alpha1 --api-versions constraints.gatekeeper.sh/v1alpha1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1alpha1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueServiceSelector --api-versions constraints.gatekeeper.sh/v1beta1 --api-versions constraints.gatekeeper.sh/v1beta1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1beta1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueServiceSelector --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions coordination.k8s.io/v1beta1 --api-versions coordination.k8s.io/v1beta1/Lease --api-versions crd.k8s.amazonaws.com/v1alpha1 --api-versions crd.k8s.amazonaws.com/v1alpha1/ENIConfig --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions discovery.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1beta1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions events.k8s.io/v1beta1 --api-versions events.k8s.io/v1beta1/Event --api-versions extensions/v1beta1 --api-versions extensions/v1beta1/Ingress --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions koudingspawn.de/v1 --api-versions koudingspawn.de/v1/Vault --api-versions linkerd.io/v1alpha1 --api-versions linkerd.io/v1alpha1/ServiceProfile --api-versions linkerd.io/v1alpha2 --api-versions linkerd.io/v1alpha2/ServiceProfile --api-versions mutations.gatekeeper.sh/v1alpha1 --api-versions mutations.gatekeeper.sh/v1alpha1/Assign --api-versions mutations.gatekeeper.sh/v1alpha1/AssignMetadata --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions networking.k8s.io/v1beta1 --api-versions networking.k8s.io/v1beta1/Ingress --api-versions networking.k8s.io/v1beta1/IngressClass --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions node.k8s.io/v1beta1 --api-versions node.k8s.io/v1beta1/RuntimeClass --api-versions policy.linkerd.io/v1alpha1 --api-versions policy.linkerd.io/v1alpha1/Server --api-versions policy.linkerd.io/v1alpha1/ServerAuthorization --api-versions policy.linkerd.io/v1beta1 --api-versions policy.linkerd.io/v1beta1/Server --api-versions policy.linkerd.io/v1beta1/ServerAuthorization --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions policy/v1beta1 --api-versions policy/v1beta1/PodDisruptionBudget --api-versions policy/v1beta1/PodSecurityPolicy --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions rbac.authorization.k8s.io/v1beta1 --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRole --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1beta1/Role --api-versions rbac.authorization.k8s.io/v1beta1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions scheduling.k8s.io/v1beta1 --api-versions scheduling.k8s.io/v1beta1/PriorityClass --api-versions split.smi-spec.io/v1alpha1 --api-versions split.smi-spec.io/v1alpha1/TrafficSplit --api-versions split.smi-spec.io/v1alpha2 --api-versions split.smi-spec.io/v1alpha2/TrafficSplit --api-versions status.gatekeeper.sh/v1beta1 --api-versions status.gatekeeper.sh/v1beta1/ConstraintPodStatus --api-versions status.gatekeeper.sh/v1beta1/ConstraintTemplatePodStatus --api-versions status.gatekeeper.sh/v1beta1/MutatorPodStatus --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIDriver --api-versions storage.k8s.io/v1beta1/CSINode --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions storage.k8s.io/v1beta1/StorageClass --api-versions storage.k8s.io/v1beta1/VolumeAttachment --api-versions templates.gatekeeper.sh/v1 --api-versions templates.gatekeeper.sh/v1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1alpha1 --api-versions templates.gatekeeper.sh/v1alpha1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1beta1 --api-versions templates.gatekeeper.sh/v1beta1/ConstraintTemplate --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --api-versions vpcresources.k8s.aws/v1beta1 --api-versions vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy --include-crds]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=38.575524
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm dependency build]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=96.118729
pthornton commented 2 years ago

ApplicationSet that creates application for above chart:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: react-frontend-service-test
  labels:
    ring: test
spec:
  generators:
    - git:
        repoURL: https://gitlab.com/somewhere/prt/example-apps/mushu-dev.git
        revision: HEAD
        files:
          - path: "config/**/config.json"
  template:
    metadata:
      name: 'react-frontend-service-{{cluster.name}}'
      labels:
        ring: test
        tenant: '{{cluster.name}}'
        domain: '{{cluster.domain}}'
    spec:
      project: '{{argocd.project}}'
      source:
        repoURL: https://gitlab.com/somwhere/prt/example-apps/mushu-dev.git
        targetRevision: HEAD
        path: "./react-frontend-service/helm_base"
        helm:
          version: v3
          releaseName: 'react-frontend-service-{{cluster.name}}'
          valueFiles:
            - ../overlays/{{cluster.name}}/globals.yaml
            - ../overlays/flags.yaml
            - ../overlays/{{cluster.name}}/values.yaml
            - ../overlays/{{cluster.name}}/hotfix.yaml
      destination:
        server: '{{cluster.address}}'
        namespace: 'example-test'
      ignoreDifferences:
      - group: apps
        jsonPointers:
        - /spec/replicas
        kind: Deployment
        namespace: 'example-test'

      # Sync policy
      syncPolicy:
        automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
          prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
          selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
          allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
        syncOptions: # Sync options which modifies sync behavior
          - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
          - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
pthornton commented 2 years ago

Worked in v2.0.5 helm chart 3.12.1. Starts failing in v2.1.0 helm chart 3.17.1.

hammeryx commented 2 years ago

@AustinSmart t I noticed you had some comments in other threads around OCI chart deps being broken in 2.1.0.... Is this related?

AustinSmart commented 2 years ago

Since there is a oci chart as a dependency in the first comment, yes it’s likely related. I don’t think the oci chart dependency bug has been fixed yet.

hammeryx commented 2 years ago

Is there a workaround for that @AustinSmart ? Couldn't find any reference as to exactly why that functionality broke.

pthornton commented 2 years ago
time="2022-01-19T16:45:42Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add 712053168757.dkr.ecr.us-west-2.amazonaws.com oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service` failed exit status 1: Error: looks like \"oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service\" is not a valid chart repository or cannot be reached: tag explicitly required" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-19T16:50:42Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-19T16:45:42Z" grpc.time_ms=1.839 span.kind=server system=grpc

Might be related to 7104

Helm 3.7 OCI

Support for the oci:// protocol is also available in various other subcommands. Here is a complete list:

helm pull
helm show
helm template
helm install
helm upgrade

Note that "helm repo" is missing....

mfbieber commented 2 years ago

I had the same issue (also AWS ECR repo that connected well but the same 401 error when adding an application).

The regression probably came in with (lines 195 to 197 in RepoAdd() in util/helm/cmd.go on October 30th 2021):

if c.helmPassCredentialsSupported && passCredentials {
    args = append(args, "--pass-credentials")
}

This is not what we need to login to the ECR, we need it without the --pass-credentials flag for helm. So for instance, this works to login:

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"

I haven't dug into why this is so, but I have a temporary workaround that is working for me with v2.2.3. You can simply set passCredentials to false in your Application definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xxxx
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: xxxx
  source:
    repoURL: xxxx
    path: xxxx
    targetRevision: xxxx
    helm:
      passCredentials: false
...
pthornton commented 2 years ago

@mfbieber I'm setting this in the application but I'm not sure it's getting honored. I'm using ApplicationSet to create the application. The ApplicationSet shows:

      Source:
        Helm:
          Pass Credentials:  false
          Release Name:      typescript-backend-service-{{cluster.name}}
          Value Files:
            ../overlays/{{cluster.name}}/globals.yaml
            ../overlays/flags.yaml
            ../overlays/{{cluster.name}}/values.yaml
            ../overlays/{{cluster.name}}/hotfix.yaml

The created Application has no mention of passCredentials:

  Source:
    Helm:
      Release Name:  typescript-backend-service-a0000000027
      Value Files:
        ../overlays/a0000000027/globals.yaml
        ../overlays/flags.yaml
        ../overlays/a0000000027/values.yaml
        ../overlays/a0000000027/hotfix.yaml

Know of any way to verify passCredentials value on the Application? I see a --helm-pass-credentials on argocd cli set command, but nothing on get? Both get/set fail with no permission to access oci. Starting to think the ApplicationSet is not honoring the setting.

pthornton commented 2 years ago

@mfbieber What version are you on? I created the application directly and same failure on 3.29.5/2.2.2.

I created the Application directly and it still fails with same error:

  Source:
    Helm:
      Pass Credentials:  false
      Release Name:      typescript-backend-service-a0000000027
      Value Files:
        ../overlays/a0000000027/globals.yaml
        ../overlays/flags.yaml
        ../overlays/a0000000027/values.yaml
        ../overlays/a0000000027/hotfix.yaml

Are you calling this directly yourself?

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
pthornton commented 2 years ago

Have a workaround using plugin. In values.yaml:

  repoServer:
    volumes:
      - name: custom-tools
        emptyDir: {}
      - name: plugin-scripts
        configMap:
          name: plugin-scripts
          defaultMode: 0755
    initContainers:
    - name: helm-setup
      image: alpine:3
      command: [
        "/bin/sh",
        "-c",
        "apk add --no-cache aws-cli;
        which aws;
        aws --version;
        cp /usr/bin/aws /custom-tools/aws; "]
      volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
    volumeMounts:
    - mountPath: /usr/local/bin/aws
      name: custom-tools
      subPath: aws
    - mountPath: /usr/local/bin/helmsetup.sh
      name: plugin-scripts
      subPath: helmsetup.sh
      readOnly: true
    - mountPath: /usr/local/bin/helmtemplate.sh
      name: plugin-scripts
      subPath: helmtemplate.sh
      readOnly: true

  server:
    config:
      configManagementPlugins: |
        - name: helm_aws_oci
          init:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmsetup.sh"]
          generate:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmtemplate.sh"]

Add scripts configmap under templates/plugin-scripts.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: plugin-scripts
data:
  helmsetup.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    aws ecr get-login-password --region us-west-2 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-west-2.amazonaws.com"
    helm dependency build
  helmtemplate.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    helm template --api-versions ${KUBE_API_VERSIONS} -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} . --include-crds ${HELM_VALUES_FILES} ${HELM_ARGS}

In application def, replace helm section with plugin:

apiVersion: argoproj.io/v1alpha1
kind: Application
*******
      source:
        path: ./lacework/helm_base
        plugin:
          env:
          - name: HELM_VALUES_FILES
            value: --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml
              --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml
          - name: HELM_ARGS
            value: --set lacework.lacework-agent.laceworkConfig.accessToken='**************************'
              --set lacework.lacework-agent.laceworkConfig.env='mushu.falkor.rocks'
              --set lacework.lacework-agent.laceworkConfig.kubernetesCluster='a0000000027'
          name: helm_aws_oci
mfbieber commented 2 years ago

@pthornton, sorry for replying late.

I am working with 3.26.12 and yes, I called this myself:

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"

I would try to check this week if something changed during the last two weeks in the code. I could also submit a PR to somehow fix this, but I am unsure of how we would like to solve this. I'll think about it and will try to make a proposal, if I can.

Good that you have the plugin working, it looks like a lot of work and I would rather have argo-cd support AWS OCI itself!

hedasaurabh commented 2 years ago

@mfbieber Any updates here? We are running into the same issue.

Version:

$ argocd version
argocd: v2.3.3+07ac038.dirty
  BuildDate: 2022-03-30T05:20:18Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: dirty
  GoVersion: go1.18
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.2.5+8f981cc
  BuildDate: 2022-02-05T01:33:25Z
  GitCommit: 8f981ccfcf942a9eb00bc466649f8499ba0455f5
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Repo was added successfully.

  $ argocd repo list
TYPE  NAME                  REPO                                                                  INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE
helm  standard-service  xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com     false     true   false  true   Successful

We are using this OCI repo as a subchart dependency, also our ArgoCD application is generated using ApplicationSet

name: sample-service
version: "0.1.0"
kubeVersion: ">= 1.19.0-0"
description: chart to deploy a ccp-sample-service
dependencies:
  - name: standard-service
    repository: oci://xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
    version: 1.0.0

This is the error I get

rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add --username ****** --password ****** standard-service oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com` failed exit status 1: Error: looks like "oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com" is not a valid chart repository or cannot be reached: object required
mfbieber commented 2 years ago

Hi guys, sorry, I don't manage to work on this efficiently at the moment (too much going on).

In general, we abandoned the OCI approach in our projects, because it didn't seem to be properly supported with ECR - at least when we checked. Also what we really disliked is the workaround around logging in every x hours with a workaround CronJob: https://github.com/argoproj/argo-cd/issues/6062#issuecomment-859382021

alexef commented 1 year ago

I believe this issue is now fixed in v2.5.4, by: https://github.com/argoproj/argo-cd/pull/11327

@hedasaurabh / @pthornton can you confirm?

blakepettersson commented 1 year ago

Closing, we've been using ECR Helm charts in prod for quite some time.