AWS KMS - secret not decrypted

I have IAM service account with automount token which allows to use KMS to decrypt sops file. When I'm running helm secrets decrypt on repo container, I can see decrypted values. The issue is when I'm trying to apply changes from ArgoCD app, then ArgoCD is literally injecting ENC[AES256... values.
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "8"
    meta.helm.sh/release-name: argo-cd
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-11-12T08:39:55Z"
  generation: 9
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-repo-server
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.13.0
    helm.sh/chart: argo-cd-7.7.1
  name: argo-cd-argocd-repo-server
  namespace: default
  resourceVersion: "1802372"
  uid: 9969f128-89e0-45b1-b77d-551753b9ccc1
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: argo-cd
      app.kubernetes.io/name: argocd-repo-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/cm: 025e7a2e86780d44bc5178c6d5faeda6b65e571eb6e41503cd85a0740bec2b6b
        checksum/cmd-params: ebd9fbd922fc090b8c3a2a28188a43d51426faaee52133addfdf402722248b02
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: repo-server
        app.kubernetes.io/instance: argo-cd
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: argocd-repo-server
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: v2.13.0
        helm.sh/chart: argo-cd-7.7.1
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: argocd-repo-server
              topologyKey: kubernetes.io/hostname
            weight: 100
      automountServiceAccountToken: true
      containers:
      - args:
        - /usr/local/bin/argocd-repo-server
        - --port=8081
        - --metrics-port=8084
        env:
        - name: ARGOCD_REPO_SERVER_NAME
          value: argo-cd-argocd-repo-server
        - name: ARGOCD_RECONCILIATION_TIMEOUT
          valueFrom:
            configMapKeyRef:
              key: timeout.reconciliation
              name: argocd-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_LOGFORMAT
          valueFrom:
            configMapKeyRef:
              key: reposerver.log.format
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_LOGLEVEL
          valueFrom:
            configMapKeyRef:
              key: reposerver.log.level
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_PARALLELISM_LIMIT
          valueFrom:
            configMapKeyRef:
              key: reposerver.parallelism.limit
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_LISTEN_ADDRESS
          valueFrom:
            configMapKeyRef:
              key: reposerver.listen.address
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_LISTEN_METRICS_ADDRESS
          valueFrom:
            configMapKeyRef:
              key: reposerver.metrics.listen.address
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_DISABLE_TLS
          valueFrom:
            configMapKeyRef:
              key: reposerver.disable.tls
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_TLS_MIN_VERSION
          valueFrom:
            configMapKeyRef:
              key: reposerver.tls.minversion
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_TLS_MAX_VERSION
          valueFrom:
            configMapKeyRef:
              key: reposerver.tls.maxversion
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_TLS_CIPHERS
          valueFrom:
            configMapKeyRef:
              key: reposerver.tls.ciphers
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_CACHE_EXPIRATION
          valueFrom:
            configMapKeyRef:
              key: reposerver.repo.cache.expiration
              name: argocd-cmd-params-cm
              optional: true
        - name: REDIS_SERVER
          valueFrom:
            configMapKeyRef:
              key: redis.server
              name: argocd-cmd-params-cm
              optional: true
        - name: REDIS_COMPRESSION
          valueFrom:
            configMapKeyRef:
              key: redis.compression
              name: argocd-cmd-params-cm
              optional: true
        - name: REDISDB
          valueFrom:
            configMapKeyRef:
              key: redis.db
              name: argocd-cmd-params-cm
              optional: true
        - name: REDIS_USERNAME
          valueFrom:
            secretKeyRef:
              key: redis-username
              name: argocd-redis
              optional: true
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              key: auth
              name: argocd-redis
        - name: REDIS_SENTINEL_USERNAME
          valueFrom:
            secretKeyRef:
              key: redis-sentinel-username
              name: argo-cd-redis-ha-haproxy
              optional: true
        - name: REDIS_SENTINEL_PASSWORD
          valueFrom:
            secretKeyRef:
              key: redis-sentinel-password
              name: argo-cd-redis-ha-haproxy
              optional: true
        - name: ARGOCD_DEFAULT_CACHE_EXPIRATION
          valueFrom:
            configMapKeyRef:
              key: reposerver.default.cache.expiration
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_OTLP_ADDRESS
          valueFrom:
            configMapKeyRef:
              key: otlp.address
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_OTLP_INSECURE
          valueFrom:
            configMapKeyRef:
              key: otlp.insecure
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_OTLP_HEADERS
          valueFrom:
            configMapKeyRef:
              key: otlp.headers
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
          valueFrom:
            configMapKeyRef:
              key: reposerver.max.combined.directory.manifests.size
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
          valueFrom:
            configMapKeyRef:
              key: reposerver.plugin.tar.exclusions
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_ALLOW_OUT_OF_BOUNDS_SYMLINKS
          valueFrom:
            configMapKeyRef:
              key: reposerver.allow.oob.symlinks
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_STREAMED_MANIFEST_MAX_TAR_SIZE
          valueFrom:
            configMapKeyRef:
              key: reposerver.streamed.manifest.max.tar.size
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_STREAMED_MANIFEST_MAX_EXTRACTED_SIZE
          valueFrom:
            configMapKeyRef:
              key: reposerver.streamed.manifest.max.extracted.size
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_HELM_MANIFEST_MAX_EXTRACTED_SIZE
          valueFrom:
            configMapKeyRef:
              key: reposerver.helm.manifest.max.extracted.size
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_DISABLE_HELM_MANIFEST_MAX_EXTRACTED_SIZE
          valueFrom:
            configMapKeyRef:
              key: reposerver.disable.helm.manifest.max.extracted.size
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_GIT_MODULES_ENABLED
          valueFrom:
            configMapKeyRef:
              key: reposerver.enable.git.submodule
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_GIT_LS_REMOTE_PARALLELISM_LIMIT
          valueFrom:
            configMapKeyRef:
              key: reposerver.git.lsremote.parallelism.limit
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_GIT_REQUEST_TIMEOUT
          valueFrom:
            configMapKeyRef:
              key: reposerver.git.request.timeout
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REVISION_CACHE_LOCK_TIMEOUT
          valueFrom:
            configMapKeyRef:
              key: reposerver.revision.cache.lock.timeout
              name: argocd-cmd-params-cm
              optional: true
        - name: ARGOCD_REPO_SERVER_INCLUDE_HIDDEN_DIRECTORIES
          valueFrom:
            configMapKeyRef:
              key: reposerver.include.hidden.directories
              name: argocd-cmd-params-cm
              optional: true
        - name: HELM_CACHE_HOME
          value: /helm-working-dir
        - name: HELM_CONFIG_HOME
          value: /helm-working-dir
        - name: HELM_DATA_HOME
          value: /helm-working-dir
        - name: HELM_PLUGINS
          value: /gitops-tools/helm-plugins/
        - name: HELM_SECRETS_CURL_PATH
          value: /gitops-tools/curl
        - name: HELM_SECRETS_SOPS_PATH
          value: /gitops-tools/sops
        - name: HELM_SECRETS_VALS_PATH
          value: /gitops-tools/vals
        - name: HELM_SECRETS_KUBECTL_PATH
          value: /gitops-tools/kubectl
        - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
          value: "true"
        - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
          value: "true"
        - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
          value: "false"
        - name: HELM_SECRETS_WRAPPER_ENABLED
          value: "true"
        - name: HELM_SECRETS_DECRYPT_SECRETS_IN_TMP_DIR
          value: "true"
        - name: HELM_SECRETS_HELM_PATH
          value: /usr/local/bin/helm
        - name: HELM_SECRETS_BACKEND
          value: sops
        image: quay.io/argoproj/argocd:v2.13.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz?full=true
            port: metrics
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: repo-server
        ports:
        - containerPort: 8081
          name: repo-server
          protocol: TCP
        - containerPort: 8084
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: metrics
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /helm-secrets-private-keys/
          name: helm-secrets-private-keys
        - mountPath: /gitops-tools
          name: gitops-tools
        - mountPath: /usr/local/sbin/helm
          name: gitops-tools
          subPath: helm
        - mountPath: /app/config/ssh
          name: ssh-known-hosts
        - mountPath: /app/config/tls
          name: tls-certs
        - mountPath: /app/config/gpg/source
          name: gpg-keys
        - mountPath: /app/config/gpg/keys
          name: gpg-keyring
        - mountPath: /app/config/reposerver/tls
          name: argocd-repo-server-tls
        - mountPath: /helm-working-dir
          name: helm-working-dir
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/cp
        - -n
        - /usr/local/bin/argocd
        - /var/run/argocd/argocd-cmp-server
        image: quay.io/argoproj/argocd:v2.13.0
        imagePullPolicy: IfNotPresent
        name: copyutil
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
      - args:
        - |
          mkdir -p /gitops-tools/helm-plugins

          GO_ARCH=$(uname -m | sed -e 's/x86_64/amd64/')
          wget -qO /gitops-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-${GO_ARCH}

          GO_ARCH=$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/') && \
          wget -qO /gitops-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${GO_ARCH}/kubectl
          wget -qO /gitops-tools/sops https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.${GO_ARCH}
          wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_${GO_ARCH}.tar.gz | tar zxv -C /gitops-tools vals
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /gitops-tools/helm-plugins -xzf-

          chmod +x /gitops-tools/*
          cp /gitops-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /gitops-tools/helm
        command:
        - sh
        - -ec
        env:
        - name: HELM_SECRETS_VERSION
          value: 4.6.2
        - name: KUBECTL_VERSION
          value: 1.31.0
        - name: VALS_VERSION
          value: 0.37.3
        - name: SOPS_VERSION
          value: 3.7.3
        image: alpine:latest
        imagePullPolicy: IfNotPresent
        name: download-tools
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /gitops-tools
          name: gitops-tools
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsUser: 999
      serviceAccount: argo-cd-argocd-repo-server
      serviceAccountName: argo-cd-argocd-repo-server
      terminationGracePeriodSeconds: 30
      volumes:
      - name: helm-secrets-private-keys
        secret:
          defaultMode: 420
          secretName: helm-secrets-private-keys
      - emptyDir: {}
        name: gitops-tools
      - emptyDir: {}
        name: helm-working-dir
      - emptyDir: {}
        name: plugins
      - emptyDir: {}
        name: var-files
      - emptyDir: {}
        name: tmp
      - configMap:
          defaultMode: 420
          name: argocd-ssh-known-hosts-cm
        name: ssh-known-hosts
      - configMap:
          defaultMode: 420
          name: argocd-tls-certs-cm
        name: tls-certs
      - configMap:
          defaultMode: 420
          name: argocd-gpg-keys-cm
        name: gpg-keys
      - emptyDir: {}
        name: gpg-keyring
      - name: argocd-repo-server-tls
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          - key: ca.crt
            path: ca.crt
          optional: true
          secretName: argocd-repo-server-tls
argoproj / argo-cd

AWS KMS - secret not decrypted #20781
