Environment variables are not read from K8S secret

Hello! Currently I'm working on to connect the ArgoCD to external cluster, and for that I would like to use the Workload Identity flow on AKS. The setup is done and working with a workaround. Previously I used the local account's certs to connect, but I would like to change this for a more secure way.

Checklist:

[✓] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[✓] I've included steps to reproduce the bug.
[✓] I've pasted the output of argocd version.

Describe the bug

Based on the documentation, to enable the workload identity, I have to setup the k8s secret (beside other steps) like this:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster.example.com
  server: https://mycluster.example.com
  config: |
    {
      "execProviderConfig": {
        "command": "argocd-k8s-auth",
        "env": {
          "AAD_ENVIRONMENT_NAME": "AzurePublicCloud",
          "AZURE_CLIENT_ID": "fill in client id",
          "AZURE_TENANT_ID": "fill in tenant id", # optional, injected by workload identity mutating admission webhook if enabled
          "AZURE_FEDERATED_TOKEN_FILE": "/opt/path/to/federated_file.json", # optional, injected by workload identity mutating admission webhook if enabled
          "AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com/", # optional, injected by workload identity mutating admission webhook if enabled
          "AAD_LOGIN_METHOD": "workloadidentity"
        },
        "args": ["azure"],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 encoded certificate>"
      }
    }

My current setup is in terraform, and looks like this:

resource "kubernetes_secret" "argocd_destination" {
  metadata {
    name      = "secret_name"
    namespace = argocd
    labels = {
      "argocd.argoproj.io/secret-type" = "cluster"
    }
  }

  data = {
    name   = "target_cluster_name"
    server = data.azurerm_kubernetes_cluster.target_cluster.kube_config[0].host
    config = jsonencode({
      execProviderConfig = {
        apiVersion = "client.authentication.k8s.io/v1beta1"
        command    = "argocd-k8s-auth"
        args = [
          "azure"
        ]
        env = {
          "AAD_LOGIN_METHOD"           = "workloadidentity"
          "AAD_ENVIRONMENT_NAME"       = "AzurePublicCloud"
          "AZURE_CLIENT_ID"            = "very good and valid clientId"
          "AZURE_TENANT_ID"            = "very good and valid tenantId"
          "AZURE_FEDERATED_TOKEN_FILE" = "very good and valid path to the tokenfile"
        }
      }
      tlsClientConfig = {
        insecure = false,
        caData   = data.azurerm_kubernetes_cluster.target_cluster.kube_config[0].cluster_ca_certificate
      }
    })
  }
}

Even if I set another value for example in the AAD_LOGIN_METHOD, it doesn't change anything, I'm getting the following error: "getting credentials: exec: executable argocd-k8s-auth failed with exit code 20"

As a workaround, I moved the environment variable settings to the K8S pod settings (via Kustomizer patch), and it works correctly.

I checked the Kubernetes, and the was created successfully with the good value, so the issue is not between the terraform and k8s. This secret previously used the admin's cert, and in that way it was working without issue, so the ArgoCD is picking up the secret correctly.

To Reproduce

Create a k8s secret based execProviderConfig for the ArgoCD, and try modify any environment variables.

Expected behavior

The environments parameters should be picked up from the secret.

Version

argocd: v2.8.2+dbdfc71
  BuildDate: 2023-08-24T20:05:39Z
  GitCommit: dbdfc712702ce2f781910a795d2e5385a4f5a0f9
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64

argoproj / argo-cd

Environment variables are not read from K8S secret #20366