Error creating configmap with Workload Identity

[eokorieFA]

I am struggling to get this feature working with Azure Workload Identity. I have an application that has been deployed into the cluster. The application already had a Workload Identity which it currently uses to fetch secrets from an Azure Keyvault and I can confirm this is working as intended. The identity has also been granted the role "App Configuration Data Reader".

However, when trying to get the App Configuration feature working, I am getting the following error:

Fail to create the target ConfigMap or Secret of AzureAppConfigurationProvider 'laravelapp-app-config' in 'laravelapp' namespace: no token file specified. Check pod configuration or set TokenFilePath in the options

I am making use of Terraform for the work I am doing and I also set the log versbosity to 3:

resource "kubernetes_namespace" "azure-app-config" {
  count = var.enable_aac ? 1 : 0
  metadata {
    name   = "azappconfig-system"
    labels = {
      deployed-by = "Terraform"
    }
  }
}

resource "helm_release" "azure-app-config" {
  count      = var.enable_aac ? 1 : 0
  name       = "azureappconfiguration.kubernetesprovider"
  repository = "oci://mcr.microsoft.com/azure-app-configuration/helmchart"
  chart      = "kubernetes-provider"
  namespace  = kubernetes_namespace.azure-app-config[0].metadata.0.name

  version = "1.2.0"

  timeout = 1800

  depends_on = [
    azurerm_kubernetes_cluster.k8s
  ]

  set {
    name = "env.azureClientId"
    value = var.azure_clientId
  }

  set {
    name = "env.azureTenantId"
    value = var.azure_tenantId
  }

  set {
    name = "logVerbosity"
    value = 3
  }

  values = [ <<-EOT
    nodeSelector:
      kubernetes.io/os: linux
  EOT
  ]
}

The above successfully installs the provider in the cluster. I also deploy my application using a helm chart, so I create a yaml file with for the application that I am working with, which looks like this:

{{- if .Values.azureAppConfiguration.enabled }}
apiVersion: azconfig.io/v1
kind: AzureAppConfigurationProvider
metadata:
  name: {{ include "laravelapp.fullname" . }}-app-config
  labels:
    {{- include "laravelapp.labels" . | nindent 4 }}
spec:
  endpoint: {{ .Values.azureAppConfiguration.storeEndpoint }}
  target:
    configMapName: configmap-app-configuration
#   keyValues:
#     selectors:
#       - keyFilter: Settings:*
#     keyVaults:
#       target:
#         secretName: my-secrets
  auth:
    workloadIdentity:
      managedIdentityClientId: {{ .Values.secretProviderClass.clientID }}
{{- end }}

The deployment.yaml file has been has had the following added:

spec:
  volumes:
    - name: app-config-volume
      configMap:
         name: configmap-app-configuration
 containers:
   - name: laravelapp
     volumeMounts:
       - name: app-config-volume
         mountPath: /app/config

However, upon deployment, I am getting a couple of issues:

The configmap does not get created

The app config volume refuses to be mounted.

Any help will be appreciated.

Thank you...

[linglingye001]

Hi @eokorieFA, have you created the federated identity credential of the workload managed identity? You can find more information about how to use workload identity in k8s provider reference doc. Please let me know if you have any further questions or encounter any issues. Thanks.

[eokorieFA]

hi @linglingye001,

I can confirm that the federated identity for the workload identity has been created. When checking the cluster and generated yaml for the laravel-app-config, I can see it has been pulled into the cluster.

The same idenity is being used to access Azure Keyvault secrets and the secrets are being pulled in with no issues...

[RichardChen820]

The error indicates the running pod can't find the federated token file that is supposed to be mounted. I believe you have confirmed that the workload identity is enabled on this cluster. Do you mind check and share the labels of the running appconfig provider deployment? It should has a label azure.workload.identity/use: "true"

[eokorieFA]

hi @RichardChen820,

I can confirm that label is there...

The deployment has the label

The pod has the label

The service account has the label... However, azure.workload.identity/client-id= is empty... Could this be the issue? Does the appconfig provider need a separate workload identity from the applications?

[RichardChen820]

@eokorieFA Do you mind double check the workloadIdentity status of your cluster?

az aks show  -g <your-cluster-resource-group> -n <your-cluster-name> --query "securityProfile"

From my end with workloadIdentity enabled, it shows:

[RichardChen820]

And should see such AZURE_FEDERATED_TOKEN_FILE env variable being set to your running pod of app config provider controller.

[eokorieFA]

This is what I have:

There is also an AZURE_FEDERATED_TOKEN_FILE env variable that has been set...

[RichardChen1122]

It's indeed bizarre, this is where the error throws. https://github.com/Azure/azure-sdk-for-go/blob/9b1d280f052510a0de6764582ba2766b3cf7c449/sdk/azidentity/workload_identity.go#L72

It only throws when the tokenFilePath and AZURE_FEDERATED_TOKEN_FILE env var is not set. But you've confirmed that env variable has been set, I'm a bit lost.

[eokorieFA]

@RichardChen1122 I changed a couple of things... I am not however getting:

Fail to create the target ConfigMap or Secret of AzureAppConfigurationProvider 'laravelapp-app-config' in 'laravelapp' namespace: invalid resource name "https://appconfigstore-example.azconfig.io": [may not contain '/']

I feel this is getting close to being resolved :-)

[RichardChen820]

@eokorieFA Are you using connectionStringReference field now? Most likely you are using such a yaml:

apiVersion: azconfig.io/v1
kind: AzureAppConfigurationProvider
metadata:
  name: appconfigurationprovider-sample
spec:
  connectionStringReference: https://appconfigstore-example.azconfig.io  #This line is not right.
  target:
    configMapName: configmap-created-by-appconfig-provider

The connectionStringReference field must be populated with the name of a Secret which contains the connectionString of the target app config store. Please see here for the detail of using connection string

[RichardChen820]

To make connection string work, you should:

1. Create a Secret with connectionString of target app config store

   kubectl create secret generic appconfig-secret --from-literal azure_app_configuration_connection_string=<read-only connection string of target app config store> -n <same namespace of azureappconfigurationprovider resource>

2. Set the name of created secret to the AzureAppConfigurationProvder yaml

   apiVersion: azconfig.io/v1
   kind: AzureAppConfigurationProvider
   metadata:
   name: appconfigurationprovider-sample
   namespace: <your namespace>
   spec:
   connectionStringReference: appconfig-secret
   target:
   configMapName: configmap-created-by-appconfig-provider

[eokorieFA]

@RichardChen820. Bit of an update. Kind of had to go back to the drawing board on this. Ended up creating a separate workload identity and patched the appconfig provider with the new client id. This eventually pointed me to a new set of errors. From those errors, I deduced that the provider's workload id needed to be applied to the laravelapp I created. For some reason, the laravelapp workload identity didnt want to work but using the provider's identity did. I also fixed the AzureAppConfigurationProvder yaml file in the application.

The configmap has now been created which I am pleased about...

[RichardChen820]

@eokorieFA Excellent, although I still can't get what exact problem you were facing, glad to know you eventually settled it. I'd appreciate it if you have more details to share. It most likely could help us make some improvement to the production or document, which can help others to avoid scrambling make the workload identity work.

[eokorieFA]

@RichardChen820 No worries... Leave it with me, I will get back to asap with the step I followed to resolve some of my issues. Mind you what I did was with Terraform, would it be ok in that case to provide the Terraform code that I used just in case it may prove useful...

[RichardChen820]

Close this issue, feel free to re-open it if you have more questions.