Failed to get [`<key>`] from Configuration store `<configstorename>`: context deadline exceeded

[cilerler]

BUG REPORT: Failed to get [<key>] from Configuration store <configstorename>: context deadline exceeded

Issue description

I'm trying to use DAPR with ACA. I created a lean example that attempts to access DAPR through both URL and SDK to retrieve configuration settings. It works on my local machine without any issues, but on Azure, it gives a context deadline exceeded error with the SDK and a 500 error with the URL.

Steps to reproduce

1. Create Azure App Configuration
2. Create Azure Container App
3. Define AAC in ACA DAPR component
4. Enable DAPR on ACA
5. Deploy an image to ACA

You can use the same sample project I use.

[!NOTE]
The following steps are optional but allow you to easily try out the repository. The workflow file building the image and updating the container image on Azure Container App.

Update the GitHub repository settings to deploy it via the action.

1. Set CONTAINER_APP_NAME_PREFIX action variable e.g. HelloDapr
2. Set AZURE_CONTAINERREGISTRY action secret e.g. <YOURREGISTRY>.azurecr.io

3. Set AZURE_CREDENTIALS action secret e.g.

   {
       "clientId": "<GUID>",
       "clientSecret": "<SECRET>",
       "subscriptionId": "<GUID>",
       "tenantId": "<GUID>"
   }

EXPECTED BEHAVIOR

It should retrieve the key via HTTP and GRPC.

ACTUAL BEHAVIOR

• HTTP returns 500 (Internal Server Error)
• GRPS returns context deadline exceeded

LOGS

CONSOLE

APP

2023-10-21T20:02:00.32025  Connecting to the container 'api'...
2023-10-21T20:02:00.34978  Successfully Connected to container: 'api' [Revision: 'spiderman-api--erqztf8', Replica: 'spiderman-api--erqztf8-5f5fcdbf65-km7c5']
2023-10-21T19:35:04.652239857Z warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
2023-10-21T19:35:04.652276589Z       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
2023-10-21T19:35:05.028070936Z warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
2023-10-21T19:35:05.028088872Z       No XML encryptor configured. Key {d040c521-ec88-4345-b036-fe00e1b8527c} may be persisted to storage in unencrypted form.
2023-10-21T19:35:05.129410863Z info: Microsoft.Hosting.Lifetime[14]
2023-10-21T19:35:05.129443288Z       Now listening on: http://[::]:80
2023-10-21T19:35:05.130630039Z info: Microsoft.Hosting.Lifetime[0]
2023-10-21T19:35:05.130643665Z       Application started. Press Ctrl+C to shut down.
2023-10-21T19:35:05.131248175Z info: Microsoft.Hosting.Lifetime[0]
2023-10-21T19:35:05.131259339Z       Hosting environment: production
2023-10-21T19:35:05.131262494Z info: Microsoft.Hosting.Lifetime[0]
2023-10-21T19:35:05.131265985Z       Content root path: /app
2023-10-21T20:01:08.023779368Z warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
2023-10-21T20:01:08.023829142Z       Failed to determine the https port for redirect.
2023-10-21T20:01:08.531823867Z info: HelloDapr.Api.Data.MyDaprService[0]
2023-10-21T20:01:08.531866061Z       Calling http://127.0.0.1:3500/v1.0/configuration/configuration/?key=Test
2023-10-21T20:01:39.725101500Z info: HelloDapr.Api.Data.MyDaprService[0]
2023-10-21T20:01:39.725150041Z       Calling http://127.0.0.1:3500/v1.0/configuration/configuration/?key=Test

DAPR

2023-10-21T20:02:30.42876  Connecting to the container 'daprd'...

2023-10-21T20:02:30.44776  Successfully Connected to container: 'daprd' [Revision: 'spiderman-api--erqztf8', Replica: 'spiderman-api--erqztf8-5f5fcdbf65-km7c5']
2023-10-21T19:35:04.270342938Z time="2023-10-21T19:35:04.270274994Z" level=info msg="starting workload cert expiry watcher. current cert expires on: 2023-10-22 19:50:04 +0000 UTC" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.grpc.internal type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.176065289Z time="2023-10-21T19:35:05.175934626Z" level=info msg="application discovered on port 80" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.176236750Z time="2023-10-21T19:35:05.176180825Z" level=info msg="actors: state store is not configured - this is okay for clients but services with hosted actors will fail to initialize!" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.176295931Z time="2023-10-21T19:35:05.176232072Z" level=info msg="actor runtime started. actor idle timeout: 1h0m0s. actor scan interval: 30s" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.actor type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.176301724Z time="2023-10-21T19:35:05.176264666Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 956ms" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.176448501Z time="2023-10-21T19:35:05.17637965Z" level=debug msg="try to connect to placement service: dns:///dapr-placement-server.k8se-system.svc.cluster.local:50005" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.actor.internal.placement type=log ver=1.11.2-msft-3
2023-10-21T19:35:05.205129962Z time="2023-10-21T19:35:05.20502444Z" level=debug msg="error connecting to placement service (will retry to connect in background): rpc error: code = Unavailable desc = last resolver error: produced zero addresses" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.actor.internal.placement type=log ver=1.11.2-msft-3
2023-10-21T19:38:47.395958156Z time="2023-10-21T19:38:47.395828595Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/metadata" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T19:43:48.597277170Z time="2023-10-21T19:43:48.597052557Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/metadata" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T19:48:46.595547154Z time="2023-10-21T19:48:46.595403459Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/metadata" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T19:53:46.994447090Z time="2023-10-21T19:53:46.99430489Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/metadata" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T19:58:50.697576767Z time="2023-10-21T19:58:50.697448344Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/metadata" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T20:01:08.728997351Z time="2023-10-21T20:01:08.728859525Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/configuration/{storeName}" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T20:01:23.729924688Z time="2023-10-21T20:01:23.729818212Z" level=debug msg="{ERR_CONFIGURATION_GET failed to get [Test] from Configuration store configuration: context deadline exceeded}" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.http type=log ver=1.11.2-msft-3
2023-10-21T20:01:24.024238807Z time="2023-10-21T20:01:24.02381985Z" level=info msg="gRPC API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method=/dapr.proto.runtime.v1.Dapr/GetConfiguration scope=dapr.runtime.grpc.api-info type=log useragent="grpc-dotnet/2.52.0 (.NET 7.0.12; CLR 7.0.12; net7.0; linux; x64) dapr-sdk-dotnet/v1.12.0+b0992c306bbc30cea8932344db425c974ffcfea5" ver=1.11.2-msft-3
2023-10-21T20:01:39.024938817Z time="2023-10-21T20:01:39.024807525Z" level=debug msg="rpc error: code = Internal desc = failed to get [Test] from Configuration store configuration: context deadline exceeded" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.grpc.api type=log ver=1.11.2-msft-3
2023-10-21T20:01:39.729550466Z time="2023-10-21T20:01:39.729462645Z" level=info msg="HTTP API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method="GET /v1.0/configuration/{storeName}" scope=dapr.runtime.http-info type=log ver=1.11.2-msft-3
2023-10-21T20:01:54.730306895Z time="2023-10-21T20:01:54.730183048Z" level=debug msg="{ERR_CONFIGURATION_GET failed to get [Test] from Configuration store configuration: context deadline exceeded}" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.http type=log ver=1.11.2-msft-3
2023-10-21T20:01:54.732501318Z time="2023-10-21T20:01:54.73240344Z" level=info msg="gRPC API Called" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 method=/dapr.proto.runtime.v1.Dapr/GetConfiguration scope=dapr.runtime.grpc.api-info type=log useragent="grpc-dotnet/2.52.0 (.NET 7.0.12; CLR 7.0.12; net7.0; linux; x64) dapr-sdk-dotnet/v1.12.0+b0992c306bbc30cea8932344db425c974ffcfea5" ver=1.11.2-msft-3
2023-10-21T20:02:09.732878431Z time="2023-10-21T20:02:09.732760538Z" level=debug msg="rpc error: code = Internal desc = failed to get [Test] from Configuration store configuration: context deadline exceeded" app_id=api instance=spiderman-api--erqztf8-5f5fcdbf65-km7c5 scope=dapr.runtime.grpc.api type=log ver=1.11.2-msft-3

SCREENSHOTS

ON LOCAL

✅ Working as expected

ON AZURE CONTAINER APPS

❌ Not working

ADDITIONAL CONTEXT

DAPR SETUP

[!NOTE]
The screenshot shows that GRPC is selected, but I also tested it with HTTP.

CONFIGSTORE SETUP

CONFIG ENTRY

CREDENTIALS

[shivamkm07]

Hi @cilerler, can you please help with the following:

1. When you are running in local, you are connecting to the same App config store resource right? What's the authentication mechanism are you using in local?
2. It possibly might be the access issue i.e., App config store not accessible from the app. Please re-verify if the managed identity is setup correctly.
3. Can you provide with the containerappName and environmentName? I was able to find one resource with cAppName spiderman-api and environmentName thankfulsmoke-3a10a192 under the same subscription with similar errors. But it seems to be created on 2023-10-23T12:57:47.308294(the logs shared above are of 2023-10-21). Also in this case, the errors are because dapr is shutting down before the Get Calls are being made and so the failures:

[shivamkm07]

Hi @cilerler, can you please help with the following:

1. When you are running in local, you are connecting to the same App config store resource right? What's the authentication mechanism are you using in local?
2. It possibly might be the access issue i.e., App config store not accessible from the app. Please re-verify if the managed identity is setup correctly.
3. Can you provide with the containerappName and environmentName? I was able to find one resource with cAppName spiderman-api and environmentName thankfulsmoke-3a10a192 under the same subscription with similar errors. But it seems to be created on 2023-10-23T12:57:47.308294(the logs shared above are of 2023-10-21). Also in this case, the errors are because dapr is shutting down before the Get Calls are being made and so the failures:

I can see another capps environment on the mentioned time kindcoast-bd7979a0 with the same logs. can you confirm if this is the one where you saw the errors mentioned in the issue? Here also Dapr shuts down first and then Get call is being made, and so it fails.

[cilerler]

Thank you very much for looking into this. 🤗

(3) Here are all the names and types I don't recall thanksfulsmoke or kindcoast and I do not see it in my list

(1) In my local DAPR, I'm not connecting to Azure; I do use local Redis. But when I use the regular credentials against Azure App Configuration via LinqPad, it works.

(2) I found an ugly workaround:

• I removed and re-created the AppConfigStore in DAPR settings (not the actual Azure App Configuration) of the ACA Environment manually.
• Then, I disabled and enabled DAPR in ACA itself,

and it worked.

So I'm not sure why it is happening, but it is a persistent issue. I'm using Terraform to create it all, and I'm now 100% able to reproduce the issue and the workaround. I can share the Terraform script with you if it helps.

[shivamkm07]

Yeah Please share the Terraform script. I will try to repro the same on my end.

[cilerler]

I added it to the same repo above

the steps needs to be follow to reproduce the issue are below, and if you prefer I can join you for screenshare. all the commands exists in the readme in the link above.

1. init terraform
2. apply terraform
3. output terraform credentials and registry name
4. clone the repository on github
5. add credentials and registry name to the secrets of the github
6. trigger manual action to deploy the image

congratulations 🥳 🐞

[shivamkm07]

I added it to the same repo above

the steps needs to be follow to reproduce the issue are below, and if you prefer I can join you for screenshare. all the commands exists in the readme in the link above.

1. init terraform
2. apply terraform
3. output terraform credentials and registry name
4. clone the repository on github
5. add credentials and registry name to the secrets of the github
6. trigger manual action to deploy the image

congratulations 🥳 🐞

I am able to repro the issue. Looking into it further.

[chielos24]

hi, i;m experiencing the same issue. It works with keyvault, but not with config.

[shivamkm07]

hi, i;m experiencing the same issue. It works with keyvault, but not with config.

Do you mean to sya using AKV as a secretstore is working, but AAC as a configstore is not?

[shivamkm07]

I added it to the same repo above

the steps needs to be follow to reproduce the issue are below, and if you prefer I can join you for screenshare. all the commands exists in the readme in the link above.

1. init terraform
2. apply terraform
3. output terraform credentials and registry name
4. clone the repository on github
5. add credentials and registry name to the secrets of the github
6. trigger manual action to deploy the image

congratulations 🥳 🐞

@cilerler The workaround doesn't seem to be working for me though yet. We can get into a call to discuss if that's okay. Pinged you on Discord.

[chielos24]

hi, i;m experiencing the same issue. It works with keyvault, but not with config.

Do you mean to sya using AKV as a secretstore is working, but AAC as a configstore is not?

Exactly. Workaround is also not working for me.

[cilerler]

@chielos24, I just demonstrated it to @shivamkm07 and proved that the workaround is effective for the issue I reported. One suggestion I would make is that the order of steps is important:

1. After the Terraform deployment, make sure to create a key named 'Test' in AAC.
2. Then, in ACE, navigate to the DAPR section, remove the 'configuration' component, and re-add it using the 'Azure component (preview)' option.
3. Finally, in ACA, go to the DAPR section, 'Disable' it, and then 'Enable' it.

[chielos24]

Great, will try this!

But i am looking for a future proof solution. I am building a production system, that is constantly updated via CI/CD pipelines. Any idea on why this is happening? Is it a timing issue?

[cilerler]

I have no idea. But I'm looking forward to @shivamkm07's update.

[shivamkm07]

Update:

1. The workaround mentioned above works on my end as well.
2. At some instances, after deployment via terraform and running the application, I got a different error: ChainedTokenCredential: failed to acquire a token. Attempted credentials: managed identity AzureCLICredential: fork/exec /bin/sh: no such file or directory. Even on restarting and Adding the Key in configuration store, still got the same error and after some more restarts, the correct error i.e. failed to get key.... It seems non-deterministic. There were runs when I didn't get this error at all. (Note here I hadn't deleted and re-created the component, that fixes the issue, yes!)
3. I tried deploying the same application via terraform with Dapr component using connectionString to authenticate to AAC and it works! No error in this case.

So, in essence, the error only surfaces when we are deploying application with Terraform and using Managed identity. And again, as the workaround says, the same deployment, on manually re-creating the component fixes the issue.

[cilerler]

Thanks for your efforts and willingness to help uncover the real reason. I wonder if this would happen when running CLI commands. As far as I know, Terraform uses az behind the scenes, so could it be something in the CLI? Regardless, I will reach out to the Terraform repo and link to this issue. Thank you very much again.

[!Note] I will keep the issue open until one of us confirms that it is working with az

[cilerler]

UPDATED According to Microsoft's article, it should actually work just fine. I don't think this is an issue on Terraform's end.

I believe the issue is that the CLI doesn't directly support creating Azure components, and that's likely the issue. ~So, Terraform team probably came up with a solution to make it work, but it seems it's not working as expected.~

@shivamkm07 what do you think about it?

[chielos24]

Actually i am not using TF, i'm using regular azure cli

[cilerler]

That's even better, it simply removes the terraform from the equation. Can you share your YAML with us to use below?

az containerapp env dapr-component set -g $deploymentResourceGroup -n $deploymentContainerAppEnvironment --dapr-component-name "configuration" --yaml ""

[shivamkm07]

That's even better, it simply removes the terraform from the equation. Can you share your YAML with us to use below?

az containerapp env dapr-component set -g $deploymentResourceGroup -n $deploymentContainerAppEnvironment --dapr-component-name "configuration" --yaml ""

Yeah @chielos24 Can you share the steps that you are doing? I was also thinking terraform might have some issue here but seems it's not

[chielos24]

We initial setup with (.. are removed):

az containerapp create \
    --name backoffice-api \
    --resource-group .. \
    --environment .. \
    --image ../container/backoffice-api:latest \
    --registry-server .. \
    --registry-username .. \
    --registry-password .. \
    --user-assigned .. \
    --min-replicas 1 \
    --max-replicas 1 \
    --dapr-app-id backoffice-api \
    --enable-dapr \
    --dapr-app-port 80

az containerapp up \
    --name backoffice-api \
    --resource-group .. \
    --image ../container/backoffice-api:latest \
    --environment .. \
    --registry-server .. \
    --logs-workspace-id .. \
    --logs-workspace-key .. \
    --registry-username .. \
    --registry-password .. \
    --service-principal-client-id .. \
    --service-principal-client-secret j.. \
    --service-principal-tenant-id ..

And Component

az containerapp env dapr-component set \
    --name .. \
    --resource-group .. \
    --yaml ./dapr/config.yaml \
    --dapr-component-name ..-config

config.yaml

componentType: configuration.azure.appconfig
version: v1
metadata:
- name: host
  value: https://<appcs>.azconfig.io
- name: azureTenantId
  value: ".."
- name: azureClientId
  value: ".."
scopes:
  - backoffice-api

CI/CD via Azure Devops: az containerapp update -n backoffice-api -g <rg> -i n<cr>/container/backoffice-api:latest

/edit should be correct now.

[cilerler]

I'm not sure if it makes any difference, but the first thing I see is config.yml is different, are you sure that it is a valid syntax?

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: <NAME>
spec:
  type: configuration.azure.appconfig
  version: v1
  metadata:
  - name: host # host should be used when Azure Authentication mechanism is used.
    value: <HOST>
  - name: connectionString # connectionString should not be used when Azure Authentication mechanism is used.
    value: <CONNECTIONSTRING>
  - name: maxRetries
    value: # Optional
  - name: retryDelay
    value: # Optional
  - name: maxRetryDelay
    value: # Optional
  - name: azureEnvironment # Optional, defaults to AZUREPUBLICCLOUD
    value: "AZUREPUBLICCLOUD"
  # See authentication section below for all options
  - name: azureTenantId # Optional
    value: "[your_service_principal_tenant_id]"
  - name: azureClientId # Optional
    value: "[your_service_principal_app_id]"
  - name: azureCertificateFile # Optional
    value : "[pfx_certificate_file_fully_qualified_local_path]"
  - name: subscribePollInterval # Optional
    value: #Optional [Expected format example - 30s]

[chielos24]

yeah, something is up with that.

And i just something else. I tried to create the app config via the portal. And it did not allow me to use dashes in the component name i had "companyname-productname-config" and that did work via CLI, but portal did not.

Retrying now with only "config"

edit..

Apparently that was the trick for me.. So no dashes.

edit2. Not sure yet, still working on it.

[chielos24]

i'm sort of stable now. When re-creating the api (delete, create and up) my config stays, and i can access some keys.

however, when i ask a couple of keys like:

var keys = new List<string>() {
            "ConnectionStrings:AccountsDb",
            "AzureAd",
            "AzureAdB2C:ApplicationId",
            "AzureAdB2C:ApplicationSecret",
            "AzureAdB2C:ExtensionId",
            "AzureAdB2C:TenantId",
            "MicrosoftDynamics"
        };

Then the 5th key is not working anymore, and i get the same error. Tried to do it in seperate calls (chunks of 4) did not help me. No idea why, is there a way to get more logging.

The strange thing also, is that when doing AKS, i had this solution working, even with 12 keys.

[cilerler]

@chielos24 Great news. However, to keep things focused, please open a new issue for your concern. It's not related to the initial issue we're addressing anymore.

That said, here's a head start for you: If you need more logs, feel free to enable DAPR logs, and don't forget to set it to 'Debug' as shown below.

[chielos24]

Thanks! Will do!

[shivamkm07]

Update: To narrow down the issue, I used a pre-existing managed identity resource and pre-created configuration store(with Test key added already) and then used both Terraform and az cli to create just the ACA environment, app and the Dapr component.

The terraform script that I used:

terraform {
  required_version = ">=1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 3.4.0"
    }
  }
}

provider "random" {}

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy    = true
      recover_soft_deleted_key_vaults = true
    }
  }
}

variable "environment" {
  type    = string
  default = "development"
}

variable "environment_variable_name" {
  type    = string
  default = "ASPNETCORE_ENVIRONMENT"
}

variable "location" {
  type    = string
  default = "eastus"
}

variable "data_location" {
  type    = string
  default = "United States"
}

variable "app_code_name" {
  type = string
}

variable "container_app_environment" {
  type = string
}

variable "container_app_suffix_api" {
  type = string
}

locals {
  resource_group_name = "${var.app_code_name}-${var.environment}"
  container_app_api = "${var.app_code_name}-${var.container_app_suffix_api}"
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "environment" {
  name     = local.resource_group_name
  location = var.location
}

resource "azurerm_container_app_environment" "container_app_environment" {
  name                       = var.container_app_environment
  location                   = azurerm_resource_group.environment.location
  resource_group_name        = azurerm_resource_group.environment.name
}

resource "azurerm_container_app_environment_dapr_component" "dapr_component_configurationstore" {
  name                         = "configuration"
  container_app_environment_id = azurerm_container_app_environment.container_app_environment.id
  component_type               = "configuration.azure.appconfig"
  version                      = "v1"
  metadata {
    name  = "host"
    value = "<CONFIG_STORE_HOST>"
  }
  metadata {
    name  = "azureClientId"
    value = "<IDENTITY_CLIENT_ID>"
  }
  scopes = [
    var.container_app_suffix_api
  ]
}

resource "azurerm_container_app" "container_app_api" {
  name                         = local.container_app_api
  resource_group_name          = azurerm_resource_group.environment.name
  container_app_environment_id = azurerm_container_app_environment.container_app_environment.id
  revision_mode                = "Single"

  template {
    container {
      name   = var.container_app_suffix_api
      image  = "<IMAGE>"
      cpu    = 0.25
      memory = "0.5Gi"
    }
    min_replicas = 1
    max_replicas = 1
  }
   lifecycle {
    ignore_changes = [
      template[0].container[0].image,
    ]
  }
  dapr {
    app_id       = var.container_app_suffix_api
    app_port     = 80
    app_protocol = "grpc"

  }
}

The equivalent az cli script that I used is below:

RESOURCE_GROUP=<RG_NAME>
LOCATION=>LOCATION>
CONTAINERAPPS_ENVIRONMENT=<ENV_NAME>
az group create --name $RESOURCE_GROUP --location $LOCATION
az containerapp env create --name $CONTAINERAPPS_ENVIRONMENT --resource-group $RESOURCE_GROUP --location "$LOCATION"
az containerapp env dapr-component set --name $CONTAINERAPPS_ENVIRONMENT --resource-group $RESOURCE_GROUP --dapr-component-name configuration --yaml configuration.yaml 
IDENTITY_ID=$(az identity show -n "<IDENTITY_NAME>" --resource-group "<IDENTITY_RG>" --query id | tr -d \")
az containerapp create   --name myapp   --resource-group $RESOURCE_GROUP   --user-assigned $IDENTITY_ID --environment $CONTAINERAPPS_ENVIRONMENT   --image <IMAGE>   --min-replicas 1   --max-replicas 1   --enable-dapr   --dapr-app-id myapp   --dapr-app-port 80 --dapr-app-protocol grpc

configuration.yaml referred above is this:

componentType: configuration.azure.appconfig
version: v1
metadata:
  - name: host # host should be used when Azure Authentication mechanism is used.
    value: "<CONFIG_STORE_HOST>"
  - name: azureClientId # Optional
    value: "<IDENTITY_CLIENT_ID>"
scopes:
  - myapp

The two scripts should ideally do the same work, as both use same identity resource and same configuration store, with similar settings. However, the Terraform script reproduced the error mentioned in description but az cli worked fine fetching data correctly. I repeated above excercise multiple times with both terraform and az cli, and got same result.

From above, it doesn't look like there is any issue from Dapr side. @torosent can you please take a look at it? It seems the issue is either in Terraform or some internal ACA management.

[cilerler]

@shivamkm07 CC:@torosent

I don't think your test is correct. I created exact match here using AZ CLI and PowerShell, and I have the same issue. The same workaround works for it.

~I think the reason you believe it's working is most likely because you didn't delete the DAPR configuration in the environment, between deployments, which definitely makes it work next time.~

I've discovered why yours is working. You have the --user-assigned line, which doesn't exist in either the Terraform script or CLI script I created. When I include that line, it works just fine. Without it, the issue occurs, and I need to apply the workaround to make it work. But that shouldn't be defined in there, it should be (and it is already) in DAPR component.