Upstream connect error when accessing a DAPR-enabled container app from Postman

[khaledhikmat]

Hi, just started looking at DAPR and ACA. I created a little app that contains 3 services using DAPR. The services use Redis as state and actors store and also as pubsub. It works well locally. Trying to use ACA, I created a Redis cache instance version 6 in Azure and dockerized my services.

Using azure cli, I deployed the first service and it returned success. However, trying to access it from Postman, I get the infamous Envoy error: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111. Looking into the logs, I see that things completed more or less successfully. Please see attached for the relevant logs. I appreciate any insight. actors.log

In addition, trying to deploy the second and third services as container apps into the same environment, I get operation expired after several minutes of twirling around. In Azure portal, it says provision failed. I also appreciate any help here.

Also...I noticed that the log analytics keeps data after it is destroyed! Yesterday, I tried this and it did not work. So I deleted the resource group and re-created everything today using the same resource names (same script). I saw log entries from yesterday in my new log analytics resource...really confusing. Is that normal?

Thank you for a beautiful product.

[dariagrigoriu]

@khaledhikmat, is it possible to clarify if Redis in this case is deployed as a managed service in Azure? Also, what is the development stack used, is this Python?

[khaledhikmat]

Hi @dariagrigoriu, thank you for your response. Yes....Redis is a managed service in Azure...I used version 6 because it supports Redis Streams for pubsub. This is C# code using the .Net SDK.

BTW....I forgot to mention that I also tried, instead of Redis, Azure storage for state store and Azure ServiceBus for pubsub with very similar results.

The application works in local hosting and also in local k8s (using Docker desktop on Mac).

Thanks for your help.

[kendallroden]

Hi @khaledhikmat ! Is this code that you can share publicly via GitHub? I would love to test myself and see if I can get some insight back to you. I agree the output logs look fine from a Dapr perspective. Is your application all http based or are you using gRPC? Thanks for the insight.

[khaledhikmat]

Hi @kendallroden, oh...that would be nice. Thank you so much. I threw it together quickly to see how it feels with DAPR ...I picked up a lot of stuff from the traffic controller sample app in DAPR.

I have 3 services: actors, entities and orders. The orders service has an API Endpoint that accepts an order, it enqueues to a pusub and returns. The pubsub notifies a controller in the actors service to process the order. The process controller in the actors service uses the order's store ID as actor ID and calls the store actor. The store actor, upon initial entry, loads its state from an external store so it can find out who its parent is. Each actor processes the order by simply accumulating counters and then notifies their parent via a reminder. Also used timers to persist actor state externally. This way the entities API Endpoint returns store state by querying the external store as opposed to querying the actor. I wanted actors be dedicated for processing since they are turn-based.

So the idea is to simulate a store chain where order revenue and quantities have to be aggregated to a city level, state level and to a country level.

Please excuse the mess in the repository's README. I will clean up once this works :-) Here is the repository in Github: https://github.com/khaledhikmat/store-manager

[khaledhikmat]

Hi @kendallroden, any chance looking into this please? Thanks.

[kendallroden]

Hey @khaledhikmat I will look into this and get back to you as soon as possible. Thanks for supplying the code and for your patience.

Adding @paulyuk for visibility

[khaledhikmat]

Sounds good. Thanks @kendallroden.

[khaledhikmat]

Hi @kendallroden I tried this again today. I confirm that it works locally and in a local k8s...but not in Azure Container Apps. In fact, today the problem got worst in ACA. The first app seems to return successful result but it is not visible in the portal at all....in other words, the environment shows no apps! The second app returns the operation expired. Please see below for how the output looks like:

az containerapp create \
>   --name storemanageractors \
>   --resource-group $RESOURCE_GROUP \
>   --environment $CONTAINERAPPS_ENVIRONMENT \
>   --image khaledhikmat/store-manager-actors:1.0 \
>   --target-port 6000 \
>   --ingress 'external' \
>   --min-replicas 1 \
>   --max-replicas 1 \
>   --enable-dapr \
>   --dapr-app-port 6000 \
>   --dapr-app-id storemanageractors \
>   --dapr-components ./components/components.yaml
Command group 'containerapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
{
  "configuration": {
    "activeRevisionsMode": "Multiple",
    "ingress": {
      "allowInsecure": false,
      "external": true,
      "fqdn": "storemanageractors.happytree-f3a15784.canadacentral.azurecontainerapps.io",
      "targetPort": 6000,
      "traffic": [
        {
          "latestRevision": true,
          "revisionName": null,
          "weight": 100
        }
      ],
      "transport": "Auto"
    },
    "registries": null,
    "secrets": null
  },
  "id": "/subscriptions/*******/resourceGroups/storemanager-rg/providers/Microsoft.Web/containerApps/storemanageractors",
  "kind": null,
  "kubeEnvironmentId": "/subscriptions/******/resourceGroups/storemanager-rg/providers/Microsoft.Web/kubeEnvironments/storemanager-env",
  "latestRevisionFqdn": "storemanageractors--iyqsms8.happytree-*****.canadacentral.azurecontainerapps.io",
  "latestRevisionName": "storemanageractors--iyqsms8",
  "location": "Canada Central",
  "name": "storemanageractors",
  "provisioningState": "Succeeded",
  "resourceGroup": "storemanager-rg",
  "systemData": {
    "createdAt": "2022-01-16T04:41:40.0312615",
    "createdBy": "*****@gmail.com",
    "createdByType": "User",
    "lastModifiedAt": "2022-01-16T04:41:40.0312615",
    "lastModifiedBy": "*****@gmail.com",
    "lastModifiedByType": "User"
  },
  "tags": null,
  "template": {
    "containers": [
      {
        "args": null,
        "command": null,
        "env": null,
        "image": "khaledhikmat/store-manager-actors:1.0",
        "name": "storemanageractors",
        "resources": {
          "cpu": 0.5,
          "memory": "1Gi"
        }
      }
    ],
    "dapr": {
      "appId": "storemanageractors",
      "appPort": 6000,
      "components": [
        {
          "metadata": [
            {
              "name": "redisHost",
              "secretRef": "",
              "value": "******.redis.cache.windows.net:6379"
            },
            {
              "name": "redisPassword",
              "secretRef": "",
              "value": "*****="
            },
            {
              "name": "actorStateStore",
              "secretRef": "",
              "value": "true"
            },
            {
              "name": "keyPrefix",
              "secretRef": "",
              "value": "name"
            }
          ],
          "name": "statestore",
          "type": "state.redis",
          "version": "v1"
        },
        {
          "metadata": [
            {
              "name": "redisHost",
              "secretRef": "",
              "value": "*****.redis.cache.windows.net:6379"
            },
            {
              "name": "redisPassword",
              "secretRef": "",
              "value": "*****="
            },
            {
              "name": "consumerID",
              "secretRef": "",
              "value": "myGroup"
            },
            {
              "name": "enableTLS",
              "secretRef": "",
              "value": "false"
            }
          ],
          "name": "pubsub",
          "type": "pubsub.redis",
          "version": "v1"
        }
      ],
      "enabled": true
    },
    "revisionSuffix": "",
    "scale": {
      "maxReplicas": 1,
      "minReplicas": 1,
      "rules": null
    }
  },
  "type": "Microsoft.Web/containerApps"
}
Khaleds-MBP:azure-container-apps khaled$ az containerapp create \
>   --name storemanagerorders \
>   --resource-group $RESOURCE_GROUP \
>   --environment $CONTAINERAPPS_ENVIRONMENT \
>   --image khaledhikmat/store-manager-orders:1.0 \
>   --target-port 6001 \
>   --ingress 'external' \
>   --min-replicas 1 \
>   --max-replicas 1 \
>   --enable-dapr \
>   --dapr-app-port 6001 \
>   --dapr-app-id storemanagerorders \
>   --dapr-components ./components/components.yaml
Command group 'containerapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Deployment failed. Correlation ID: 89aeccfe-fc61-4172-9fa1-433b095e70d2. Failed to provision revision for container app 'storemanagerorders'. Error details: Operation expired.

Also attached is how the portal looks like....I refreshed several times.....there are no apps in the env....not sure where the first app went: !

[khaledhikmat]

Hi @kendallroden i guess I am giving up on this. I think there is something seriously wrong when DAPR is involved...it does seem you can only deploy one DAPR service/app in an environment. Unfortunately, DAPR is the only reason I wanted to try this.

[jeffhollan]

I think you hit this issue #77 - which I think may have been fixed by now? I just spun up a new container app environment in canadacentral and was able to deploy all apps. I think submitted an order and it seemed to work. Maybe try on a fresh new environment but I think you hit a bug that appears to be fixed now

[khaledhikmat]

@jeffhollan Yay....thank you very much. I confirm ....it is working now.