Crash loop with eventhub/kafka with scale rule custom kafka and minReplicas 0

adeturner commented 2 years ago

This issue is a: (mark with an x)

[x] bug report -> please search issues before submitting
[ ] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description

Firstly I'd like to say that I'm really excited by the promise of containerapps, and thank you it is taking an excellent direction.

I'm trying to implement a custom scale rule with kafka keda settings against an eventhub with kafka enabled.

My go/librdkafka container starts, polls the topic, writes any events it finds to a blob, loops if there are more events, then exits. It works fine in test.

At deployment time the topic is empty

I hit a side issue - az cli doesnt work with scale rules currently; --scale-rules has disappeared which I reported against https://github.com/microsoft/azure-container-apps/issues/152.

I switched to deployment templates (ARM) reproduced below. Note the "scale" section with "minReplicas=0" and the custom rule.

The template deploys ok, but I'm getting container crash loops.

The logs show the container starting and stopping repeatedly, with no errors as this is expected behaviour when there are no events to process.

My assumption is that it should be waiting for the KEDA to start it up

Checking:

In the JSON View in the portal for the containerapp, the scale settings are visible.

In the JSON View in the portal for the revision, the scale settings are missing

Also in the portal revision page the scale section is empty:

Scale rule setting > Min / max replicas > 0 - 1
Scale rule > There are no scaling rules defined for this revision

Running

az containerapp revision show -n ${CONTAINERAPP_NAME} -g ${FRONTEND_RESOURCE_GROUP} --revision ${REVISION}

produces mostly expected output but without any scale section at all.

Second side issue: if we can get this to work, maybe we can add some more examples of complex templates to the docs

Steps to reproduce

az deployment group create -g $FRONTEND_RESOURCE_GROUP -n $CONTAINERAPP_DEPLOYMENTNAME --template-file $CONTAINERAPP_TEMPLATE --parameters $CONTAINERAPP_PARAMS

For anyone following these are the eventhub kafka parameter values:

        "kafka-username": {
            "value": "$ConnectionString"
        },
        "kafka-password": {
            "value": "Endpoint=sb://blah.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=blah"
        },
        "sasl-type": {
            "value": "plaintext"
        },

Expected behavior [What you expected to happen.]

Container deploys and waits for KEDA to start

Actual behavior [What actually happened.]

Crash loop

Screenshots

n/a

Additional context

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "environmentName": {
            "type": "string"
        },
        "appName": {
            "type": "string"
        },
        "acr-username": {
            "type": "string"
        },
        "acr-password": {
            "type": "string"
        },
        "acr-server": {
            "type": "string"
        },
        "container-image": {
            "type": "string"
        },
        "location": {
            "type": "string"
        },
        "tenant-id": {
            "type": "string"
        },
        "subscription-id": {
            "type": "string"
        },
        "client-id": {
            "type": "string"
        },
        "client-secret": {
            "type": "string"
        },
        "vault-url": {
            "type": "string"
        },
        "secretname": {
            "type": "string"
        },
        "source": {
            "type": "string"
        },
        "target": {
            "type": "string"
        },
        "url": {
            "type": "string"
        },
        "kafka-username": {
            "type": "string"
        },
        "kafka-password": {
            "type": "string"
        },
        "bootstrap-servers": {
            "type": "string"
        },
        "sasl-type": {
            "type": "string"
        },
        "topic": {
            "type": "string"
        },
        "consumer-group": {
            "type": "string"
        }
    },
    "resources": [
        {
            "type": "Microsoft.App/containerApps",
            "apiVersion": "2022-01-01-preview",
            "name": "[parameters('appName')]",
            "location": "[parameters('location')]",
            "properties": {
                "managedEnvironmentId": "[resourceId('Microsoft.App/managedEnvironments/', parameters('environmentName'))]",
                "configuration": {
                    "registries": [{
                        "server": "[parameters('acr-server')]",
                        "username": "[parameters('acr-username')]",
                        "passwordSecretRef": "acr-password"
                    }],
                    "secrets": [
                        {
                            "name": "tenant-id",
                            "value": "[parameters('tenant-id')]"
                        },
                        {
                            "name": "subscription-id",
                            "value": "[parameters('subscription-id')]"
                        },
                        {
                            "name": "client-id",
                            "value": "[parameters('client-id')]"
                        },
                        {
                            "name": "client-secret",
                            "value": "[parameters('client-secret')]"
                        },
                        {
                            "name": "vault-url",
                            "value": "[parameters('vault-url')]"
                        },
                        {
                            "name": "secretname",
                            "value": "[parameters('secretname')]"
                        },
                        {
                            "name": "source",
                            "value": "[parameters('source')]"
                        },
                        {
                            "name": "target",
                            "value": "[parameters('target')]"
                        },
                        {
                            "name": "url",
                            "value": "[parameters('url')]"
                        },
                        {
                            "name": "acr-password",
                            "value": "[parameters('acr-password')]"
                        }
                    ]
                },
                "template":{
                    "containers":[
                        {
                            "image": "[parameters('container-image')]",
                            "name": "datafabric-eventdriven-keda",
                            "resources": {
                                "cpu": 1.00,
                                "memory": "2Gi"
                            },
                            "env": [
                                {
                                    "name": "TENANT_ID",
                                    "secretref": "tenant-id"
                                },
                                {
                                    "name": "SUBSCRIPTION_ID",
                                    "secretref": "subscription-id"
                                },
                                {
                                    "name": "CLIENT_ID",
                                    "secretref": "client-id"
                                },
                                {
                                    "name": "CLIENT_SECRET",
                                    "secretref": "client-secret"
                                },
                                {
                                    "name": "VAULT_URL",
                                    "secretref": "vault-url"
                                },
                                {
                                    "name": "SECRET_NAME",
                                    "secretref": "secretname"
                                },
                                {
                                    "name": "SOURCE",
                                    "secretref": "source"
                                },
                                {
                                    "name": "TARGET",
                                    "secretref": "target"
                                },
                                {
                                    "name": "URL",
                                    "secretref": "url"
                                }
                            ]
                        }
                    ],
                    "scale": {
                        "minReplicas": 0,
                        "maxReplicas": 1,
                        "rules": [
                            {
                                "name": "kafka-rule",
                                "custom": {
                                    "type": "kafka",
                                    "auth": [
                                        {
                                            "triggerParameter": "sasl",
                                            "secretRef": "sasl-type"
                                        },
                                        {
                                            "triggerParameter": "username",
                                            "secretRef": "kafka-username"
                                        },
                                        {
                                            "triggerParameter": "password",
                                            "secretRef": "kafka-password"                                           
                                        }
                                    ],
                                    "metadata": {
                                        "bootstrapServers": "[parameters('bootstrap-servers')]",
                                        "consumerGroup": "[parameters('consumer-group')]",
                                        "topic": "[parameters('topic')]",
                                        "lagThreshold": "5"
                                    }
                                }
                            }
                        ]
                    }
                }
            }
        }
    ]
}

andreaskasc commented 1 year ago

Not sure if you have already solved this bug, but I am experimenting with a similar setup (custom kafka scale rule connecting to Azure Event Hub with "KafkaEnabled"). I had to add a trigger parameter for tls in my rule to make it work. So my auth section of the rule looks like:

"auth":
 [
    {
        "secretRef": "event-hub-sasl",
        "triggerParameter": "sasl"
    },
    {
        "secretRef": "event-hub-sasl-username",
        "triggerParameter": "username"
    },
    {
        "secretRef": "event-hub-sasl-password",
        "triggerParameter": "password"
    },
    {
        "secretRef": "event-hub-tls",
        "triggerParameter": "tls"
    }
]

And the value for tls is 'enable'.

Hope this helps, Andreas

simonjj commented 1 month ago

This seems resolved, please reopen if it still exists. Closing now.

microsoft / azure-container-apps