[KEDA][Kafka] DNS resolution not working with a custom DNS server

[pgourlain]

This issue is a: (mark with an x)

• [ x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

We use custom kafka KEDA rule on a pod, but the dns resolution doesn't work as expected. Our Kafka cluster is linked to a private endpoint. So the dns resolution inside a pod is xxxx.westeurope.azure.confluent.cloud => X.240.Y.Z and from KEDA xxxx.westeurope.azure.confluent.cloud =>10.1.0.4

our scaling rule :

"scale": {
    "minReplicas": 1,
    "maxReplicas": 6,
    "rules": [
        {
            "name": "kafka-rule",
            "custom": {
                "type": "kafka",
                "metadata": {
                    "bootstrapServers": "xxxx.westeurope.azure.confluent.cloud:9092",
                    "consumerGroup": "myconsumergroup",
                    "lagThreshold": "20",
                    "offsetResetPolicy": "earliest",
                    "sasl": "plaintext",
                    "tls": "enable",
                    "topic": "mytopic"
                },
                "auth": [
                    {
                        "secretRef": "kafka-keda-username",
                        "triggerParameter": "username"
                    },
                    {
                        "secretRef": "kafka-keda-password",
                        "triggerParameter": "password"
                    }
                ]
            }
        }
    ]
},

note that the kafka cluster url has a public resolution => 10.1.0.4, but in our network is X.240.Y.Z.

Steps to reproduce

1) use confluent KAfka cluster with private connectivity 2) create Azure VNET and set DNS servers to custom 3) Create Container App Environment in subnet of previous vnet (Workload profile is used) 4) Create a Container App that consume Kafka (Dapr is not used) 5) Setup scaling rule as above

Expected behavior [What you expected to happen.]

KEDA should resolve url 'xxxx.westeurope.azure.confluent.cloud' to X.240.Y.Z

Actual behavior [What actually happened.]

KEDA resolve url 'xxxx.westeurope.azure.confluent.cloud' to 10.1.0.4, so scaling doesn't works

Screenshots
If applicable, add screenshots to help explain your problem. extacted from system logs :

VNET DNS configuration :

workload profile configuration :

Additional context

[stweb1963]

We are seeing similar issues

Issue

• Container App Jobs Scaling not working
• Container App Apps (Event Based) not start from zero instances
• Numerous Authentication issues
  • Source address from our Egress Firewall (Odd???)

Our implementation

• Dedicated Workload profiles
• Custom DNS servers (VNETs configured pointing to these DNS servers)
  • DNS configured on Container App VNET
• Private Endpoints (private link zones owned and resolved by these servers)
  • Private link zones own and managed on our DNS servers
  • Private link endpoints in dedicated VNETs (peered to Container Apps VNET)
  • We do not use Private DNS for 99% of our DNS resolution needs (SQL MI special case)
• We are using KEDA scaling rules (Azure Storage Queues)
• VNET/Subnet has UDR and NSG applied
  • Egress default route 0.0.0.0/0 => firewall

Observations/Issues

• Scaling does not work
• Numerous authentication requests sourcing from our Egress firewall to internal private link addresses
  • Suspicious of DNS resolution not resolving to private DNS addresses
  • See this in Log Analytics logging on Storage Queues
• ACR Image Pulls failing
• Container App Jobs not scaling and Container App Apps not starting (Event scaling from Zero)

Band-Aid solution / resolution

• Deploy Private DNS Zones
  • (Queues/Tables/ACR
  • Link to Container App VNET
  • Manually add A records pointing to Private Ling Endpoint IP

End result / outcome

• All issues are resolved
• Scaling now working as expected
• ACR image pulls succeeding
• No more authentication failures in Log Analytics for storage/queues, etc.
• Jobs scaling
• App Event scaling working as expected

Issues Still Observed

Although all appeared to be ok for now, we uncovered one last issue that is blocking the scaling

• ACR Image Pulls are failing
• This is only observed on Container Apps with scaling configured and the minimum replicas is set to zero
• Unfortunately there are no logs/events/etc that can indicate the underlying problem

[cachai2]

Hi @pgourlain and @stweb1963, we identified a bug and should have a fix out by end of January.

[mohandoukaci]

Hello, do we have any news about this issue? if it's fixed, do we need to redeploy our Container App environnement?