Kafka Binding v1.11.3: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.

[Andemki]

Expected Behavior

Dapr and Azure Event Hub integration using Kafka binding work without errors

Actual Behavior

I've created a new issue item based on the discussion in already resolved item: https://github.com/dapr/components-contrib/issues/2874

I've been using bindings.kafka component for recieving messages from Azure Event Hub in my application. Everything work properly in Dapr versions 1.8.7. But when I updated dapr components to any version which is greater than 1.8.7 (1.9.0 and higher), my ASP.Net Core (.Net 6) application stopped receiving new messages from EventHub. I've noticed the following error message in the sidecar container (daprd):

2023-10-02 18:17:19 time="2023-10-02T17:17:19.517325Z" level=error msg="Error processing Kafka message: my-eh-data/20/219107532 [key=MTgwOA==]. Error: fails to send binding event to http app channel, status code: 400 body: ." app_id=my-eventhublistener component="my-eventhub-receiver (bindings.kafka/v1)" instance=7aac653b4cf6 scope=dapr.contrib type=log ver=1.11.3
2023-10-02 18:17:19 time="2023-10-02T17:17:19.5173532Z" level=debug msg="Processing Kafka message: my-eh-data/20/219107533 [key=MTQwMQ==]" app_id=my-eventhublistener component="my-eventhub-receiver (bindings.kafka/v1)" instance=7aac653b4cf6 scope=dapr.contrib type=log ver=1.11.3
2023-10-02 18:17:19 time="2023-10-02T17:17:19.5210655Z" level=debug msg="error from app consumer for binding [my-eventhub-receiver]: fails to send binding event to http app channel, status code: 400 body: " app_id=my-eventhublistener instance=7aac653b4cf6 scope=dapr.runtime type=log ver=1.11.3

And I also noticed the following error message in my ASP.Net Core application's logs:

2023-10-02 17:30:21 [16:30:21 Debug] Microsoft.AspNetCore.Server.Kestrel.BadRequests
2023-10-02 17:30:21 Connection id "0HMU3CJ88DNTV" bad request data: "Malformed request: invalid headers."
2023-10-02 17:30:21 Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.
2023-10-02 17:30:21    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
2023-10-02 17:30:21    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2023-10-02 17:30:21    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

And what I also found is that requests from different dapr versions have different headers: v.1.11.3 (doesn't work for me):

2023-10-02 17:30:12 [16:30:12 Information] Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware
2023-10-02 17:30:12 Request:
2023-10-02 17:30:12 Protocol: HTTP/1.1
2023-10-02 17:30:12 Method: POST
2023-10-02 17:30:12 Scheme: http
2023-10-02 17:30:12 PathBase: 
2023-10-02 17:30:12 Path: /receive_predictions
2023-10-02 17:30:12 Host: 127.0.0.1:80
2023-10-02 17:30:12 User-Agent: Go-http-client/1.1
2023-10-02 17:30:12 Accept-Encoding: gzip
2023-10-02 17:30:12 Content-Type: application/cloudevents+json
2023-10-02 17:30:12 traceparent: [Redacted]
2023-10-02 17:30:12 Content-Length: 3171
2023-10-02 17:30:12 Pubsubname: [Redacted]

v.1.8.7 (it works properly without errors):

2023-05-31 16:19:46 [15:19:46 Information] Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware
2023-05-31 16:19:46 Request:
2023-05-31 16:19:46 Protocol: HTTP/1.1
2023-05-31 16:19:46 Method: OPTIONS
2023-05-31 16:19:46 Scheme: http
2023-05-31 16:19:46 PathBase:
2023-05-31 16:19:46 Path: /my-eventhub-receiver
2023-05-31 16:19:46 Host: 127.0.0.1:80
2023-05-31 16:19:46 User-Agent: fasthttp
2023-05-31 16:19:46 Content-Type: application/json
2023-05-31 16:19:46 traceparent: [Redacted]
2023-05-31 16:19:46 Content-Length: 0

The latest working version is 1.8.7. Starting from Dapr v.1.9.0 the Kafka binding component doesn't work for us.

Yaml output of my bindings.kafka component:

apiVersion: dapr.io/v1alpha1
auth:
  secretStore: my-kv-secrets
kind: Component
metadata:
  annotations:
    meta.helm.sh/release-name: my-eventhublistener
    meta.helm.sh/release-namespace: my-namespace
  labels:
    app.kubernetes.io/managed-by: Helm
  name: my-eventhub-receiver
  namespace: my-namespace
spec:
  metadata:
  - name: version
    value: 1.0.0
  - name: brokers
    value: my-eh-namespace.servicebus.windows.net:9093
  - name: topics
    value: my-eh-data
  - name: consumerGroup
    value: my-eventhublistener
  - name: authRequired
    value: "true"
  - name: saslUsername
    value: $ConnectionString
  - name: saslPassword
    secretKeyRef:
      key: eventhub-connectionstring
      name: eventhub-connectionstring
  type: bindings.kafka
  version: v1

Steps to Reproduce the Problem

1. Create a kafka.binding component to receive messages from Event Hub in your ASP.Net Core application
2. Deploy your application and the component to the Kubernetes
3. Check that your application doesn't receive messages from Event Hub

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[ssm3ll]

I have same problem when use EventHub via Kafka

[Andemki]

@berndverst Hello Bernd. Could you please take a look on this issue? We started to discuss it on the issue item #2874 and you suggested to create a separate issue for it. @ItalyPaleAle @DeepanshuA FYI

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[Andemki]

@berndverst Hello Bernd. Could you please take a look on this issue? We started to discuss it on the issue item https://github.com/dapr/components-contrib/issues/2874 and you suggested to create a separate issue for it. @ItalyPaleAle @DeepanshuA FYI