azure eventhub kafka exporter

[curlyromania]

Hi,

I am trying to configure the otel-collector in k8s to export to azure eventhub which is kafka enabled.

My configuration looks like this: (Azure requires the username to be exactly $ConnectionString so starting with $ sign)

---

exporters:
  logging:
  kafka:
    brokers:
      - eventhub-namespace.servicebus.windows.net:9093
    auth:
      sasl:
          username: "$ConnectionString"
          password: "Endpoint=sb://eventhub-namespace.servicebus.windows.net/;SharedAccessKeyName=sas;SharedAccessKey=AAAAAABBBBBBBBCCCCCCCCCDDDDDDDDD=;EntityPath=azure-eventhub"
          mechanism: PLAIN

---

When I apply the manifest, the otel-collector pods restart and I get the following error in the logs:

---

2022-06-30T18:19:53.407Z info builder/exporters_builder.go:255 Exporter was built. {"kind": "exporter", "name": "logging"} Error: cannot build exporters: error creating kafka exporter: username have to be provided 2022/06/30 18:19:53 collector server run finished with error: cannot build exporters: error creating kafka exporter: username have to be provided

---

I assume it is because of the $ sign. If I remove the $ sign, I get a message which suggests that it cannot connect to the event hub

---

Error: cannot build exporters: error creating kafka exporter: kafka: client has run out of available brokers to talk to: EOF 2022/06/30 18:21:36 collector server run finished with error: cannot build exporters: error creating kafka exporter: kafka: client has run out of available brokers to talk to: EOF

---

Is there a way to pass that username as $ConnectionString as this is how Azure Event Hub requires it?

Thank you!

[TylerHelmuth]

pinging @pavolloffay @MovieStoreGuy as code owners

[dyladan]

@curlyromania try using double dollar signs like "$$ConnectionString". The dollar sign is used to expand environment variables in configs but a double dollar sign is how that behavior is escaped.

@TylerHelmuth this is not specific to the kafka exporter but best I can tell this is an undocumented feature. Code for the $$ escaping here https://github.com/open-telemetry/opentelemetry-collector/blob/0bc2777111d1f9b385d4776742c5cf5acc8fb3c1/config/internal/configsource/manager.go#L364

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[MovieStoreGuy]

Hey @curlyromania ,

Is this still an issue you're experiencing?

[dvelben]

Hi, I am trying the same scenario, kafka exporter to EventHub. I'm using $$ConnectionString as username and password obtained from ConnectionString from Azure Portal. I'm always getting:

2022-12-14T11:34:51.740Z info service/telemetry.go:111 Setting up own telemetry... 2022-12-14T11:34:51.740Z info service/telemetry.go:141 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} 2022-12-14T11:34:51.740Z info components/components.go:30 Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging", "stability": "Development"} Error: cannot build pipelines: failed to create "kafka" exporter, in pipeline "traces": kafka: client has run out of available brokers to talk to: EOF 2022/12/14 11:34:52 collector server run finished with error: cannot build pipelines: failed to create "kafka" exporter, in pipeline "traces": kafka: client has run out of available brokers to talk to: EOF

as result. This is the yml I'm using in otel operator:

kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: simplest spec: config: | receivers: zipkin: exporters: logging: kafka:
brokers:

• ***.servicebus.windows.net:9093 topic: otel protocol_version: 2.0.0 auth: sasl: username: "$$ConnectionString" password: "Endpoint=sb://***" mechanism: PLAIN service: pipelines: traces: receivers: [zipkin] processors: [] exporters: [logging,kafka]

Do you have any suggestion I could try?

Thanks!!

[dvelben]

After some testing, the failure reason is both sasl AND tls auth entries are required to be configured. In my lab case,

       auth:
            sasl:
              username: "$$ConnectionString"
              password: "Endpoint=sb://..."
              mechanism: PLAIN
            tls:
              insecure: true

Thanks!

[github-actions[bot]]

Pinging code owners for exporter/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

Do you still need any more help on this one or you happy if I close it @dvelben, @curlyromania ,?

[616b2f]

@MovieStoreGuy I have similar issue, can kafka_exporter be configured to be more verbose in the logging? sarama library can give us more hints about the actual issue, but I didn't saw any possibility that kafka exporter can be configured to capture debugging output of sarama (maybe via sarama.Logger?)

[jakedern-msft]

@MovieStoreGuy @616b2f There's seems to be a regression here with the Kafka exporter with respect to integrating with Event Hub. I did a POC a while back with version 0.48.0 of the collector which worked just fine.

I was just dusting that off today for a different project and spent some time scratching my head at this with version 0.79.0 of the collector. Once I reverted back to 0.48.0 again it worked just fine. I used the same configuration for both:

kafka:
  brokers: 
        - <namespace>.servicebus.windows.net:9093
  topic: structured-logs
  protocol_version: 2.0.0
  encoding: otlp_proto
  auth:
    tls:
      insecure: true
     sasl:
       username: "$$ConnectionString"
       password: "Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=structured-logs-all;SharedAccessKey=AAAAAABBBBBBBCCCCCCCCC=;EntityPath=<topic>"
       mechanism: PLAIN

I have been unable to find a working configuration with 0.79.0.

[jakedern-msft]

@MovieStoreGuy @616b2f In case you're curious I did a quick bisect on this this morning and I can export to Event Hub just fine all the way through version 0.69.0, but not with version 0.70.0.

My first guess was a Sarama change, but it only had a patch version bump since then here.

[616b2f]

@MovieStoreGuy @616b2f In case you're curious I did a quick bisect on this this morning and I can export to Event Hub just fine all the way through version 0.69.0, but not with version 0.70.0.

My first guess was a Sarama change, but it only had a patch version bump since then here.

@jakedern-msft Thank you very much for sharing your observations. My first guess was that it has something Todo with sarama as well and this issue sounds kind of related here

But if it worked for you in a version where the sarama v1.38.0 was used than it can not be the same issue.

I am somehow glad that I am not the only one who had issues with that, because after a couple of hours I spend too try to get it work I started to think that I just doing something wrong. :)

[tesharp]

I ran into this problem as well. This issue suggest it could be with metadata request. I tried disabling full metadata, with no success. It looks like based on kafka protocol it uses different metadata request versions. I tried setting kafka_protocol to 1.0.0 and it worked.

Haven't looked at the code, but possible to somehow set the metadata request version?

[dnwe]

The problem with supporting Azure Event Hubs is that it isn't Apache Kafka under the covers, it's an intermediate proxy that supports a subset of the Kafka APIs at various versions and then maps them onto Event Hubs protocol(s) at the backend. As as result Sarama's current mechanism of specifying the KAFKA_VERSION to determine what protocol versions to support and use doesn't really work properly with Event Hubs because it supports an unusual set of protocols and even defines minimum versions for ProduceRequest (v3) and FetchRequest (v4). For some reason EventHubs is very behind on FetchRequest and MetadataRequest so the max configuration you can use in Sarama is V1_0_0_0.

Here are the equivalent max versions for Kafka 1.0 and EventHubs:

    APIs                                 1.0  EventHubs
0   ProduceRequest                       v5   v7
1   FetchRequest                         v6   v6
2   ListOffsetsRequest                   v2   v7
3   MetadataRequest                      v5   v5
8   OffsetCommitRequest                  v3   v8
9   OffsetFetchRequest                   v3   v8
10  FindCoordinatorRequest               v1   v1
11  JoinGroupRequest                     v2   v7
12  HeartbeatRequest                     v1   v4
13  LeaveGroupRequest                    v1   v4
14  SyncGroupRequest                     v1   v5
15  DescribeGroupsRequest                v1   v5
16  ListGroupsRequest                    v1   v4
17  SaslHandshakeRequest                 v1   v1
18  ApiVersionsRequest                   v1   v3
19  CreateTopicsRequest                  v2   v6
20  DeleteTopicsRequest                  v1   v2
22  InitProducerIdRequest                v0   v1
23  OffsetForLeaderEpochRequest          v0   v0
32  DescribeConfigsRequest               v0   v2
36  SaslAuthenticateRequest              v0   v1
37  CreatePartitionsRequest              v0   v1
42  DeleteGroupsRequest                       v0
47  OffsetDeleteRequest                       v0

[dnwe]

As discussed, pinning version to V1_0_0_0 manually would have prevented this issue when using Azure Event Hubs. However, it should also be resolved in v1.41.1 onward where we will force a pin to V1_0_0_0 if we think we're connecting to Event Hubs

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

This update has been adopted by the kafka exporter, hopefully this fixes the underlying issue. Closing this issue as resolved, please reopen if more discussion is needed.

[tonymkhael]

Hello,

Is there a working configuration that someone has managed to make it work with the latest OTEL version? I'm using `kafka: topic: #### protocol_version: 2.1.0 brokers:

• :9093

  auth: sasl: username: "$$ConnectionString" password: "Endpoint=sb://####.servicebus.windows.net/;SharedAccessKeyName=send-policy;SharedAccessKey=####;EntityPath=####" mechanism: PLAIN version: 0`

I've also tried playing around with sasl versions and kafka protocol versions but I always get Error: cannot start pipelines: kafka: client has run out of available brokers to talk to: EOF without any additional information.

[etiago]

@tonymkhael and others still facing issues, I just got this to work with the latest otel-collector version. The key thing for it to work for me was this:

kafka:
  auth:
    tls:
      insecure: false

Obviously you need all of the other settings as the examples above, but unless you explicitly set insecure to false, the connection fails.

[clemensv]

Having to set "insecure: false" explicitly is bizarre and in 2024, "insecure by default" ought not to exist. If @curlyromania doesn't want to reopen it, I suggest that @etiago opens a new one referencing this one if "insecure: true" is indeed the default when the flag is absent.