search

streamdal / plumber

A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
https://streamdal.com
MIT License
2.07k stars 71 forks source link

[Kafka] Fail to emit cloudevents on scram authed cluster #371

Open WDaan opened 2 months ago

WDaan commented 2 months ago

Describe the bug Emitting events with --encode-type=cloudevent works on an unauthenticated kafka-cluster but not on an authenticated one.

To Reproduce Steps to reproduce the behavior:

  1. Launch XYZ message broker
  2. Execute the following command: plumber write kafka --topics=uptime --input-file=/home/event.json --ce-id=$(cat /proc/sys/kernel/random/uuid) --encode-type=cloudevent

Expected behavior Expected the message to be sent.

Screenshots If applicable, add screenshots to help explain your problem.

Emitting a cloudevent on an unauthenticated cluster

image

Emitting a cloud event on an authenticated cluster

Screenshot 2024-05-04 at 09 10 18

Omitting the encode-type, does work but message is no longer in the cloudevent format.

Also, all events in a non cloud-event format, even if you set the --key-option end up on the same partition 🤔

Screenshot 2024-05-04 at 09 13 36

CLI (please complete the following information):

Additional context Add any other context about the problem here.

OcamsRazor commented 2 months ago

In the examples an SCRAMClientGeneratorFunc is set. Might be a fix based on the log message https://github.com/IBM/sarama/blob/0ab2bb77aeca321f41a0953a8c6f52472607a59e/examples/sasl_scram_client/main.go#L96-L98