Kafa sink not retaining the key value.

[vegesana]

Describe the bug I added key to my message (kafka usecase) in UDF vertex as part of mapping. I pinned two sink vertices to the UDF - one to sink log and one to kafka sink.

• In the sink logs i see that the key exists in messages. But the message sent by kafka sink to the kafka topic does not have the key. Basically i lost the key after it is sent by kafka sink. It looks like kafka sink is not setting the headers.

To Reproduce Steps to reproduce the behavior:

1) Send message to kafka sink via UDF with key value set for the message. 2) Check the message on hte kafka topic on to which the kafka sink is sending. 3) Key is not present.

Expected behavior Need key value to be ratained by the kafka sink. Else with multiple partitions in kafka topic will cause out of order situations

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Numaflow: [v1.3.3]
• Client Version: [v1.31.0]
• Kustomize Version: [v5.4.2]
• Server Version: [v1.31.0]

Additional context Add any other context about the problem here.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

For quick help and support, join our slack channel.

[vigith]

how do you want the keys to be added to the header? keys is an array, while header is key/value.

[vegesana]

Thats true.. when i started making changes to fix this, even i realized it. Like addkeys() and addtags() for messages, we might need to add addHeaders() api as well. For key, we might need to add new field in the header. That is my opinion which i am planning to tryOn Oct 10, 2024, at 7:55 PM, Vigith Maurice @.***> wrote: how do you want the keys to be added to the header? keys is an array, while header is key/value.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

[vigith]

closed via #2143 2143