Kafka Extension for isolated mode not sending key

[MO2k4]

Hi 👋

i am currently working on switching a kafka producer in-proc function towards a out-proc function. During the switch we noticed, that the produced message is not transported the same way as before

With the old method we used the KafkaEventData object and have seen a message on the kafka with a key and a value.

With the new binding for isolated, there was no KafkaEventData object any longer, so we decided to use our own class with just a key and a value property. Now the problem is, that we can see this message on the topic, but the message is only shown with a value. The new class is packed inside.

What can i do to make it work like before?

I've seen that there was previously a class Message from the confluent package used, but i was not able to find the new implementation for the communication against kafka.

[kshyju]

@jainharsh98 @raorugan Could you help with this?

[jainharsh98]

@MO2k4 As you pointed out the support for isolated is limited to string and string arrays. Did you try using headers to retrieve the key as in the function "KafkaTriggerWithHeaders" here.

[MO2k4]

@jainharsh98 will there be support for keys in the future? I tried it with the headers, but I was trying to get it look like before in lenses without having to adapt the model. This currently changes the model which results in a breaking change.

Can I get the Kafka producer injected, so that I am able to send the message the old way?

could you elaborate, why it is not implemented the old way and why it is not documented that this is not supported any longer?

[jainharsh98]

@MO2k4 I'll work with the documentation team to make sure that the documents are clear on this.

In the kafka extension here the object is searched for all the following keys -

1. Offset
2. Partition
3. Topic
4. Timestamp
5. Value
6. Headers

Only then it adds the key in the headers of the output msg here.

To understand if it is possible in current implementation to set the key without changing your model - Can you please share the custom class you are creating and also mention how are you returning the output in the function.

[MO2k4]

The example you are pointing out to, is the old one and not the one for isolated.

We tried using the old KafkaEventData model, we tried a class with just a value and a key property but it never worked.

[jainharsh98]

Sorry but which example are you pointing out as old and not for isolated? The examples in the documentations are pulled from the repository which are kept upto date - For example: here.

As explained above just adding a value and key would not work due to the checks in the code. Also, Are you serializing your KafkaEventData object for the ouput?

[MO2k4]

The example you pointed to is in the webjobs namespace, but the extension used is this extensions/Worker.Extensions.Kafka - does this mean we need to use both extensions?

We tried it serialized and not serialized but it did not work.

It was working before and it should just have been a switch of libraries and a change to the output binding - but it is not.

[MO2k4]

This confuses me, because I can't see a connection between both libs

[jainharsh98]

Understood the confusion. The Worker.Extension.Kafka contains the definitions of the attributes and some other classes that you can directly use in your function in the isolated model. The actual logic for how to produce/consume msgs for kafka lies in the function extension. As a user of the isolated model - you don't have to use the function extension directly in your function - The functions host internally uses the extension to perform the required operations. Without going into the exact details - The host also knows how to communicate with your function to provide trigger msgs or get the output msgs.

As a summary you don't have to use the function extension directly in your function for isolated model - but the logic on how to produce the msgs comes from the function extension.

Thanks for the information on the serialization. I'll try to repro it with all the above properties once and let you know.

Thanks for the feedback. We are working on improving the experience for kafka functions on isolated.

[ciaplant]

I came here from the far off lands of 2024 still extremely confused on how this ties together. The examples especially were a source of confusion for me.

[jainharsh98]

@ciaplant to be able to help you better can you please clarify what is the source of confusion for you. Are you facing the same issue as the original author where your keys are not being sent as a part of the output binding?

[ciaplant]

@ciaplant to be able to help you better can you please clarify what is the source of confusion for you. Are you facing the same issue as the original author where your keys are not being sent as a part of the output binding?

Every example given for Isolated-Process for Kafka functions in https://github.com/Azure/azure-functions-kafka-extension/tree/dev/samples/dotnet-isolated/confluent mentions only consuming or producing events using string or string[], and only through this post learned that other methods are available. It might be the only place this is mentioned at all.