Filter Messages by Header Before Deserialization

[amine-abdelli]

Describe the solution you'd like

Hello KafkaJS team,

I hope this message finds you well. I would like to suggest a feature enhancement for KafkaJS that would be highly beneficial for our use case and potentially for many others.

Current Behavior: As it stands, KafkaJS does not support filtering messages by header before deserializing the value and key. This means that by default, all messages are deserialized, and any filtering based on headers is performed post-deserialization within the application.

Problem Statement: This current behavior poses performance and efficiency challenges. For applications with a high volume of messages, unnecessary deserialization of messages that do not meet the header criteria can lead to significant overhead and resource consumption.

Proposed Feature: We propose adding the capability to filter messages by header before the deserialization process. This would allow users to specify header filters that KafkaJS could apply before deserializing the message value and key. By doing so, only messages that match the specified header criteria would be deserialized and processed by the application.

Thank you for considering this feature request. We are happy to provide further details or assist in any way to help implement this feature.

Best regards,

Amine

[patrykwegrzyn]

As far i can tell the current behaviour is: KafkaJS delivers messages as raw buffers to the consumer without imposing any structure(serde) for both key and value.